Underwater target detection method based on multi-scale feature cross fusion

Abstract

The invention discloses an underwater target detection method based on multi-scale feature cross fusion, and belongs to the technical field of underwater target detection. According to the underwater target detection network, a serial double encoder-decoder structure is adopted, a lightweight encoder structure is adopted to perform multi-scale feature extraction, and enhancement processing is performed on a small-scale feature map through an MSFCA module, so that the purpose of fully extracting target features in an underwater target image is achieved; the decoder is responsible for gradually recovering and enhancing the features of the small-scale multi-channel feature maps, and splicing the feature maps reserved according to the down-sampling operation with the feature maps of the corresponding levels in the decoding process so as to enhance the feature propagation and compensate the loss generated in the down-sampling process in the encoder stage; and finally, carrying out a coding and decoding process, carrying out multi-level feature splicing, sending the features of three different levels obtained in the last decoding process into a corresponding detection head structure, and carrying out fusion output to finally realize accurate detection of the underwater target.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target detection, and in particular to an underwater target detection method based on multi-scale feature cross-fusion. Background Art

[0002] In recent years, my country has faced unprecedented challenges in maritime security, marine resource development, and ecological environmental protection. In the era of artificial intelligence, it is urgent to vigorously develop intelligent safety algorithms for complex underwater environments. However, due to the complexity and limited visibility of the underwater environment, there are small targets that gather and accumulate in the water, and are affected by light absorption and scattering, which can easily lead to problems such as blur, occlusion, and color distortion. Therefore, it is of great significance to improve the accuracy of underwater target detection algorithms through research on network models and data.

[0003] Underwater scene environments are diverse, and images collected in different waters / seas have different image quality and target types. In order to meet different types of detection needs, researchers have proposed a variety of underwater target detection methods. Existing underwater target detection methods are divided into three categories: two-stage detection methods, one-stage detection methods, and Transformer-based detection methods.

[0004] The two-stage detection method is mainly based on the improvement of the RCNN series of methods to make it more suitable for underwater target detection tasks. Some scholars have improved the two-stage algorithm of Faster-RCNN for detecting sea cucumbers, thorns, scallops, starfish and water plants. The VGG16 (Visual Geometry Group network 16) in the original feature extraction module is replaced by the Res2Net101 network to enhance the expression ability of the receptive field of each network layer; secondly, the OHEM (Online Hard Example Mining) algorithm is introduced to solve the problem of imbalance between positive and negative samples of the bounding box; thirdly, the regression mechanism of the bounding box is optimized by GIOU (Generalized Intersection OverUnion) and Soft-nms (Soft non-maximum suppression). The improved algorithm has better performance in underwater target detection. In addition, the RetinaRPN proposed by Boosting RCNN replaces the original region proposal network, and uses the method of boosting re-weighting to solve the inaccuracy of the candidate region when the region proposal network incorrectly calculates the object prior probability of the sample.

[0005] The one-stage detection method is mainly based on the improvement of YOLO. Compared with the two-stage algorithm, which needs to use a neural network to extract features first and then use a classifier to classify the features, YOLO directly predicts the category and position of the target object, which greatly improves the detection speed. In response to the problem that YOLOv5s is difficult to extract features and small targets cannot be detected when detecting underwater targets due to the influence of underwater environmental noise and target scale changes, some scholars have constructed a new underwater target detector based on YOLOv5s. It uses a feature enhancement gating module to selectively suppress or enhance multi-level features, reduce the interference of complex underwater environmental noise on feature fusion, and use adjacent feature fusion mechanism and dynamic fusion module to dynamically learn fusion weights and gradually perform multi-level feature fusion. Compared with YOLOv5s, this method has achieved good improvement in underwater small target detection tasks. Cai proposed an underwater detection method based on a weakly supervised learning framework, training two YOLOv5 deep learning detectors at the same time, and letting them train each other based on the selection of cleaner samples seen during training, achieving a balance between accuracy and speed. Some scholars integrated the decoder and auxiliary prediction head in RT-DETR into YOLOv8 and applied repulsion loss, providing solutions to the common occlusion and density problems in fish detection.

[0006] The Transformer-based object detection model relies on the self-attention mechanism to provide a new solution for underwater object detection. DETR is an end-to-end object detection based on Transformer proposed by the Facebook team. It regards object detection as a direct set prediction problem, abandons the non-maximum suppression NMS post-processing step and prior knowledge and constraints such as anchors, and simplifies the object detection process based on the global loss of the set, through bipartite matching and the encoder-decoder structure of Transformer. RT-DETR is the first real-time end-to-end object detector improved based on DETR. Specifically, RT-DETR integrates an efficient hybrid encoder to efficiently process multi-scale features by decoupling intra-scale interactions and cross-scale fusion. At the same time, the authors proposed an IoU-aware query selection mechanism to optimize the initialization of decoder queries. In addition, RT-DETR supports flexible adjustment of inference speed by using different decoder layers without retraining, which is conducive to the practical application of real-time object detectors. Compared with DETR, RT-DETR has brought great improvements in both real-time performance and accuracy. To address the problem that underwater images are severely degraded and cannot detect objects quickly and accurately, HTDet uses a lightweight hybrid Transformer network to extract global context information, and then uses a fine-grained feature pyramid network to overcome the problem of weak signal disappearance and uses test time enhancement for reasoning without introducing additional parameters. However, Transformer-based object detection methods often generate a lot of computational overhead and redundant parameters due to the calculation of self-attention, making it difficult to achieve lightweight deployment.

[0007] The inventors of this application found that in order to improve the accuracy of underwater target detection by neural networks, the following two aspects should be focused on and optimized: First, in the feature extraction stage, how to fully extract the features of underwater blurred targets and small targets during the downsampling process, so that the network's perception of these targets can be improved and accurate underwater target detection can be achieved; second, in the multi-level feature fusion stage, how to make full use of multi-scale features, reduce the problem of feature loss during upsampling and downsampling, and make the prediction frame more accurate. No matter which aspect, it is a great challenge to achieve the ideal effect. Summary of the invention

[0008] The present invention proposes an underwater target detection method based on multi-scale feature cross-fusion to achieve lightweight detection of underwater targets and improve the accuracy of underwater target detection.

[0009] The technical solution adopted by the present invention is:

[0010] An underwater target detection method based on multi-scale feature cross fusion, the method comprises the following steps:

[0011] Step 1, input underwater target images and preprocess them to obtain training data sets;

[0012] Step 2: Build an underwater target detection network based on multi-scale feature cross-fusion;

[0013] The underwater target detection network adopts a series of dual encoder-decoder structures. For the first encoder-decoder structure, a lightweight encoder structure is used as the backbone network to extract multi-scale features from the input image. A multi-scale cross-fusion attention MSCFA module is connected to the end of the encoder to enhance the small-scale feature map. The decoder is used to gradually restore and enhance the small-scale multi-channel feature map, and splice the feature map retained by the downsampling operation in the encoding process with the feature map of the corresponding level in the decoding process to enhance feature propagation and compensate for the loss caused by the downsampling process in the encoder stage.

[0014] For the second encoder-decoder structure, a MSCFA module is connected between the encoder and the decoder, the encoder is used to encode the output feature map of the first encoder-decoder structure, the decoder is used to gradually restore and enhance the feature map of the output feature map of the MSCFA module, and the encoder of the second encoder-decoder structure splices the feature map retained by the upsampling operation with the feature map of the corresponding level in the decoding process of the first encoder-decoder structure;

[0015] The feature maps of three different levels obtained in the last decoding process are sent to the corresponding detection head structure, and the outputs of the three detection head structures are fused to obtain the detection results of underwater targets;

[0016] Step 3: Based on the training data set, the underwater target detection network based on multi-scale feature cross fusion is trained to optimize the network parameters. When the preset training end conditions are met, a trained underwater target detection network is obtained.

[0017] Step 4: Input the underwater target image to be detected into the trained underwater target detection network to obtain the detection result.

[0018] Furthermore, in the first encoder-decoder structure, each encoding layer of the encoder includes a downsampling convolution layer and a lightweight dense bottleneck architecture LDBA module in sequence; each decoding layer of the decoder includes an upsampling convolution layer and a fast feature enhancement FFE module in sequence;

[0019] In the second encoder-decoder structure, each encoding layer of the encoder includes a downsampling convolution layer and a fast feature enhancement FFE module in sequence; each decoding layer of the decoder includes an upsampling convolution layer and a fast feature enhancement FFE module in sequence;

[0020] For the first encoder-decoder structure, define represents the number of coding layers of the first encoder, Indicates the number of decoding layers of the first decoder. The encoding layers in the first encoder-decoder structure are defined as the first encoding layer 1 to the first encoding layer according to the forward propagation direction. And define the decoding layers as first decoding layer 1 to first decoding layer Among them, the feature map of the first decoding layer i is the first encoding layer The output feature map of and Greater than

[0021] For the second encoder-decoder structure, define represents the number of coding layers of the second encoder, Indicates the number of decoding layers of the second decoder. The encoding layers in the second encoder-decoder structure are defined as the second encoding layer 1 to the second encoding layer according to the forward propagation direction. And define the decoding layers as second decoding layer 1 to second decoding layer Among them, the feature map of the second encoding layer j is the first decoding layer The output feature map of and when When the second coding layer The concatenated feature map is the output feature map of the MSCFA module in the first encoder-decoder structure, that is, the input feature map of the first decoding layer;

[0022] The feature map of the second decoding layer p is the second encoding layer The output feature map of and when When , the second decoding layer The concatenated feature map is the input feature map of the first second encoding layer;

[0023] Number of decoding layers for the second decoder when When , the three feature maps of different levels obtained in the last decoding process are: the output feature map of the MSCFA module in the second encoder-decoder structure, the output feature map of the two decoding layers of the second decoder; When , the feature maps of three different levels obtained in the last decoding process are the output feature maps of the last three decoding layers of the second decoder.

[0024] Furthermore, the structure of the MSCFA module is set as follows:

[0025] The input of the MSCFA module is feature maps of three different scales. The MSCFA module includes three parallel branches. Each branch obtains the intermediate fusion feature map of each scale through the channel multi-head self-attention mechanism. When executing the channel multi-head self-attention mechanism calculation, a feature map of one of the three scales is selected as the query Q, and the feature maps of the other two scales are scaled to the same size as the current Q scale and used as the key K and value V respectively. The intermediate fusion feature map of the current branch is obtained based on the multi-head channel self-attention calculation between Q, K, and V. Thus, the intermediate fusion feature maps of three scales are obtained: small-scale intermediate fusion feature map, medium-scale intermediate fusion feature map, and large-scale intermediate fusion feature map.

[0026] The large-scale intermediate fusion feature map is subjected to a downsampling convolution layer with a convolution kernel of 3×3, and then weighted fused with the medium-scale intermediate fusion feature map with a weight of 1-β using the adjustment factor β as the weight to obtain the first weighted fusion result;

[0027] The first weighted fusion result is weightedly fused with the small-scale intermediate fusion feature map with a weight of 1-α after passing through a downsampling convolution layer with a convolution kernel of 3×3, using the adjustment factor α as the weight, to obtain the second weighted fusion result;

[0028] The second weighted fusion result is passed through a convolution layer with a convolution kernel of 3×3 to obtain the output feature map of the MSCFA module.

[0029] Furthermore, the structure of the LDBA module is set as:

[0030] The main branch of the LDBA module includes: the first convolution layer, the slicing layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the splicing layer and the fifth convolution layer;

[0031] The splitting layer is used to evenly divide the input feature map into two sub-feature maps according to the number of channels. One sub-feature map is sent to the second convolutional layer, and the other sub-feature map is sent to the efficient multi-scale attention EMA module; the EMA module is used to generate pixel-level attention for the input feature map;

[0032] The output feature map of the third convolutional layer and the input feature map of the second convolutional layer are added pixel by pixel through a residual connection and then sent to the fourth convolutional layer;

[0033] The output feature map of the fourth convolutional layer and the output feature map of the second convolutional layer are added pixel by pixel through residual connection and then sent to the splicing layer; and the input of the splicing layer also includes the output of the EMA module and the output feature maps of the first to fourth convolutional layers.

[0034] Furthermore, the FFE module includes, in sequence: a first convolutional layer, a slicing layer, a partial convolutional layer, a second convolutional layer, a batch normalization layer, a concatenation layer, a third convolutional layer, and a fourth convolutional layer;

[0035] Among them, the splitting layer is used to evenly divide the input feature map into two sub-feature maps b2 and b3 according to the number of channels. The sub-feature map b2 is sent to the partial convolution layer, and the sub-feature map b3 is sent to the splicing layer;

[0036] The sub-feature map b2 and the output feature map of the second convolutional layer are added pixel by pixel through a residual connection and then sent to the batch normalization layer; and the input feature map of the batch normalization layer and the output feature map of the second convolutional layer are added pixel by pixel through a residual connection and then sent to the concatenation layer.

[0037] The technical solution provided by the present invention brings at least the following beneficial effects:

[0038] In underwater target detection tasks, the target to be detected often appears blurred, color-biased, and other characteristics due to problems such as light scattering, which greatly hinders the accuracy of detection. In order to achieve accurate detection of underwater targets, an embodiment of the present invention builds an underwater target detection network based on multi-scale feature cross-fusion. In this network, after the input image is processed by a convolutional neural network, multiple feature maps of different scales and different numbers of channels are obtained. Based on the constructed feature fusion network, these feature maps of different levels are fused to enhance the expression ability of the features of the target to be detected, which will improve the model's ability to extract underwater target features, thereby improving the accuracy of underwater target detection.

[0039] The underwater target detection network constructed by the embodiment of the present invention is a series of dual encoder-decoder structures. The encoder obtains multiple feature maps of different scales to perform multi-scale feature cross-fusion to fully extract the target features in the input image. The decoder is responsible for gradually restoring and enhancing the feature maps of small-scale multi-channels. Finally, the feature maps of three different sizes are sent to the respective detection heads to achieve accurate positioning of underwater targets. Specifically, the embodiment of the present invention first designs a lightweight encoder structure for efficient feature extraction as a backbone network to extract features from the input image, then obtains feature maps of different scales and processes them through the designed multi-scale feature cross-fusion module to enhance the deep features; in the decoder stage, the feature maps retained by the downsampling operation are spliced ​​with the feature maps of the corresponding levels in the decoding process to enhance feature propagation and compensate for the loss caused by the downsampling process in the encoder stage; finally, after another encoding and decoding process, multi-level feature splicing is performed to enhance the feature expression ability, and the features of three different levels obtained in the last decoding process are sent to the corresponding detection head structure and fused and output to finally achieve accurate detection of underwater targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of the overall process of an underwater target detection method based on multi-scale feature cross-fusion provided by an embodiment of the present invention;

[0042] Figure 2 The label distribution of the underwater target data set and some targets in the embodiment of the present invention;

[0043] Figure 3 It is a schematic diagram of the structure of an underwater target detection network based on multi-scale feature cross-fusion according to an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a lightweight dense bottleneck architecture according to an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a multi-scale cross-fusion attention structure according to an embodiment of the present invention;

[0046] Figure 6 A design diagram of a fast feature enhancement module according to an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of the detection results of an underwater target data set by the method proposed in an embodiment of the present invention;

[0048] Figure 8 The interpretability analysis result of the underwater target detection network constructed by the embodiment of the present invention;

[0049] Fig. 9 This is the detection result of the URPC2020 dataset by the method proposed in the embodiment of the present invention;

[0050] Fig.10 This is the detection result of the RUOD dataset using the method proposed in the embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0052] In underwater target detection tasks, the target to be detected often appears blurred, color-biased, and other characteristics due to problems such as light scattering, which greatly hinders the accuracy of detection. In order to achieve accurate detection of underwater targets, an embodiment of the present invention builds an underwater target detection network based on multi-scale feature cross-fusion. In this network, after the input image is processed by a convolutional neural network, multiple feature maps of different scales and different numbers of channels are obtained. Based on the constructed feature fusion network, these feature maps of different levels are fused to enhance the expression ability of the features of the target to be detected, which will improve the model's ability to extract underwater target features, thereby improving the accuracy of underwater target detection.

[0053] See also Figure 1 , the underwater target detection network constructed by the embodiment of the present invention is a series of dual encoder-decoder structures. The encoder obtains multiple feature maps of different scales to perform multi-scale feature cross-fusion to fully extract the target features in the input image. The decoder is responsible for gradually restoring and enhancing the feature maps of small-scale multi-channels. Finally, the feature maps of three different sizes are sent to the respective detection heads to achieve accurate positioning of underwater targets. Specifically, the embodiment of the present invention first designs a lightweight encoder structure for efficient feature extraction as a backbone network to extract features from the input image, then obtains feature maps of different scales and processes them through the designed multi-scale feature cross-fusion module to enhance the deep features; in the decoder stage, the feature maps retained by the downsampling operation are continued to be spliced ​​with the feature maps of the corresponding levels in the decoding process to enhance feature propagation and compensate for the loss caused by the downsampling process in the encoder stage; finally, after another encoding and decoding process, multi-level feature splicing is performed to enhance the feature expression ability, and the features of three different levels obtained in the last decoding process are sent to the corresponding detection head structure and fused and output to obtain the underwater target detection result, thereby finally achieving accurate detection of underwater targets.

[0054] As a possible implementation manner, the implementation steps of the underwater target detection method based on multi-scale feature cross fusion provided in the embodiment of the present invention include:

[0055] Step 1: Underwater target image preprocessing.

[0056] The underwater target images used in the embodiment of the present invention are 877 sample images obtained after processing (such as image deduplication and video frame extraction) the images and video files downloaded from public websites (such as Youtube, Google, etc.). This dataset contains five types of underwater targets: frogmen, non-autonomous remote-controlled submersibles, autonomous underwater submersibles, submarines, and torpedoes. The images are annotated using the CVAT annotation tool and an annotation file in the YOLO format is obtained. Due to the small number of sample images, in order to ensure the stability of the underwater target detection network training constructed by the embodiment of the present invention and the reliability of the detection results, the original dataset is subjected to data enhancement operations such as horizontal flipping, random cropping, and vertical flipping, and the original dataset is expanded to 3508 images. Random sampling is performed at a ratio of 50%, and finally 1754 images are selected as the new dataset. The sample and label distribution of the dataset are shown in the figure. Figure 2 By preprocessing the downloaded original data set, image data that can be used for the underwater target detection network constructed by the embodiment of the present invention is obtained.

[0057] Step 2: Build an underwater target detection network based on multi-scale feature cross-fusion.

[0058] In the embodiment of the present invention, the underwater target detection network based on multi-scale feature cross fusion is constructed as follows Figure 3 As shown, it includes two serially connected encoder-decoder structures and a detection head. The two serially connected encoder-decoder structures are used to perform image feature extraction and multi-scale feature cross-fusion. The detection head is used to output underwater target detection results: including: the object category of the target frame, the confidence of the target frame and the position information of the target frame.

[0059] For underwater target images, in order to achieve efficient extraction of underwater target features and improve the model's perception of blurred and small targets in the water, the embodiment of the present invention designs and implements a lightweight backbone network as the first encoder for underwater target feature extraction. The backbone network includes a lightweight dense bottleneck architecture (LDBA), such as Figure 4 In LDBA, an Efficient Multi-Scale Attention (EMA) module is used to further improve the network’s attention to underwater blurred targets and small targets.

[0060] In order to better utilize the deep feature map information of small-scale multi-channel to improve the network's detection ability for underwater blurred targets and small targets, the embodiment of the present invention designs a multi-scale cross-fusion attention (MSCFA) module (such as Figure 5 As shown in Figure 2, the multi-level feature map is fused with the cross self-attention idea to further enhance the deep features. In addition, a fast feature enhancement (FFE) module is also included in the first decoding and the second encoding-decoding process, such as Figure 5 This module has borrowed the idea of ​​the Transformer structure and has a very small number of parameters, which can ensure that the feature expression is enhanced while paying a small computational cost.

[0061] In the second encoder-decoder stage, the deepest feature map is still subjected to multi-level feature fusion enhancement processing through the MSCFA module. Finally, in order to enable the network to simultaneously detect small, medium, and large underwater targets, the three-level feature maps of different scales generated in the decoder stage are respectively sent to three detection heads for different scales to obtain three outputs, and then the three outputs are fused to produce the final detection result, achieving accurate detection of underwater targets.

[0062] Step 3: Train the underwater target detection network and optimize the network parameters.

[0063] In order to reduce the overfitting phenomenon of the network during training, the embodiment of the present invention adopts an early stopping strategy, which stops the training when the network has converged by observing the changes in indicators in a certain number of training rounds.

[0064] The constructed underwater target detection network continuously iterates and optimizes the network parameters through back propagation, and the trained network parameters and network model structure are saved as HDF5 format files. Thus, a trained neural network model based on multi-scale feature cross fusion is obtained.

[0065] Step 4: Input the underwater target image to be detected into the trained underwater target detection network to obtain the detection result.

[0066] The test set of underwater target images is input into the pre-trained underwater target detection network for testing experiments to obtain a complete and high-precision underwater target detection result map.

[0067] In addition, the embodiment of the present invention also provides a verification process for interpretability analysis and generalization experiments on the constructed underwater target detection network based on multi-scale feature cross-fusion. In order to better verify the accuracy and generalization of underwater scene detection in the underwater target detection network based on multi-scale feature cross-fusion constructed in the embodiment of the present invention, the attribution-based interpretability method LayerCAM is used to output the interpretable heat map of the underwater target dataset detection results, analyze the attention of the underwater target detection network to the underwater targets, and verify the generalization of the model on two public datasets, URPC2020 and RUOD.

[0068] In one embodiment, the specific construction process of the underwater target detection network based on multi-scale feature cross-fusion includes:

[0069] (1) Lightweight dense bottleneck architecture design.

[0070] For underwater target images, in order to achieve efficient feature extraction of underwater targets and enhance the feature expression of underwater targets in the model, an embodiment of the present invention designs and implements a lightweight dense bottleneck architecture LDBA to constitute the first encoder, which performs efficient feature extraction under the condition of introducing a small number of parameters.

[0071] The LDBA module mainly adopts the idea of ​​segmentation and combination, which divides the original feature map into two parts according to the channel, processes the two parts of the feature map respectively, and fuses the processing results of different stages, thereby ensuring full mining of features while reducing the number of parameters, and thus obtaining diversified image features. The specific structure of LDBA is as follows: Figure 3 shown.

[0072] See also Figure 3 , the specific processing process of the LDBA module is:

[0073] S1, for the input C×H×W feature map a (C is the number of channels, H and W are the height and width of the feature map respectively), it will first pass through a convolution layer with a convolution kernel of 3×3 to compress the channels of the original input feature map, and produce an intermediate result a1, whose size is C / 2×H×W, in order to reduce the number of parameters in the subsequent processing. The processing process of this convolution layer is shown in formula (1):

[0074] f(a)=SiLU(BatchNorm(COnvolution(a))) (1)

[0075] Among them, Convolution() represents the convolution operation, BatchNorm() represents the batch normalization operation, and SiLU() represents the activation function;

[0076] S2. For a1, the Split operation is used to evenly divide it into a2 and a3 according to the number of channels, and their sizes are both C / 4×H×W. The purpose is to further reduce the amount of parameters introduced while ensuring the diversified processing of the feature map.

[0077] S3, for a2, through an efficient multi-scale attention EMA module (its structure is as follows Figure 4 The module allows learning effective channel descriptions without reducing the channel dimension in the convolution operation, establishes short-term and long-term dependencies between multi-scale cross-space features, and generates better pixel-level attention for high-level feature maps. The feature map after processing by the EMA module is a ema , the size is C / 4×H×W. The processing of the EMA module is as follows:

[0078] The input is first grouped by the Groups operation, and then passes through two branches with different processing. Branch 1 performs one-dimensional horizontal and vertical global pooling operations and then performs feature fusion, while branch 2 performs feature extraction through a convolution layer with a convolution kernel size of 3×3. The features output by the two branches are modulated by the sigmoid function and normalization operation, and then a cross-domain fusion operation is performed. Finally, the enhanced features are obtained through the sigmoid function adjustment.

[0079] S4. For the processing of a3, the embodiment of the present invention is based on the idea of ​​dense blocks, so that different feature maps that have been processed at multiple levels can be fused, enriching the gradient information in the back propagation process. First, a3 will pass through a convolution layer with a convolution kernel of 1×1 and a convolution layer with a convolution kernel of 3×3, respectively, to obtain intermediate results a4 and a5 of size C / 4×H×W. Then a3 and a5 are added pixel by pixel through a residual connection, and then processed through a convolution layer with a convolution kernel of 1×1, and the result is added pixel by pixel with a4 to obtain an intermediate result a6 of size C / 4×H×W.

[0080] S5, in order to avoid feature loss caused by channel reduction and fuse feature information processed at multiple levels, a1, a ema , a3, a4, a5 and a6 are concatenated by channel to obtain a feature map of size 7C / 4×H×W.

[0081] S6: For the feature map obtained in S5, a convolution layer with a convolution kernel of 3×3 is passed to finally obtain the LDBA processing result a ldba , the size is C×H×W, that is, the dimension of the input feature map of the LDBA module is consistent with the output feature map.

[0082] (2) Multi-scale cross-fusion attention structure design.

[0083] Due to the accumulation of small underwater targets and blurred targets, it is difficult to fully mine the positions and pixel information of small and blurred targets using only the backbone network for underwater target feature extraction. Therefore, the embodiment of the present invention designs a multi-scale cross-fusion attention MSCFA module (such as Figure 5 As shown in the figure), it can realize the precise positioning of small underwater targets and blurred targets.

[0084] The MSCFA module receives features of three scales (hereinafter referred to as small, medium, and large scales), and obtains three different scales of Query (Q) based on the calculation idea of ​​the self-attention mechanism. When the feature map of one scale is used as Q, the feature maps of the other two scales will be scaled to the same size as the current Q scale and used as Key (K) and Value (V) respectively. Multi-channel self-attention calculations are performed between Q, K, and V. By taking turns to participate in the calculation of Q, the features of the three scales are associated with each other, and the three multi-head self-attention calculation results are scaled and fused to obtain the final small-scale feature map output, so as to enhance the small-scale features and improve the expressiveness of the features in the network.

[0085] See also Figure 5 ,The specific processing process of the MSCFA module is as follows:

[0086] S1. Input three feature maps of different scales, namely C×H×W, 2C×H / 2×W / 2 and 4C×H / 4×W / 4, representing the features of large, medium and small scales.

[0087] S2. Select 4C×H / 4×W / 4 as Q1, and pass the feature maps of 2C×H / 2×W / 2 and C×H×W through the downsampling convolution layer with a convolution kernel of 3×3 once and twice respectively to obtain K1 and V1 of size C×H×W.

[0088] S3. Select 2C×H / 2×W / 2 as Q2, perform an upsampling convolution with a convolution kernel of 3×3 on the feature map of 4C×H / 4×W / 4 to obtain K2, and perform a downsampling convolution with a convolution kernel of 3×3 on the feature map of C×H×W to obtain V2 of size 2C×H / 2×W / 2.

[0089] S4. Select C×H×W as Q3, and pass the feature maps of 2C×H / 2×W / 2 and 4C×H / 4×W / 4 through the upsampling convolution layer with a convolution kernel of 3×3 once and twice respectively to obtain K3 and V3 of size C×H×W.

[0090] S5, calculate Q, K and V obtained by S2, S3 and S4 through the channel multi-head self-attention mechanism to obtain intermediate feature maps of size 4C×H / 4×W / 4, 2C×H / 2×W / 2 and C×H×W respectively. The calculation process of the channel multi-head self-attention mechanism is shown in formula (2):

[0091]

[0092] Among them, MultiHead() represents the channel multi-head self-attention mechanism, Concat() represents the concatenation operation, the subscript i is used to distinguish features of different scales, the subscript j represents the head number, and W 0 represents the weights for the linear transformation pair after the concatenation operation, W j Q , W j K , W j V are the weights of query Q, key K and value V respectively; d k Represents the dimensions of the column vectors in matrices Q and K.

[0093] S6, for the C×H×W feature map obtained in S5, after a downsampling convolution layer with a convolution kernel of 3×3, a feature map of size 2C×H / 2×W / 2 is obtained, and it is added pixel by pixel with the feature map of size 2C×H / 2×W / 2 obtained in S5. In addition, a trainable adjustment factor β will be introduced to make the feature fusion process adaptive and dynamic, and the proportion of feature maps of each scale participating in the fusion can be adjusted during the model training process to achieve adaptive fusion of features.

[0094] S7. For the feature map of size 2C×H / 2×W / 2 obtained in S6, a downsampling convolution layer with a convolution kernel of 3×3 is obtained to obtain a feature map of size 4C×H / 4×W / 4, and a trainable adjustment factor α is introduced to dynamically fuse it with the feature map of size 4C×H / 4×W / 4 obtained in S5.

[0095] S8. For the feature map obtained in S7, a convolution layer with a convolution kernel of 3×3 is passed to obtain the final processing result of the MSCFA module, with a size of 4C×H / 4×W / 4.

[0096] That is, in the MSCFA module, the channel multi-head self-attention mechanism can be used to obtain the calculation results of the channel multi-head self-attention mechanism at three scales, namely, small, medium and large. The calculation result of the large scale is adjusted to be consistent with the calculation result of the medium scale by a downsampling convolution layer with a convolution kernel of 3×3, and then the adjustment factor β is used as the weight of the output feature map of the downsampling convolution layer, and 1-β is used as the weight of the medium-scale calculation result for weighted fusion to obtain the first fusion calculation result; the first fusion calculation result is adjusted to be consistent with the calculation result of the small scale by a downsampling convolution layer with a convolution kernel of 3×3, and then the adjustment factor α is used as the weight of the output feature map of the downsampling convolution layer, and 1-α is used as the weight of the small-scale calculation result for weighted fusion to obtain the second fusion calculation result, and finally the second fusion calculation result is sent to a convolution layer with a convolution kernel of 3×3 to obtain the output feature map of the MSCFA module. The adjustment factors β and α are trainable parameters, and their initial values ​​are random values. They are adaptively updated during the training process of the network, that is, they are updated together as network parameters.

[0097] (3) Rapid feature-enhanced structural design.

[0098] The fast feature enhancement FFE module designed in the embodiment of the present invention is as follows: Figure 6 As shown in the figure, the network depth can be increased without introducing too many parameters, which improves the encoder process's ability to perceive features and the decoder process's ability to restore pixels and locate underwater targets. This module draws on the idea of ​​the Transformer Layer and introduces two residual structures to prevent the gradient vanishing problem in back propagation and protect the transmission of input information. It also introduces the PConv operation (partial convolution operation) to reduce the number of parameters while retaining most of the feature information compared to the traditional Conv operation.

[0099] See also Figure 6 ,The specific process of the FFE module is as follows:

[0100] S1. For the feature map with an input size of C×H×W, it is processed by a convolution layer with a convolution kernel size of 3×3 to obtain the processing result b1, and the shape remains unchanged.

[0101] S2, split b1 evenly according to the number of channels to obtain b2 and b3 of size C / 2×H×W. The purpose of this step is to reduce the number of parameters required in the process of processing feature maps.

[0102] S3, for b2, it is processed by PConv operation with a convolution kernel size of 3×3 and convolution layer processing with a convolution kernel size of 1×1, and the intermediate processing result is added pixel by pixel with the result before processing to obtain a feature map b4 of size C / 2×H×W. Convolution kernels of different sizes will enrich the receptive field during the processing process, enable the network to obtain information from different receptive fields, and enhance the model's perception of underwater targets. Since in traditional convolution operations, part of the feature map after convolution processing is redundant and does not affect the ability of the model. Therefore, PConv performs partial convolution on the original feature map, and only selects a part of the feature map that is continuous on the channel for convolution calculation, so that efficient convolution is achieved and the required amount of calculation is reduced. Preferably, the feature map processed by PConv used in this embodiment is the continuous part of the first 1 / 2 channel of the original feature map, that is, the original feature map size is C×H×W, and the feature map size actually required to be calculated is C / 2×H×W.

[0103] S4. For b4 obtained in S3, it is first subjected to batch normalization processing through a Batch Norm layer, and then passed through a convolution layer with a convolution kernel size of 3×3. The processed result is added pixel by pixel with the result before processing to obtain a feature map b5 of size C / 2×H×W.

[0104] S5. In order to reduce the feature loss in the processing process, b3 and b5 are concat-concatenated to obtain a feature map b6 of size C×H×W.

[0105] S6, pass b6 obtained in S5 through a convolution layer with a convolution kernel of 3×3 to obtain the final processing result b of the FFE module ffe (Output feature map of the FFE module), the size is C×H×W.

[0106] (4) Build a neural network model based on cross-fusion of multi-scale features.

[0107] Based on the lightweight dense bottleneck architecture LDBA module, the multi-scale cross-fusion attention MSCFA module and the fast feature enhancement FFE module, the underwater target detection network based on multi-scale feature cross-fusion of this embodiment is constructed. Figure 3As shown in the figure, it adopts a dual encoder-decoder structure connected in series. For the first encoder-decoder structure, the encoder consists of a common downsampling convolution layer (Down Conv) and an LDBA module, while the decoder consists of a common upsampling convolution layer (Up Conv) and an FFE module. The end of the encoder is connected to an MSFCA module for enhancing the small-scale feature map. For the input 3×H×W original RGB image, the first encoder is used as the feature extraction backbone network to reduce the scale of the feature map and increase the number of channels through downsampling convolution, and an LDBA module is connected after each downsampling convolution to perform diversified processing on the reduced feature map, fully explore the image features, and complete efficient feature extraction with only a small number of parameters. Finally, the first encoder will output a feature map of size 16C×H / 32×W / 32 for subsequent processing. For the MSFCA module, the feature maps of scales 16C×H / 32×W / 32, 8C×H / 16×W / 16, and 4C×H / 8×W / 8 obtained by the encoder are input, and multi-scale feature cross-fusion is performed to achieve small-scale feature enhancement and improve the model's perception of small underwater targets and blurred targets. For the decoder stage, an FFE module is connected after each upsampling convolution layer. While restoring the feature map, the multi-level feature maps obtained in the encoder stage are reused and feature propagation is promoted through skip connection operations. At the same time, the FFE module can ensure that the number of network layers is deepened while introducing a small number of parameters, and each level of feature enhancement is achieved with a small amount of computational overhead. In addition, in order to reduce the complexity of the entire model, the decoder does not restore the feature map to the original input size, but only restores it to 4C×H / 8×W / 8, which is consistent with the shape of the maximum scale feature map finally sent to the detection head.

[0108] For the second encoder-decoder structure, the encoder and decoder are composed of a common upsampling convolution layer and a common downsampling convolution layer and an FFE module. The reason why the LDBA module is not continued to be used in the encoder is that the existence of the second encoder is to promote the fusion of multi-level features and avoid the loss and forgetting of features, rather than to extract features. Therefore, this embodiment uses an FFE module with fewer parameters and faster operation in the second encoder-decoder stage. At the same time, an MSFCA module will be connected to the end of the second encoder to enhance small-scale features.

[0109] Finally, the second encoder will output feature maps of three scales, namely 16C×H / 32×W / 32, 8C×H / 16×W / 16, and 4C×H / 8×W / 8. These three scale feature maps will be sent to three corresponding detection heads (DetctHead) respectively, and the output results of the three detection heads will be fused to finally achieve accurate detection of underwater targets.

[0110] When deep learning is performed on the underwater target detection network based on multi-scale feature cross-fusion built in this example to train its network parameters, in order to reduce the overfitting phenomenon of the network during training, this embodiment adopts an early stopping strategy, and stops training the model when it has converged by observing the changes in indicators in a certain number of training rounds. In this embodiment, the deep learning framework used is Pytorch 2.0, the experimental platform is Windows 10, the server is configured as an NVIDIA GeForce RTX 3090, the optimizer used for training is AdamW, the learning rate is set to 0.0001, and the batchsize is set to 8. The pre-trained weights are not loaded during the training process. The network model continuously iterates and optimizes the parameters and model through back propagation. After the training is completed, the network parameters and model are saved as HDF5 format files. Thus, a trained underwater target detection network based on multi-scale feature cross-fusion is obtained.

[0111] The test set of underwater target images is input into the trained underwater target detection network for testing experiments. The detection results are as follows: Figure 7 As shown, the first line is the original image, and the second line is the detection image output by the underwater target detection network based on multi-scale feature cross fusion in this embodiment. Figure 7 It can be seen from the experimental results shown that the targets in each underwater image are accurately detected with good detection effect and high precision.

[0112] In order to further verify the detection performance of the underwater target detection network based on multi-scale feature cross-fusion of this embodiment, the attribution-based interpretability method LayerCAM is used to perform interpretability analysis on the detection results of the underwater target detection network of this embodiment: the test set of underwater target images is input into the trained underwater target detection network, and the interpretability heat map is output in combination with the LayerCAM method, as shown in FIG. Figure 8 As shown in the figure, the depth of color reveals the attention of the underwater target detection network model to the target in the underwater image. It can be seen that the model can pay high attention to the target, which is an important factor for the model to accurately detect underwater targets.

[0113] In addition, in order to further verify the detection performance of the underwater target detection network based on multi-scale feature cross-fusion in this embodiment, the generalization and effectiveness of the constructed network are verified under different data sets: This embodiment uses the public network data sets URPC2020 and RUOD to conduct generalization experiments on the underwater target detection network based on multi-scale feature cross-fusion. The training is performed on the training sets of the two data sets respectively, and the output detection results are as follows: Fig. 9 and Fig.10As shown, it can be seen that the underwater target detection network based on multi-scale feature cross-fusion built in this embodiment has a good effect in detecting various types of targets in the water.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0115] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An underwater target detection method based on multi-scale feature cross fusion, characterized in that: The following steps are involved: Step 1, input underwater target images and preprocess them to obtain training data sets; Step 2: Build an underwater target detection network based on multi-scale feature cross-fusion; The underwater target detection network adopts a series of dual encoder-decoder structures. For the first encoder-decoder structure, a lightweight encoder structure is used as the backbone network to extract multi-scale features from the input image. A multi-scale cross-fusion attention MSCFA module is connected to the end of the encoder to enhance the small-scale feature map. The decoder is used to gradually restore and enhance the small-scale multi-channel feature maps, and to splice the feature maps retained by the downsampling operation in the encoding process with the feature maps of the corresponding levels in the decoding process; For the second encoder-decoder structure, a MSCFA module is connected between the encoder and the decoder, the encoder is used to encode the output feature map of the first encoder-decoder structure, the decoder is used to gradually restore and enhance the feature map of the output feature map of the MSCFA module, and the encoder of the second encoder-decoder structure splices the feature map retained by the upsampling operation with the feature map of the corresponding level in the decoding process of the first encoder-decoder structure; The feature maps of three different levels obtained in the last decoding process are sent to the corresponding detection head structure, and the outputs of the three detection head structures are fused to obtain the detection results of underwater targets; Step 3: training the underwater target detection network based on multi-scale feature cross fusion based on the training data set, and obtaining the trained underwater target detection network when the preset training end condition is met; Step 4: Input the underwater target image to be detected into the trained underwater target detection network to obtain the detection result.

2. The method according to claim 1, characterized in that In the first encoder-decoder structure, each encoding layer of the encoder includes a downsampling convolution layer and a lightweight dense bottleneck architecture LDBA module in sequence; each decoding layer of the decoder includes an upsampling convolution layer and a fast feature enhancement FFE module in sequence; In the second encoder-decoder structure, each encoding layer of the encoder includes a downsampling convolution layer and a fast feature enhancement FFE module in sequence; each decoding layer of the decoder includes an upsampling convolution layer and a fast feature enhancement FFE module in sequence; For the first encoder-decoder structure, define represents the number of coding layers of the first encoder, Indicates the number of decoding layers of the first decoder. The encoding layers in the first encoder-decoder structure are defined as the first encoding layer 1 to the first encoding layer according to the forward propagation direction. And define the decoding layers as first decoding layer 1 to first decoding layer Among them, the feature map of the first decoding layer i is the first encoding layer The output feature map of , where i = 1, 2, ..., and Greater than For the second encoder-decoder structure, define represents the number of coding layers of the second encoder, Indicates the number of decoding layers of the second decoder. The encoding layers in the second encoder-decoder structure are defined as the second encoding layer 1 to the second encoding layer according to the forward propagation direction. And define the decoding layers as second decoding layer 1 to second decoding layer Among them, the feature map of the second encoding layer j is the first decoding layer The output feature map of , where j = 1, 2, ..., and when When the second coding layer The concatenated feature map is the input feature map of the first decoding layer; The feature map of the second decoding layer p is the second encoding layer The output feature map of , where p = 1, 2, ..., and when When , the second decoding layer The concatenated feature map is the input feature map of the first second encoding layer; Number of decoding layers for the second decoder when When , the three feature maps of different levels obtained in the last decoding process are: the output feature map of the MSCFA module in the second encoder-decoder structure, the output feature map of the two decoding layers of the second decoder; When , the feature maps of three different levels obtained in the last decoding process are the output feature maps of the last three decoding layers of the second decoder.

3. The method according to claim 2, characterized in that The number of coding layers N of the first encoder E 1 Set to 5.

4. The method according to claim 2, characterized in that The structure of the MSCFA module is set up as: The input of the MSCFA module is feature maps of three different scales. The MSCFA module includes three parallel branches. Each branch obtains the intermediate fusion feature map of each scale through the channel multi-head self-attention mechanism. When executing the channel multi-head self-attention mechanism calculation, a feature map of one of the three scales is selected as the query Q, and the feature maps of the other two scales are scaled to the same size as the current Q scale and used as the key K and value V respectively. The intermediate fusion feature map of the current branch is obtained based on the multi-head channel self-attention calculation between Q, K, and V. Thus, the intermediate fusion feature maps of three scales are obtained: small-scale intermediate fusion feature map, medium-scale intermediate fusion feature map, and large-scale intermediate fusion feature map. The large-scale intermediate fusion feature map is subjected to a downsampling convolution layer with a convolution kernel of 3×3, and then weighted fused with the medium-scale intermediate fusion feature map with a weight of 1-β using the adjustment factor β as the weight to obtain the first weighted fusion result; The first weighted fusion result is weightedly fused with the small-scale intermediate fusion feature map with a weight of 1-α after passing through a downsampling convolution layer with a convolution kernel of 3×3, using the adjustment factor α as the weight, to obtain the second weighted fusion result; The second weighted fusion result is passed through a convolution layer with a convolution kernel of 3×3 to obtain the output feature map of the MSCFA module.

5. The method according to claim 2, characterized in that The structure of the LDBA module is set up as: The main branch of the LDBA module includes: the first convolution layer, the slicing layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the splicing layer and the fifth convolution layer; The splitting layer is used to evenly divide the input feature map into two sub-feature maps according to the number of channels. One sub-feature map is sent to the second convolutional layer, and the other sub-feature map is sent to the efficient multi-scale attention EMA module. The output feature map of the third convolutional layer and the input feature map of the second convolutional layer are added pixel by pixel through a residual connection and then sent to the fourth convolutional layer; The output feature map of the fourth convolutional layer and the output feature map of the second convolutional layer are added pixel by pixel through residual connection and then sent to the splicing layer; and the input of the splicing layer also includes the output of the EMA module and the output feature maps of the first to fourth convolutional layers.

6. The method according to claim 2, characterized in that The FFE module includes: the first convolution layer, the slicing layer, the partial convolution layer, the second convolution layer, the batch normalization layer, the concatenation layer, the third convolution layer and the fourth convolution layer in sequence; Among them, the splitting layer is used to evenly divide the input feature map into two sub-feature maps b2 and b3 according to the number of channels. The sub-feature map b2 is sent to the partial convolution layer, and the sub-feature map b3 is sent to the splicing layer; The sub-feature map b2 and the output feature map of the second convolutional layer are added pixel by pixel through a residual connection and then sent to the batch normalization layer; and the input feature map of the batch normalization layer and the output feature map of the second convolutional layer are added pixel by pixel through a residual connection and then sent to the concatenation layer.

7. The method according to claim 6, characterized in that The feature map processed by the partial convolution layer is the continuous part of the 1 / 2 channel of the input feature map of the partial convolution layer.

8. The method according to any one of claims 1 to 7, characterized in that In step 3, the training end condition adopts the early stopping strategy.

Images