An underwater target detection method based on cross-scale attention

By using the cross-scale attention module and top-down feature fusion path in underwater target detection, the problems of image degradation and scale transformation in underwater target detection are solved, the detection effect is improved, and missed and missed detection is reduced , achieving more efficient underwater target detection.

CN118537714BActive Publication Date: 2024-12-03JIANGSU OCEAN UNIV +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410422399.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-09
Publication Date
2024-12-03
Estimated Expiration
2044-04-09

AI Technical Summary

Technical Problem

Underwater target detection method is affected by the target distance, light source distribution and light absorption and attenuation in a natural environment, resulting in serious image degradation, making it difficult to extract the characteristics of underwater targets, and there are problems of occlusion and overlap, poor detection effect, and leakage Inspection and false inspection.

Method used

Using the underwater object detection method based on cross-scale attention, multi-scale feature extraction module and cross-scale attention module are used to extract and enhance multi-scale feature maps, combined with top-down fusion paths, to generate features with stronger representation capabilities. Figure, used for object detection.

Benefits of technology

The scale perception ability of underwater target detection is improved, the detection effect of targets of different sizes is enhanced, missed and missed detection is reduced, and more efficient underwater target detection is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118537714B_ABST
    Figure CN118537714B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of underwater target detection, and discloses an underwater target detection method based on cross-scale attention. A multi-scale feature map is obtained through a backbone network, and the number of channels of the multi-scale feature map is unified through lateral connection. Then, the intra-layer correlation and inter-layer correlation of the multi-scale feature map are captured through a cross-scale attention module, and then the feature enhancement is completed in conjunction with a top-down feature fusion path, so that the model can better focus on the target. The present invention adapts to the multi-scale transformation of underwater targets through the multi-scale features extracted by the backbone network, and improves the detection effect of the network on underwater targets of different sizes; a cross-scale attention module is proposed to explicitly model the long- and short-range dependencies between feature channels of different scales, enhance multi-scale features, and improve the feature extraction capability of the network; the present invention can improve the scale perception capability of the detector in an underwater environment, and the cross-scale attention mechanism can be combined with any FPN network to achieve efficient underwater target detection tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target detection, and in particular to an underwater target detection method based on cross-scale attention. Background Art

[0002] Underwater target detection methods use data obtained from sensors such as sonar, radar, and cameras to identify, locate, and track underwater targets. Underwater target detection has low cost and strong environmental adaptability, which is helpful for exploring marine resources, monitoring marine ecological environment, and the development of underwater robots and other technologies.

[0003] Object detection methods based on deep learning have achieved good results on large open source datasets such as PASCAL VOC and MS COCO. However, compared with imaging methods in natural environments, underwater optical imaging is affected by factors such as target distance, spectral distribution of light sources, and absorption and attenuation of light of different wavelengths. Underwater images are severely degraded, which greatly increases the difficulty of underwater target detection. Object detection methods cannot fully extract the features of underwater targets. When there are occlusions and overlaps between targets of different sizes, the detection effect is poor, and there are problems of missed detection and false detection. Therefore, an underwater target detection method based on cross-scale attention is provided. Summary of the invention

[0004] The purpose of the present invention is to provide an underwater target detection method based on cross-scale attention in view of the defects of the prior art, so as to solve the problems raised by the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: an underwater target detection method based on cross-scale attention, the specific steps are as follows:

[0006] S1: After the input underwater image x is extracted by the backbone network of the multi-scale feature extraction module, a multi-scale feature map {C2, C3, C4, C5} is obtained;

[0007] S2: The multi-scale feature maps {C2, C3, C4, C5} with the same number of channels are input into the cross-scale attention module. The cross-scale attention module compresses the spatial dimension of each scale feature map to obtain the channel feature vector, connects all channel vectors to obtain the global context information pool; based on SENet, the global channel dependency is captured to generate the attention map; the attention map is weighted with the global context information pool to complete the intra-layer and inter-layer feature enhancement of the multi-scale features to obtain the multi-scale feature map {N2, N3, N4, N5};

[0008] S3: The multi-scale feature maps {N2, N3, N4, N5} are fused through a top-down path to obtain the fused multi-scale feature maps {P2, P3, P4, P5};

[0009] S4: Input the multi-scale feature map {P2, P3, P4, P5} into the detection head network to obtain the confidence, classification information and location information of the target.

[0010] As a preferred technical solution of the present invention, the backbone network of the multi-scale feature extraction module in step S1 is ResNet. ResNet has a total of 5 stages, each stage is composed of several Residual Blocks, and Residual Block is composed of 1×1 convolution and 3×3 convolution. After each stage, the resolution of the feature map decreases by 2 times, and the number of channels increases by 2 times. {stage2, stage3, stage4, stage5} corresponds to the multi-scale feature map {C2, C3, C4, C5}.

[0011] As a preferred technical solution of the present invention, the specific construction process of the cross-scale attention module in step S2 is:

[0012] S21: Input feature map The last block output of each stage in ResNet uses a 1×1 convolution kernel to modify the number of channels C to 256;

[0013] S22: Perform global average pooling on the feature maps of each scale and compress the spatial dimension to obtain context information e i ∈R C ;

[0014]

[0015] S23: Connect the channel features of each scale end to end in the order of stages to obtain a one-dimensional vector s∈R C×i ;

[0016] S24: Use the F function to capture the dependencies between channel features and use two fully connected layers to fully learn the nonlinear relationship between features a∈R C×i ;

[0017] a=F(s)=σ(W2δ(W1s)) (2)

[0018] Where δ represents the ReLU function and σ represents the Sigmoid activation function, which is used to generate the attention map;

[0019] S25: Cut the obtained cross-scale channel attention map into i separate attention maps a i ∈R C , and multiply them by the input feature maps of each stage to obtain the calibrated feature maps

[0020] Ni =a i ·C i (3).

[0021] As a preferred technical solution of the present invention, the specific construction process of the top-down fusion path in step S3 is:

[0022] P i =conv 3×3 (N i+1 ↑ 2× +N i ) (4)

[0023] N5 is P5. The P5 layer is concat with the N4 layer through the upsampling module, and then passes through a 3×3 convolution layer to eliminate the aliasing effect to obtain the P4 layer. The P4 layer repeats the above operation to obtain a multi-scale feature map {P2, P3, P4, P5}.

[0024] After the above steps, a multi-scale feature map with stronger representation ability is obtained for subsequent multi-scale prediction.

[0025] As a preferred technical solution of the present invention, the confidence in step S4 represents the probability score of the target category contained in the bounding box; the classification loss represents the difference between the predicted category and the true category, using the cross entropy loss function; the regression loss represents the difference between the predicted position and the true position, using the Smooth L1 loss function, and the final loss loss_total of the entire network is expressed as:

[0026] loss_total = L cls +λ[y i ≥1]L reg (5)

[0027]

[0028]

[0029] Among them, L cls and L reg They represent classification loss and regression loss respectively, and λ is the weight parameter set to 10.

[0030] A system for implementing an underwater target detection method based on cross-scale attention includes a multi-scale feature extraction module, a cross-scale attention module, a multi-scale feature fusion module and a detection head.

[0031] As a preferred technical solution of the present invention, the feature extraction module is based on a backbone network, takes the underwater image as the input of the module, extracts feature maps with different texture information and semantic information at each stage, and thus obtains a multi-scale feature map.

[0032] As a preferred technical solution of the present invention, the cross-scale attention module is used to guide intra-layer and inter-layer feature enhancement of multi-scale features, and simultaneously enhance all multi-scale feature maps based on channel attention.

[0033] As a preferred technical solution of the present invention, the multi-scale feature fusion module is used to supplement the information of the low-level feature map, integrate the rich semantic information of the top-level feature map into the underlying features, and obtain a feature pyramid with stronger expressive ability.

[0034] Compared with the existing technologies, the beneficial effects of the present invention are as follows: the present invention adapts to the multi-scale transformation of underwater targets through the multi-scale features extracted by the backbone network, and improves the detection effect of the network on underwater targets of different sizes; a cross-scale attention module is proposed to explicitly model the long- and short-range dependencies between feature channels of different scales, enhance the multi-scale features, and improve the feature extraction capability of the network; experiments have proved that the present invention can improve the scale perception capability of the detector in the underwater environment, and the cross-scale attention mechanism can be combined with any FPN network to achieve efficient underwater target detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the detection method of the present invention;

[0036] Figure 2 It is a structural diagram of the underwater target detection method based on cross-scale attention of the present invention;

[0037] Figure 3 It is the backbone network ResNet structure diagram of the present invention;

[0038] Figure 4 is a structural diagram of the cross-scale attention module of the present invention;

[0039] Figure 5 It is a partial image of the Brackish dataset of the present invention;

[0040] (a) Fish; (b) Crab; (c) Small fish;

[0041] Figure 6 This is a comparison chart of the method of the present invention and two other advanced underwater target detection methods on the Brackish dataset. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0043] Example: Figure 1As shown, the present invention provides an underwater target detection method based on cross-scale attention, which obtains a multi-scale feature map through a backbone network, unifies the number of channels of the multi-scale feature map through lateral connection, and then captures the intra-layer and inter-layer correlations of the multi-scale feature map through a cross-scale attention module, and then cooperates with a top-down feature fusion path to complete feature enhancement, so that the model can better focus on the target. Experiments have shown that the present invention can improve the scale perception ability of the detector in an underwater environment, and the cross-scale attention mechanism can be combined with any FPN network to achieve efficient underwater target detection tasks.

[0044] like Figure 2 As shown in the figure, an underwater target detection method based on cross-scale attention has an overall structure based on the FPN detection method, and is composed of a multi-scale feature extraction module, a cross-scale attention module, a multi-scale feature fusion module and a detection head; the feature extraction module is based on the backbone network, takes the underwater image as the input of the module, extracts feature maps with different texture information and semantic information at each stage, and thus obtains a multi-scale feature map. The cross-scale attention module is used to guide the intra-layer and inter-layer feature enhancement of multi-scale features, and simultaneously enhances all multi-scale feature maps based on channel attention. The multi-scale feature fusion module is used to supplement the information of the low-level feature maps, integrate the rich semantic information of the top-level feature maps into the bottom-level features, and obtain a feature pyramid with stronger expressive power.

[0045] 1. Extract multi-scale features;

[0046] After the input underwater image x is extracted through the backbone network, as shown in Figure 3 As shown, a multi-scale feature map {C2, C3, C4, C5} is obtained. The backbone network described in the multi-scale feature extraction module is ResNet. ResNet has a total of 5 stages, each stage consists of several Residual Blocks, and Residual Block consists of 1×1 convolution and 3×3 convolution. After each stage, the resolution of the feature map decreases by 2 times, and the number of channels increases by 2 times. {stage2, stage3, stage4, stage5} corresponds to the multi-scale feature map {C2, C3, C4, C5}.

[0047] 2. Cross-scale attention module;

[0048] The multi-scale feature maps {C2, C3, C4, C5} with the same number of channels are input into the cross-scale attention module, such as Figure 4As shown in Figure 1, this module compresses the spatial dimension of each scale feature map to obtain a channel feature vector, connects all channel vectors to obtain a global context information pool, and captures global channel dependencies based on SENet to generate an attention map. The attention map is weighted with the global context information pool to complete the intra-layer and inter-layer feature enhancement of multi-scale features to obtain a multi-scale feature map {N2, N3, N4, N5}. The specific construction process of the cross-scale attention module is as follows:

[0049] First, input feature map The last block output of each stage in ResNet uses a 1×1 convolution kernel to modify the number of channels C to 256;

[0050] Then, global average pooling is performed on the feature maps of each scale to compress the spatial dimension and obtain the context information e i ∈R C ;

[0051]

[0052] Secondly, the channel features of each scale are connected end to end in the order of stages to obtain a one-dimensional vector s∈R C×i ;

[0053] Using the F function to capture the dependencies between channel features, the present invention uses two fully connected layers to fully learn the nonlinear relationship a∈R between features. C×i ;

[0054] a=F(s)=σ(W2δ(W1s)) (2)

[0055] Where δ represents the ReLU function and σ represents the Sigmoid activation function, which is used to generate the attention map;

[0056] Finally, the obtained cross-scale channel attention map is cut into i separate attention maps a i ∈R C , and multiply them by the input feature maps of each stage to obtain the calibrated feature maps

[0057] N i =a i ·C i (3).

[0058] 3. Multi-scale feature fusion;

[0059] The multi-scale feature map {N2, N3, N4, N5} is fused through a top-down fusion path to obtain a fused multi-scale feature map {P2, P3, P4, P5}. The specific construction process of the top-down fusion path is as follows:

[0060] P i =conv 3×3 (N i+1 ↑ 2× +N i ) (4)

[0061] N5 is P5. The P5 layer is concat with the N4 layer through the upsampling module, and then passes through a 3×3 convolution layer to eliminate the aliasing effect to obtain the P4 layer. The P4 layer repeats the above operation to obtain a multi-scale feature map {P2, P3, P4, P5}.

[0062] After the above steps, a multi-scale feature map with stronger representation ability can be obtained for subsequent multi-scale prediction.

[0063] 4. Multi-scale feature prediction;

[0064] The multi-scale feature map {P2, P3, P4, P5} is input into the detection head network to obtain the confidence, classification information and location information of the target.

[0065] The specific training methods are as follows:

[0066] The experimental platform for model training uses Ubuntu 18.04, CUDA 11.0, CUDNN 8.0.3, Intel Core i7-11700 CPU, and Nvidia GeForce RTX 3080ti. All experiments are implemented using the Pytorch deep learning framework and the mmdetection toolbox, using the SGD optimizer, with an initial learning rate of 0.005, a batch_size of 4, and a total of 20 epochs. The learning rate is multiplied by 0.1 after the 16th and 19th epochs. Data enhancement includes random flipping, motion blur, and AutoAugmentv1.

[0067] The loss function used in the above training method is:

[0068] The entire training process of the method includes classification loss and regression loss. Classification loss represents the difference between the predicted category and the true category, using the cross entropy loss function; regression loss represents the difference between the predicted position and the true position, using the Smooth L1 loss function. The final loss loss_total of the entire network is expressed as:

[0069] loss_total = L cls +λ[y i ≥1]L reg (5)

[0070]

[0071]

[0072] Among them, L cls and L reg They represent classification loss and regression loss respectively, and λ is the weight parameter, which is set to 10 according to experience.

[0073] Data set selection:

[0074] According to the degradation problems of low contrast, color cast, low illumination, blur and so on in underwater images, experiments were conducted on the real underwater dataset Brackish. Figure 5 As shown. The Brackish dataset was collected in the straits north of Denmark and includes 6 species, namely crabs, fish, small fish, shrimps, jellyfish and starfish. There are 12,444 images with an image size of (960×540) and 35,565 annotations. As part of a marine species monitoring program, these images were taken under different visibility conditions. The dataset shows the impact of floating objects, complex backgrounds, and turbidity changes on target detection. In addition, due to changes in shooting distance and the movement habits of organisms, the scale of the target changes extremely.

[0075] Comparison of experimental results on the Brackish dataset:

[0076] Brackish is a real underwater dataset that contains floating objects, complex backgrounds, turbidity changes, and other factors. 10% of the images in the Brackish dataset are randomly selected as a test set, and the proposed method is compared with two existing feature pyramid networks, such as Figure 6 shown.

[0077] Figure 6 (a) and Figure 6 (b) shows the comparison between FPN and FPN+CSAFPN. In the first column of images, FPN mistakenly identifies the fish as two separate targets due to their distorted posture, while FPN+CSAFPN correctly treats it as a single fish target. The second column of images shows the situation of low visibility and background containing floating debris, where FPN mistakes the debris for small fish, while FPN+CSAFPN effectively avoids this problem. In the third and fourth columns of images, the targets occupy a large part, and although FPN accurately detects the category, it still misinterprets some pixels of some targets as another target.

[0078] Figure 6 (c) and Figure 6(d) shows the comparison between PANet and PANet+CSAFPN. In the first column of images, PANet mistakenly detects them as two targets due to the occlusion of the reference object placed in the water. In the second column of images, PANet misidentifies marine debris as fish due to low visibility. The third column of images contains relatively large fish, and PANet detects the back of the fish as a crab. In the fourth column of images, an unrelated object blocks the camera's view and becomes the target of PANet.

[0079] By comparing the results before and after the method of the present invention is integrated into the existing feature pyramid network, it can be shown that the method of the present invention can stably improve the detection results and ensure that the model is immune to the challenges of underwater background noise and scale changes.

[0080] In order to further demonstrate the performance of the method of the present invention, the present invention also compares the method of the present invention with other 8 relatively advanced underwater target detection methods on the Brackish dataset. The commonly used evaluation indicators of target detection tasks, mAP (meanAverage Precision), AP50, AP75, APS, APM and APL, are used to objectively evaluate the method proposed by the present invention. The larger the above value, the better the underwater target detection effect. The evaluation results are shown in the following table, where the first place of each indicator is marked in bold.

[0081]

[0082]

[0083] From the table, we can see that the method of the present invention achieved 82.5% mAP, 92.8% AP75 and 84.4 APM on the Brackish dataset, which is superior to other conventional target detection methods in the field of underwater target detection.

[0084] The above embodiments only express the implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A method for underwater target detection based on cross-scale attention, characterized by: The specific steps are as follows: S1: After the input underwater image x is extracted by the backbone network of the multi-scale feature extraction module, a multi-scale feature map {C2, C3, C4, C5} is obtained; The backbone network of the multi-scale feature extraction module is ResNet. ResNet has a total of 5 stages. Each stage consists of several ResidualBlocks. ResidualBlock consists of 1×1 convolution and 3×3 convolution. After each stage, the feature map resolution decreases by 2 times, and the number of channels increases by 2 times. {stage2, stage3, stage4, stage5} corresponds to the multi-scale feature maps {C2, C3, C4, C5}. S2: The multi-scale feature maps {C2, C3, C4, C5} with the same number of channels are input into the cross-scale attention module. The cross-scale attention module compresses the spatial dimension of each scale feature map to obtain the channel feature vector, connects all channel vectors to obtain the global context information pool; based on SENet, the global channel dependency is captured to generate the attention map; the attention map is weighted with the global context information pool to complete the intra-layer and inter-layer feature enhancement of the multi-scale features to obtain the multi-scale feature map {N2, N3, N4, N5}; The specific construction process of the cross-scale attention module is as follows: S21: Input feature map The last block output of each stage in ResNet uses a 1×1 convolution kernel to modify the number of channels C to 256; S22: Perform global average pooling on the feature maps of each scale and compress the spatial dimension to obtain context information e i ∈R C ; S23: Connect the channel features of each scale end to end in the order of stages to obtain a one-dimensional vector s = [e2; e3; e4; e5] ∈ R 4C ; S24: Capture the dependencies between channel features and fully learn the nonlinear relationship between features a∈R through two fully connected layers 4C ; a=σ(K2δ(K1s))(2) Among them, K1 and K2 represent two fully connected layers, δ represents the ReLU function, and σ represents the Sigmoid activation function, which is used to generate the attention map; S25: Cut the obtained cross-scale channel attention map into i separate attention maps a i ∈R C , and multiply them by the input feature maps of each stage to obtain the calibrated feature maps N i =a i ·C i i=2,3,4,5(3); S3: The multi-scale feature maps {N2, N3, N4, N5} are fused through a top-down path to obtain the fused multi-scale feature maps {P2, P3, P4, P5}; The specific construction process of the top-down fusion path is as follows: P i =conv 3×3 (P i+1 ↑ 2× +N i )i=2,3,4(4) N5 is P5. The P5 layer is concat with the N4 layer through the upsampling module, and then passes through a 3×3 convolution layer to eliminate the aliasing effect to obtain the P4 layer. The P4 layer repeats the above operation to obtain a multi-scale feature map {P2, P3, P4, P5}. After the above steps, a multi-scale feature map with stronger representation ability is obtained for subsequent multi-scale prediction; S4: Input the multi-scale feature map {P2, P3, P4, P5} into the detection head network to obtain the confidence, classification information and location information of the target; The confidence level indicates the probability score of the target category contained in the bounding box; the classification loss indicates the difference between the predicted category and the true category, using the cross entropy loss function; the regression loss indicates the difference between the predicted position and the true position, using the SmoothL1 loss function. The final loss loss_total of the entire network is expressed as: total_loss=L cls +λ[y i ≥1]L reg (5) Among them, L cls and L reg They represent classification loss and regression loss respectively, and λ is the weight parameter set to 10.

2. A system for implementing the underwater target detection method based on cross-scale attention according to claim 1, characterized in that: It includes a multi-scale feature extraction module, a cross-scale attention module, a multi-scale feature fusion module and a detection head.

3. The system according to claim 2, characterized in that: The feature extraction module is based on a backbone network, takes the underwater image as the input of the module, extracts feature maps with different texture information and semantic information at each stage, and thus obtains a multi-scale feature map.

4. The system according to claim 2, characterized in that: The cross-scale attention module is used to guide intra-layer and inter-layer feature enhancement of multi-scale features, and simultaneously enhances all multi-scale feature maps based on channel attention.

5. The system according to claim 2, characterized in that: The multi-scale feature fusion module is used to supplement the information of the low-level feature map, integrate the rich semantic information of the top-level feature map into the underlying features, and obtain a feature pyramid with stronger expressive ability.

Citation Information

Patent Citations

  • Multi-scale underwater fish school detection method based on attention module

    CN114170497A

  • Underwater target detection method based on attention fusion

    CN114782798A

  • Underwater target tracking method and system based on adaptive cross attention

    CN116343017A

  • Underwater target detection algorithm based on image enhancement and attention mechanism

    CN117372853A

  • Multi-scale ship rotation detection method based on cross-level feature fusion

    CN117541541A