A general multi-level fusion attention model for ship detection in urban coastal SAR images

Abstract

The present invention relates to a universal multi-level fusion attention model for ship detection in SAR images of urban coasts, comprising establishing submodules DFEM, CMAFM and SMAFM. Firstly, DFEM is designed to extract rich features with different resolutions and semantic information for multi-scale ship detection; then CMAFM and SMAFM are designed to fuse channel attention blocks and spatial attention blocks on feature maps of different levels to better learn dependent features from channel and spatial dimensions to enhance feature representation; finally, the fused feature maps are input into an existing ship detection network to obtain detection results; the model can focus on more important part of the input image, enhance the position information of the ship, weaken unimportant feature information, improve model performance through a small number of additional parameters, and improve the accuracy of identifying large and small ships under the complex urban coast background.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to a universal multi-level fusion attention model for ship detection in SAR images of urban coasts. Background Art

[0002] Synthetic aperture radar (SAR) is a high-resolution imaging radar with the characteristics of high resolution, all-day and all-weather operation, and effective identification of camouflage and penetration of cover. In recent years, due to the good performance of neural networks in feature extraction, the research on neural networks in various fields has become more and more in-depth, and the target detection method based on deep learning has been widely used in SAR ship detection. However, for ship targets in SAR images, due to the multi-scale and multi-angle characteristics of ships, small ships occupy fewer pixels and have lower contrast than large ships; when detecting ships at multiple scales, the scattering characteristics of ships in SAR images will lead to unclear outlines of ships, especially in the case of urban coasts, where the complex urban coast background is prominent and the ships are less obvious. Common SAR target detection methods often only focus on the local features of ships, ignoring the relationship between image contexts, which increases the difficulty of multi-scale ship detection in images. Therefore, designing a general multi-level fusion attention model for ship detection in urban coastal SAR images that can highlight the important features of ships at multiple levels and reduce the difficulty of ship detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a universal multi-level fusion attention model for ship detection in SAR images of urban coasts.

[0004] The technical solution of the present invention is a general multi-level fusion attention model for ship detection in SAR images of urban coasts. The establishment steps are as follows:

[0005] Step 1, establish a deep feature extraction submodule DFEM;

[0006] Step 2, establish a channel multi-layer attention fusion submodule CMAFM and a spatial multi-layer attention fusion submodule SMAFM including a multi-layer perceptron MLP;

[0007] Step 3, link the DFEM submodule with the CMAFM and SMAFM submodules to form a channel multi-level attention branch and a spatial multi-level attention branch;

[0008] Step 4: Combine the channel multi-level attention branch and the spatial multi-level attention branch to establish a preliminary model, and combine it with the existing SAR image ship detection network to form a SAR image ship detection network based on MFAM;

[0009] Step 5: Collect historical SAR image detection data to form a data set, and divide it into a training set and a test set. Set the training parameters, use the training set to train the detection network with and without MFAM and obtain the general multi-level fusion attention model MFAM. Input the SAR images in the test set into MFAM for detection.

[0010] As a further improvement of the present invention, the DFEM submodule extracts deep features with a convolution with a kernel size of 3×3, executes RELU function and batch normalization operations in sequence, and then downsamples through a maximum pooling operation with a size half of the shallow feature map, and uses it twice to obtain a multi-scale feature map.

[0011] As a further improvement of the present invention, the multi-layer perceptron MLP performs a convolution operation to change the number of channels of the feature map to the original 1 / r, where r is the reduction rate, and then performs batch normalization operation and ReLU function in sequence to normalize and activate the data, and then changes the original number of channels through the convolution layer, and uses the sigmoid activation function to obtain the channel attention map .

[0012] As a further improvement of the present invention, the CMAFM submodule inputs a multi-scale feature map, compresses the spatial dimension of the input feature map, performs average pooling and maximum pooling on the multi-scale feature map obtained by inputting the DFEM submodule, compresses the features of each layer in dimension, and obtains two feature maps of size c×1×1; and then uses a multi-layer perceptron MLP to process the results of average pooling and maximum pooling, and the processing formula is,

[0013] {C}_{1}=BN\left [ {\sum ^{3}_{i=1} {MLP\left ( {Maxpool\left ( {{F}_{i}} \right )} \right )}} \right ] (1)

[0014] {C}_{2}=BN\left [ {\sum ^{3}_{i=1} {MLP\left ( {Avgpool\left ( {{F}_{i}} \right )} \right )}} \right ] (2)

[0015] (3)

[0016] Where i = {1,2,3}, Maxpool represents the maximum pooling operation, Avgpool represents the average pooling operation, BN represents the batch normalization operation, and Sigmoid is the activation function.

[0017] As a further improvement of the present invention, the SMAFM submodule extracts feature maps of three different levels through the DFEM submodule and inputs them to generate a spatial attention block; then, maximum pooling and average pooling are applied along the channel direction of each layer of the feature map to generate two feature maps of size 1×H×W, and the feature maps of each layer are added and stacked into a feature descriptor of size 2×H×W; the feature descriptors are normalized in a batch normalization layer, connected and convolved using a standard convolutional layer with a channel number of 1, and the number of channels is adjusted; and then a sigmoid activation function is used to obtain a spatial attention map , get the weight value of each feature point in the input feature map, the weight value is between 0 and 1; the expression formula of the SMAFM submodule is,

[0018] (1)

[0019] (2)

[0020] (3)

[0021] Where i = {1,2,3}, Maxpool represents the maximum pooling operation, Avgpool represents the average pooling operation, Sigmoid is the activation function, Indicates that the feature map is upsampled by 2i-1 times, BN indicates batch normalization operation, Con 7×7 Indicates that the convolution kernel is a 7×7 convolution layer.

[0022] As a further improvement of the present invention, the existing SAR image ship detection network includes Faster-rcnn and RetinaNet; the detection network training is based on the Pytorch framework, and is trained using the stochastic gradient descent method. The weight decay used in the training is 0.0005, the momentum is 0.9, and the learning rate is 0.005.

[0023] After adopting the above method, by designing three sub-modules DFEM, CMAFM and SMAFM, the channel multi-level attention branch and the spatial multi-level attention branch are constructed, and the multi-level fusion attention model MFAM is constructed through these two branches.

[0024] First, DFEM is used to extract rich features with different resolutions and semantic information, which can highlight ship targets of different scales in the image and suppress the urban coastal background. The RELU function and batch normalization operations are executed in sequence to normalize the data and optimize the parameters. Downsampling is performed through the maximum pooling operation to reduce parameters while retaining the features, prevent overfitting, and improve the generalization ability of the model.

[0025] By introducing the multi-layer perceptron MLP, the convolution of the feature area can be sampled to enhance the learning of the feature area, and the weights can be output along the channel dimension of the feature map to reduce parameter overhead, which helps to obtain the importance of feature information at different levels in the multi-layer branches; by performing batch normalization operations and ReLU functions in sequence to normalize and activate the data, the convergence speed of the network is accelerated to prevent gradient explosion, gradient disappearance and overfitting.

[0026] Then, CMAFM and SMAFM are used to fuse the channel attention blocks and spatial attention blocks on feature maps of different levels to better learn the dependent features from the channel and spatial dimensions to enhance the feature representation; the CMAFM sub-module is used to determine which level of feature information is more important in the multi-layer branches of the channel, and the maximum pooling takes the maximum value of the pixels in the pooling area to obtain a feature map that is more sensitive to texture feature information, and the average pooling takes the average value of the image in the pooling area to obtain feature information that is more sensitive to background information. The combination of the two can enhance the characteristics of the image; the feature maps generated by the maximum pooling and average pooling are stacked through the SMAFM sub-module to obtain a feature descriptor, which is then processed through a standard convolutional layer and a sigmoid activation function to obtain a spatial attention map, which can focus on the more important part of the input image, enhance the location information of the ship, and weaken unimportant feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is the structural diagram of the general multi-level fusion attention module MFAM of the present invention.

[0028] Figure 2 This is the structural diagram of the DFEM submodule of the present invention.

[0029] Figure 3 It is a structural diagram of the multi-layer perceptron MLP of the present invention.

[0030] Figure 4 This is a structural diagram of the CMAFM submodule of the present invention.

[0031] Figure 5 This is a structural diagram of the SMAFM submodule of the present invention.

[0032] Figure 6 This is a comparison chart of the results of the general detection model.

[0033] Figure 7 The image conversion evaluation index table is a result comparison chart. DETAILED DESCRIPTION

[0034] like Figure 1-5 A general multi-level fusion attention model for ship detection in SAR images of urban coasts is shown in the figure. The establishment steps are as follows:

[0035] Step 1, establish a deep feature extraction submodule DFEM;

[0036] Step 2, establish a channel multi-layer attention fusion submodule CMAFM and a spatial multi-layer attention fusion submodule SMAFM including a multi-layer perceptron MLP;

[0037] Step 3, link the DFEM submodule with the CMAFM and SMAFM submodules to form a channel multi-level attention branch and a spatial multi-level attention branch;

[0038] Step 4: Combine the channel multi-level attention branch and the spatial multi-level attention branch to establish a preliminary model, and combine it with the existing SAR image ship detection network to form a SAR image ship detection network based on MFAM;

[0039] Step 5: Collect historical SAR image detection data to form a data set, and divide it into a training set and a test set. Set the training parameters, use the training set to train the detection network with and without MFAM and obtain the general multi-level fusion attention model MFAM. Input the SAR images in the test set into MFAM for detection.

[0040] The DFEM submodule extracts deep features with a convolution with a kernel size of 3×3, performs RELU function and batch normalization operations in sequence, and then downsamples through a maximum pooling operation with a size half of the shallow feature map. This is used twice to obtain a multi-scale feature map.

[0041] The multi-layer perceptron MLP performs a convolution operation to change the number of channels of the feature map to the original 1 / r, where r is the reduction rate, and then performs batch normalization and ReLU functions in sequence to normalize and activate the data. After that, the original number of channels is changed through the convolution layer, and the channel attention map is obtained using the sigmoid activation function. .

[0042] The CMAFM submodule inputs the multi-scale feature map, compresses the spatial dimension of the input feature map, performs average pooling and maximum pooling on the multi-scale feature map obtained by inputting the DFEM submodule, compresses the features of each layer in dimension, and obtains two feature maps of size c×1×1; then uses the multi-layer perceptron MLP to process the results of average pooling and maximum pooling, and the processing formula is:

[0043] {C}_{1}=BN\left [ {\sum ^{3}_{i=1} {MLP\left ( {Maxpool\left ( {{F}_{i}} \right )} \right )}} \right ] (1)

[0044] {C}_{2}=BN\left [ {\sum ^{3}_{i=1} {MLP\left ( {Avgpool\left ( {{F}_{i}} \right )} \right )}} \right ] (2)

[0045] (3)

[0046] Where i = {1,2,3}, Maxpool represents the maximum pooling operation, Avgpool represents the average pooling operation, BN represents the batch normalization operation, and Sigmoid is the activation function.

[0047] The SMAFM submodule extracts three different levels of feature maps from the DFEM submodule and inputs them to generate a spatial attention block; then, the maximum pooling and average pooling are applied along the channel direction of each layer of feature maps to generate two feature maps of size 1×H×W, and the feature maps of each layer are added and stacked into a feature descriptor of size 2×H×W; the feature descriptors are normalized in the batch normalization layer, connected and convolved using a standard convolutional layer with a channel number of 1, and the number of channels is adjusted; then, the sigmoid activation function is used to obtain the spatial attention map , get the weight value of each feature point in the input feature map, the weight value is between 0 and 1; the expression formula of the SMAFM submodule is,

[0048] (1)

[0049] (2)

[0050] (3)

[0051] Where i = {1,2,3}, Maxpool represents the maximum pooling operation, Avgpool represents the average pooling operation, Sigmoid is the activation function, Indicates that the feature map is upsampled by 2i-1 times, BN indicates batch normalization operation, Con 7×7 Indicates that the convolution kernel is a 7×7 convolution layer.

[0052] The existing SAR image ship detection network includes Faster-rcnn and RetinaNet; the detection network training is based on the Pytorch framework and is trained using the stochastic gradient descent method. The weight decay used in the training is 0.0005, the momentum is 0.9, and the learning rate is 0.005.

[0053] By designing three sub-modules DFEM, CMAFM and SMAFM, the channel multi-level attention branch and the spatial multi-level attention branch are constructed, and the multi-level fusion attention model MFAM is constructed through these two branches.

[0054] First, DFEM is used to extract rich features with different resolutions and semantic information, which can highlight ship targets of different scales in the image and suppress the urban coastal background. The RELU function and batch normalization operations are executed in sequence to normalize the data and optimize the parameters. Downsampling is performed through the maximum pooling operation to reduce parameters while retaining the features, prevent overfitting, and improve the generalization ability of the model.

[0055] By introducing the multi-layer perceptron MLP, the convolution of the feature area can be sampled to enhance the learning of the feature area, and the weights can be output along the channel dimension of the feature map to reduce parameter overhead, which helps to obtain the importance of feature information at different levels in the multi-layer branches; by performing batch normalization operations and ReLU functions in sequence to normalize and activate the data, the convergence speed of the network is accelerated to prevent gradient explosion, gradient disappearance and overfitting.

[0056] Then, CMAFM and SMAFM are used to fuse the channel attention blocks and spatial attention blocks on feature maps of different levels to better learn the dependent features from the channel and spatial dimensions to enhance the feature representation; the CMAFM sub-module is used to determine which level of feature information is more important in the multi-layer branches of the channel, and the maximum pooling takes the maximum value of the pixels in the pooling area to obtain a feature map that is more sensitive to texture feature information, and the average pooling takes the average value of the image in the pooling area to obtain feature information that is more sensitive to background information. The combination of the two can enhance the characteristics of the image; the feature maps generated by the maximum pooling and average pooling are stacked through the SMAFM sub-module to obtain a feature descriptor, which is then processed through a standard convolutional layer and a sigmoid activation function to obtain a spatial attention map, which can focus on the more important part of the input image, enhance the location information of the ship, and weaken unimportant feature information.

[0057] As attached Figure 6 As shown in the figure, group a is the target detection map used as a reference, group b is the Faster-rcnn detection result, group c is the Faster-rcnn+MFAM detection result, group d is the retinanet detection result, and group e is the retinanet+MFAM detection result. After the existing detection network is added with MFAM, the detection effect is greatly improved. In group b, there are redundant frames in the Faster-rcnn detection. After adding MFAM to group c, the redundant frames are filtered. In group d, there are missing frames in the retinanet detection. After adding MFAM to group e, small ships have stronger recognition capabilities. As shown in the attached figure Figure 7As shown in the figure, after converting the image into indicators, the detection model with MFAM added has an indicator that is 3.2% higher than that in Fast-RCNN and 2.3% higher than that in Retinanet, and the detection effect is significantly improved.

Claims

1. A general multi-level fusion attention method for ship detection in urban coastal SAR images, characterized by: The steps to build are as follows: Step 1, establish a deep feature extraction submodule DFEM; Step 2, establish a channel multi-layer attention fusion submodule CMAFM and a spatial multi-layer attention fusion submodule SMAFM including a multi-layer perceptron MLP; Step 3, link the DFEM submodule with the CMAFM and SMAFM submodules to form a channel multi-level attention branch and a spatial multi-level attention branch; Step 4: Combine the channel multi-level attention branch and the spatial multi-level attention branch to establish a preliminary model, and combine it with the existing SAR image ship detection network to form a SAR image ship detection network based on MFAM; Step 5: Collect historical SAR image detection data to form a data set, and divide it into a training set and a test set. Set the training parameters, use the training set to train the detection network with and without MFAM and obtain the general multi-level fusion attention model MFAM, and input the SAR images in the test set into MFAM for detection. The multi-layer perceptron MLP performs a convolution operation to change the number of channels of the feature map to the original 1 / r, where r is the reduction rate, and then performs batch normalization and ReLU functions in sequence to normalize and activate the data. After that, the original number of channels is changed through the convolution layer, and the channel attention map is obtained using the sigmoid activation function. ; The CMAFM submodule inputs the multi-scale feature map, compresses the spatial dimension of the input feature map, performs average pooling and maximum pooling on the multi-scale feature map obtained by inputting the DFEM submodule, compresses the features of each layer in dimension, and obtains two feature maps of size c×1×1; then uses the multi-layer perceptron MLP to process the results of average pooling and maximum pooling, and the processing formula is: Where i = {1, 2, 3}, Maxpool represents the maximum pooling operation, Avgpool represents the average pooling operation, BN represents the batch normalization operation, and Sigmoid is the activation function; The SMAFM submodule generates a spatial attention block by inputting the feature maps of three different levels extracted by the DFEM submodule; then applying maximum pooling and average pooling along the channel direction of each layer of feature maps to generate two feature maps of size 1×H×W, and adding and stacking the feature maps of each layer into a feature descriptor of size 2×H×W; The feature descriptors are normalized in the batch normalization layer, connected and convolved using a standard convolutional layer with a channel number of 1, and the number of channels is adjusted; then the sigmoid activation function is used to obtain the spatial attention map , get the weight value of each feature point in the input feature map, the weight value is between 0 and 1; the expression formula of the SMAFM submodule is, Where i = {1, 2, 3}, Maxpool represents the maximum pooling operation, Avgpool represents the average pooling operation, Sigmoid is the activation function, Up 2i-1 Indicates that the feature map is upsampled by 2i-1 times, BN indicates batch normalization operation, Con 7×7 Indicates that the convolution kernel is a 7×7 convolution layer.

2. A general multi-level fusion attention method for ship detection in urban coastal SAR images according to claim 1, characterized in that: The DFEM submodule extracts deep features with a convolution with a kernel size of 3×3, performs RELU function and batch normalization operations in sequence, and then downsamples through a maximum pooling operation with a size half of the shallow feature map. This is used twice to obtain a multi-scale feature map.

3. A general multi-level fusion attention method for ship detection in urban coastal SAR images as claimed in claim 1, characterized in that: The existing SAR image ship detection network includes Faster-rcnn and RetinaNet; the detection network training is based on the Pytorch framework and is trained using the stochastic gradient descent method. The weight decay used in the training is 0.0005, the momentum is 0.9, and the learning rate is 0.005.

Images