A remote sensing ship target detection method based on deformation attention pyramid

Abstract

A remote sensing ship target detection method based on deformable attention pyramid belongs to the field of remote sensing detection technology. They are: deformable attention pyramid network, multi-scale feature enhancement network, multi-scale adaptive pooling network and rotation prediction network. The detected image passes through the above sub-networks in sequence, and gradually realizes feature extraction, feature enhancement, generation of regions of interest and marking of ships. Advantages: The deformable attention pyramid network suppresses the background noise of remote sensing images from multiple scales and reduces the interference of irrelevant features on the model. The multi-scale feature enhancement network uses parallel Laplacian convolution kernels and dilated convolution kernels to sharpen the features of weak and small targets, and enhances the model's feature description of small target ships. The multi-scale adaptive pooling network uses a multi-scale adaptive pooling strategy to obtain a more accurate detection region of interest. The rotation prediction network generates a rotation prediction box based on a regression strategy, which is used to accurately mark ships in any direction.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing detection, and in particular to a remote sensing ship target detection method based on deformation attention pyramid. Background Art

[0002] Automated detection of ships is of great significance and value in maritime traffic management, maritime rescue, and national defense security applications. In recent years, with the rapid development of remote sensing technology, researchers can obtain high-quality optical remote sensing images from satellite or airborne sensors at a reasonable cost. Therefore, optical remote sensing images have become the mainstream image source for ship detection. This makes ship target detection technology based on optical remote sensing images a research hotspot.

[0003] At present, there are two main types of ship detection methods for remote sensing images: traditional detection methods and deep learning-based ship detection methods. Before the emergence of deep learning ship detection methods, many researchers applied machine learning technology to the field of ship detection. However, since ships in remote sensing images are affected by background noise such as light, clouds, and particles in the atmosphere, and ships themselves are usually small, arbitrarily pointed, and crowded. Therefore, it is often difficult for traditional methods to accurately summarize the characteristics of ships under various imaging conditions. As a result, the accuracy of traditional detection methods is low and they lack generalization ability.

[0004] With the widespread application of convolutional neural networks, it has shown strong performance in the field of target detection with its outstanding feature extraction ability. Many ship detection methods based on deep learning have been born, but most methods use horizontal bounding boxes to detect ships, resulting in a lot of redundant information in the detection results and the inability to perform multi-scale ship detection. Subsequently, some scholars proposed a rotation Libra R-CNN method. The idea of ​​this method is to balance the three levels of the neural network. First, a balanced pyramid is used to extract multi-scale ship features; then, the intersection-over-union balanced sampling is used to generate region proposals; finally, the balanced L1 Loss is introduced in the rotation branch, which greatly improves the ship detection performance in crowded scenes. However, this method is still affected by background noise and has a high missed detection rate for small target ships. How to maximize the extraction of ship features with large aspect ratios, arbitrary shapes and sizes in crowded scenes, and arbitrary directions and effectively suppress background noise is a current problem. Summary of the invention

[0005] In view of the above-mentioned problems or shortcomings, the present invention provides a remote sensing ship target detection method based on a deformable attention pyramid to solve the problem that ships in optical remote sensing images are interfered by background noise, and that ships usually have the characteristics of small targets, large aspect ratios, crowded arrangements and arbitrary directions, resulting in the inability of existing target detection methods to accurately detect ships.

[0006] The object of the present invention is achieved in this way: the remote sensing ship target detection method includes: the following four sub-networks, namely: deformation attention pyramid network, multi-scale feature enhancement network, multi-scale adaptive pooling network and rotation prediction network; the detected image passes through the above sub-networks in sequence, and gradually realizes feature extraction, feature enhancement, generation of regions of interest and marking of ships;

[0007] Firstly, the ship deformation convolution unit and the ship attention module are integrated into the deformation attention pyramid network to fully extract the features of ships in any direction;

[0008] The ship deformation convolution unit can distribute the convolution sampling points that were originally fixed in position inside the ship; the ship attention module can effectively allocate the weights of different components in the multi-scale feature map, so that the model pays attention to ships of different scales and suppresses multi-scale background noise.

[0009] Thirdly, a multi-scale feature enhancement module is used in the multi-scale feature enhancement network to extract ship features, so as to enhance ship features of different scales and enhance ship target features;

[0010] The multi-scale feature enhancement module uses parallel Laplace convolution kernels and dilated convolution kernels. The Laplace convolution kernel is used to enhance the ship targets in the feature maps of different scales, so that the detection performance of the network is further improved; the dilated convolution kernel is used to extract the background information of small-scale targets, which is conducive to improving the network's detection performance for small-target ships.

[0011] Then, a multi-scale adaptive pooling strategy is adopted in the multi-scale adaptive pooling network to accurately obtain the region of interest of the large aspect ratio ship and learn the multi-scale contextual information of the ship;

[0012] The multi-scale adaptive pooling strategy is based on the bilinear interpolation method, which can adaptively obtain horizontal and vertical sampling points according to the region proposal, so that the model samples the ship target more evenly, so as to obtain a more effective description of the ship characteristics and realize the reasoning of nearby unknown targets to a certain extent.

[0013] Finally, the rotation prediction network generates a ship-based RBB through five-parameter regression and foreground and background classification to accurately mark ships in any direction.

[0014] The specific steps are as follows:

[0015] Step S1, input the image to be detected into the deformation attention pyramid network for processing, the network uses the ship deformation convolution unit as the basic feature extraction module, and uses the ship attention module to suppress background noise;

[0016] Step S2, inputting the multi-scale feature map processed by the deformation attention pyramid network into the multi-scale feature enhancement network, and using the multi-scale feature enhancement module therein to enhance its features;

[0017] Step S3, inputting the data enhanced by the multi-scale feature enhancement network into the multi-scale adaptive pooling network, the multi-scale adaptive pooling network first uses the Oriented RPN to generate a multi-scale rotation region proposal, and then uses the multi-scale adaptive pooling strategy to obtain a more accurate region of interest; the Oriented RPN is a method based on the anchor strategy, which can generate region proposals of multiple scales and arbitrary directions;

[0018] Step S4, the rotation prediction network further identifies the ship in the remote sensing image based on the region of interest. First, a rotation bounding box is generated to mark the ship in any direction. Then, the skew IoU strategy is used to calculate the intersection-over-union ratio of the rotation bounding box, so as to better distinguish the positive and negative samples. Finally, the soft-rotate-NMS strategy is used to reasonably select the rotation prediction box. The skew IoU strategy is based on the triangulation idea and can accurately calculate the intersection-over-union ratio of the rotation bounding box. The soft-rotate-NMS strategy first updates the confidence of the remaining adjacent rotation prediction boxes according to the overlapping area, and then reasonably selects the rotation prediction box.

[0019] Step S5, adopting a multi-task loss function to train the above-mentioned ship detection network, including regression loss and classification loss, the regression loss is used to adjust the position of the rotation bounding box, and the classification loss is used to determine whether the target to be detected is a ship; gradually train and optimize parameters to realize the detection of remote sensing ship targets.

[0020] In step S1, the deformation attention pyramid network processing adopts two processing processes: bottom-up and top-down;

[0021] The bottom-up processing process is: use the SDC unit to fully extract the features of the input image to form a feature pyramid {C2, C3, C4, C5} with increasing dimensions and decreasing resolution;

[0022] The top-down processing process is: taking {C2, C3, C4, C5} as input, constructing a top-down feature pyramid {P2, P3, P4, P5} by upsampling and lateral connection; in the two processing processes, the ship attention module is integrated to suppress the background noise in the feature maps of different scales;

[0023] When identifying a ship of complex size, the following processing is performed:

[0024] For the detection of smaller ships, a feature map with higher resolution is used, and the semantic information in the low-resolution feature map {P4, P5} is integrated into the high-resolution feature map {P2, P3} by using multi-scale feature fusion;

[0025] For larger ships, a downsampling operation with a step size of 2 is performed on the basis of the high-dimensional feature map P5 to generate the P6 feature map, which is specifically used for the detection of large ships.

[0026] The feature extraction process of the SDC unit is as follows:

[0027] First, a 1×1 traditional convolution is used to adjust the number of channels of the input feature map, and a 3×3 traditional convolution is used to adaptively obtain the two-dimensional offset. Then, based on the two-dimensional offset, a 3×3 deformable convolution is used to extract ship features of any shape, size, and direction. Finally, a 1×1 traditional convolution is used to reduce the dimension, and a linear activation function is used to activate the features to generate an output feature map.

[0028] The ship deformation convolution unit is:

[0029] Remote sensing images have a high resolution, which means that the model needs to occupy a large amount of video memory resources when training such images; if the batch size is set too large, the computer video memory will overflow and training will fail; compared with batch normalization, the calculation method of group normalization is optimized; the group normalization operation does not depend on the size of the batch size, and it calculates the mean and variance of each group in the channel direction; group normalization is more suitable for remote sensing image training; the batch size represents the number of samples selected for a single training;

[0030] The linear activation function is used in the ship deformation convolution unit to ensure that the model retains more complete feature information; if the input features are located in a non-low-dimensional subspace of the input space, the nonlinear activation function will destroy the extracted features and affect the detection performance;

[0031] The ship attention module is:

[0032] The ship attention module suppresses the background noise of remote sensing images from multiple scales, reduces the interference of irrelevant features on the model, and suppresses the background noise to the greatest extent;

[0033] The ship attention module is based on the inherent multi-scale architecture of the Deformable Attention Pyramid Network, including SA2, SA3 and SA4 modules; the process of adjusting attention in the SA2 module is described as follows:

[0034]

[0035] V3=Flatten(Conv(F3))

[0036] A3=Grid(Sigmoid(V3))

[0037]

[0038] in, Indicates the fusion of feature maps of the same size; F3 indicates the fused feature map; Flatten indicates the flattening operation, which can transform the feature map into a one-dimensional vector V3; Sigmoid is an activation function, which can obtain the attention coefficient of each feature point of V3; Grid indicates the gridding operation, which transforms the one-dimensional attention coefficient into an attention coefficient map A3 of the same size as P3; Represents the multiplication of corresponding elements of two matrices;

[0039] The attention adjustment process of SA3 and SA4 is the same as that of SA2 module.

[0040] In step S2, a multi-scale feature enhancement module in a multi-scale feature enhancement network is used, and the multi-scale feature enhancement module uses parallel Laplace convolution kernels and hole convolution kernels; wherein the Laplace convolution kernel is used to sharpen the features of small targets; the hole convolution kernel is used to enhance the receptive field of small targets, and the receptive field is the area where the network can perceive the input feature map, so that the multi-scale feature enhancement network enhances the model's feature description of small target ships; the obtained feature map is feature enhanced, and after the deformation attention pyramid network fully extracts the features of the ship targets in the remote sensing image, the multi-scale ships are accurately detected, and the missed detection rate of small target ships is eliminated;

[0041] The enhancement process of the multi-scale feature enhancement module is as follows:

[0042]

[0043] Among them, GN(·) represents group normalization; LConv 3×3 Conv represents a Laplace convolution kernel of size 3×3, which is used to sharpen ship targets of different scales through Laplace features; 1×1 Represents a standard convolution kernel of size 1×1, used to adjust the number of channels; DConv 3×3 represents a 3×3 dilated convolution, which extracts background information of small-scale targets by expanding the receptive field to improve the model's feature description of small target ships; Represents the feature fusion operation.

[0044] In step S3, Oriented RPN is used to generate region proposals of various scales and arbitrary directions; then a pooling operation is performed to generate a k×k fixed-size ROI region;

[0045] The multi-scale adaptive pooling strategy is based on a bilinear interpolation method, which can adaptively and uniformly obtain horizontal and vertical sampling points and aggregate them into multiple closed loop rectangles to form a compact feature representation;

[0046] In the multi-scale adaptive pooling network, the deep feature map has low resolution, large receptive field, and weak geometric information representation ability, which makes it easier to highlight the contour features of the ship, which is beneficial to improving the positioning ability of the model; the deep feature is represented as a closed loop rectangle close to the center point;

[0047] The shallow feature map has high resolution, a small receptive field, and strong geometric information representation capability, which is beneficial to improving the classification ability of the model; the shallow feature is represented as a closed loop rectangle away from the center point;

[0048] Finally, the multi-scale adaptive pooling network generates a fixed-size feature map covering deep and shallow feature information through these closed loop rectangles.

[0049] In step S4, the rotation prediction network further identifies ships in the remote sensing image based on the region of interest, and generates a rotation bounding box to mark ships in any direction.

[0050] The rotated bounding box can be represented by five parameters, namely x, y, w, h, θ. (x, y) represents the coordinates of the center point of the rotated bounding box; w and h represent the width and height of the rotated bounding box respectively; θ represents the angle of w of the rotated bounding box relative to the x-axis. The angle range is [-90°, 0°]; the regression process of the rotated bounding box is as follows:

[0051] t x =(xx a ) / w a ,t y =(yy a ) / h a ,t w =log(w / w a ),t h =log(h / h a ),t θ =θ-θ a

[0052] t′ x =(x′-x a ) / w a ,t′ y =(y′-y a ) / h a ,t′ w =log(w′ / w a ),t′ h =log(h′ / h a ),t′θ =θ′-θ a

[0053] In the above formula, x, y, w, h, θ represent the parameters of the label bounding box; x a ,y a ,w a ,h a ,θ a represents the parameters of the auxiliary bounding box marked according to the region proposal; x', y', w', h', θ' represent the parameters of the rotation prediction box.

[0054] After the rotation bounding box is generated, in order to better distinguish positive and negative samples, the skewIoU strategy is used to calculate the intersection-over-union ratio of the rotation bounding box; the skewIoU strategy is based on the triangulation idea and can calculate the intersection-over-union ratio more accurately;

[0055] Finally, the soft-rotate-NMS strategy is used to remove redundant rotation prediction boxes. When selecting the rotation prediction box with the highest confidence, the soft-rotate-NMS strategy updates the confidence of the remaining adjacent rotation prediction boxes according to the overlapping area, reasonably selects the rotation prediction box, and accurately marks the ship.

[0056] In step S5, the ship detection network is trained to optimize parameters to achieve remote sensing ship target detection: a multi-task loss function is used, including regression loss and classification loss;

[0057] The regression loss is used to adjust the position of the rotation bounding box, and the classification loss is used to determine whether the target is a ship; the definitions are as follows:

[0058]

[0059] Where i refers to the i-th auxiliary bounding box; L cls is the logarithmic loss of the foreground and background, where the foreground is the ship and the background is the non-ship target; L reg It is expressed as the regression loss of the target candidate box. In order to improve the robustness of the model and effectively solve the problem of imbalance between positive and negative samples, the present invention uses balanced L1 Loss as the regression loss; P i Refers to the probability that each auxiliary bounding box belongs to the target ship; t i Represents the coordinate parameters of the predicted candidate box of the sample; Represents the coordinate parameters of the real candidate box of the sample; λ is the balance parameter of the weight;

[0060] Category L cls and regression L reg The loss is specifically expressed as:

[0061] L cls=-(1-y i )log(1-p i )-(y i )log(p i )

[0062]

[0063] Among them, y i Indicates the positive or negative of the sample. When y i When it is 1, it means the corresponding sample is a positive sample. i When y is 0, it means that the corresponding sample is a negative sample. In particular, when y i When it is 0, there is no regression loss. |x| γ is defined as

[0064] Beneficial effect, due to the adoption of the above scheme, it is composed of a deformable attention pyramid network, a multi-scale feature enhancement network, a multi-scale adaptive pooling network and a rotation prediction network. First, the deformable attention pyramid network integrates the independently designed ship deformation convolution unit and the ship attention module to fully extract the features of ships in any direction and guide the network to pay attention to ship areas of different scales. Thirdly, a multi-scale feature enhancement module is designed, which uses parallel Laplace convolution kernels and hole convolution kernels to extract features of ships to enhance the features of ships of different scales. Then, a multi-scale feature enhancement network is designed to accurately obtain the region of interest of large aspect ratio ships and learn the multi-scale contextual information of ships. Finally, a rotation detection box is generated through the rotation prediction network to accurately mark ships in any direction. Compared with other mainstream target detection algorithms, it has obvious advantages.

[0065] The invention solves the problem that ships in optical remote sensing images are disturbed by background noise, and that existing target detection methods cannot accurately detect ships due to the characteristics of ships being small, having a large aspect ratio, being crowded, and having arbitrary directions, thereby achieving the purpose of the invention.

[0066] Advantages: The Deformable Attention Pyramid Network solves the problem that other advanced target detection models cannot fully extract ship features, and suppresses the background noise of remote sensing images from multiple scales, reducing the interference of irrelevant features on the model.

[0067] The multi-scale feature enhancement network uses parallel Laplacian convolution kernels and dilated convolution kernels to sharpen the features of small targets and improve the receptive field of small targets. The receptive field is the area where the network can perceive the input feature map, which enhances the model's feature description of small target ships.

[0068] The multi-scale adaptive pooling network uses the multi-scale adaptive pooling strategy to obtain more accurate detection of regions of interest.

[0069] The rotation prediction network generates a rotation prediction box based on a regression strategy to accurately mark ships in any direction in the remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of the present invention.

[0071] Figure 2 This is a diagram of the remote sensing ship target detection structure of the present invention.

[0072] Figure 3 This is a structural diagram of the deformation attention pyramid network of the present invention.

[0073] Figure 4 This is a diagram of the multi-scale feature enhancement network structure of the present invention.

[0074] Figure 5 This is a diagram of the multi-scale adaptive pooling network structure of the present invention.

[0075] Figure 6 This is the working principle of the multi-scale adaptive pooling of the present invention.

[0076] Figure 7 This is the rotation bounding box representation method of the present invention.

[0077] Figure 8 It is the detection result diagram of the present invention. DETAILED DESCRIPTION

[0078] The remote sensing ship target detection method includes the following four sub-networks: deformation attention pyramid network, multi-scale feature enhancement network, multi-scale adaptive pooling network and rotation prediction network; the detected image passes through the above sub-networks in sequence to gradually realize feature extraction, feature enhancement, generation of regions of interest and marking of ships;

[0079] Firstly, the ship deformation convolution unit and the ship attention module are integrated into the deformation attention pyramid network to suppress multi-scale background noise;

[0080] The ship deformation convolution unit can distribute the convolution sampling points that were originally fixed in position inside the ship; the ship attention module can effectively allocate the weights of different components in the multi-scale feature map, so that the model pays attention to ships of different scales and suppresses multi-scale background noise.

[0081] Then, a multi-scale feature enhancement module is used in the multi-scale feature enhancement network to extract the features of the ship, so as to enhance the features of the ship at different scales and enhance the target features of the ship.

[0082] The multi-scale feature enhancement module uses parallel Laplace convolution kernels and dilated convolution kernels. The Laplace convolution kernel is used to enhance the ship targets in the feature maps of different scales, so that the detection performance of the network is further improved; the dilated convolution kernel is used to extract the background information of small-scale targets, which is conducive to improving the network's detection performance for small-target ships.

[0083] Thirdly, a multi-scale adaptive pooling strategy is adopted in the multi-scale adaptive pooling network to accurately obtain the region of interest of large aspect ratio ships and learn the multi-scale contextual information of the ships;

[0084] The multi-scale adaptive pooling strategy is based on the bilinear interpolation method, which can adaptively obtain lateral and longitudinal sampling points according to the region proposal, so that the model samples the ship target more evenly, so as to obtain a more effective description of the ship characteristics and realize the reasoning of the nearby unknown targets to a certain extent.

[0085] Finally, a rotation detection box is generated through the rotation prediction network, which generates a rotation bounding box based on the ship target through five-parameter regression and foreground and background classification to accurately mark ships in any direction.

[0086] The deformable attention pyramid network is expressed in English as Deformable Attention Pyramid Network, abbreviated as DAPNet; the multiscale feature enhancement network is expressed in English as Multiscale Feature Enhancement Network, abbreviated as MFENet; the multiscale adaptive pooling network is expressed in English as Multiscale Adaptive Pooling Network, abbreviated as MAPNet; the rotation prediction network is expressed in English as Rotation Prediction Network, abbreviated as RPNet.

[0087] The specific steps are as follows:

[0088] Step S1, input the image to be detected into DAPNet for processing, the network uses the ship deformation convolution unit as the basic feature extraction module, and uses the ship attention module to suppress background noise; the ship deformation convolution is expressed in English as Ship Deformation Convolution, abbreviated as SDC; the ship attention is expressed in English as ShipAttention, abbreviated as SA;

[0089] Step S2, inputting the multi-scale feature map processed by DAPNet into MFENet, and using the multi-scale feature enhancement module therein to enhance its features; the multi-scale feature enhancement module is expressed in English as Multiscale Feature Enhancement, abbreviated as MFE;

[0090] Step S3, data is input into MAPNet, MAPNet first uses Oriented RPN to generate multi-scale rotation region proposals, and then uses a multi-scale adaptive pooling strategy to obtain a more accurate region of interest; the Oriented RPN is a method based on an anchor strategy, which can generate region proposals of multiple scales and arbitrary directions; the multi-scale adaptive pooling is expressed in English as Multiscale Adaptive Pooling, abbreviated as MA pooling; the region of interest is expressed in English as Region of Interest, abbreviated as ROI;

[0091] Step S4, RPNet further identifies ships in the remote sensing image based on the ROI area, first generates a rotating bounding box to mark ships in any direction; then uses the skew IoU strategy to calculate the intersection over union ratio of the rotating bounding box, so as to better distinguish positive and negative samples; finally, uses the soft-rotate-NMS strategy to reasonably select the rotating prediction box; the rotating bounding box is expressed in English as Rotating Bounding Box, abbreviated as RBB; the intersection over union ratio is expressed in English as Intersection over Union, abbreviated as IoU; the skew IoU strategy, which is based on the triangulation idea, can accurately calculate the intersection over union ratio of the rotating bounding box; the soft-rotate-NMS strategy first updates the confidence of the remaining adjacent rotating prediction boxes according to the overlapping area, and then reasonably selects the rotating prediction box;

[0092] Step S5, adopting a multi-task loss function to train the above-mentioned ship detection network, including regression loss and classification loss, the regression loss is used to adjust the position of the RBB, and the classification loss is used to determine whether the target to be detected is a ship; gradually train and optimize parameters to realize the detection of remote sensing ship targets.

[0093] In step S1, DAPNet processing adopts two processing processes: bottom-up and top-down;

[0094] The bottom-up processing process is: use the SDC unit to fully extract the features of the input image to form a feature pyramid {C2, C3, C4, C5} with increasing dimensions and decreasing resolution;

[0095] The top-down processing process is: taking {C2, C3, C4, C5} as input, constructing a top-down feature pyramid {P2, P3, P4, P5} by upsampling and lateral connection; in the above two processing processes, the SA module is integrated to suppress the background noise in the feature maps of different scales;

[0096] When identifying a ship of complex size, the following processing is performed:

[0097] For the detection of smaller ships, a feature map with higher resolution is used, and the semantic information in the low-resolution feature map {P4, P5} is integrated into the high-resolution feature map {P2, P3} by using multi-scale feature fusion;

[0098] For larger ships, a downsampling operation with a step size of 2 is performed on the basis of the high-dimensional feature map P5 to generate the P6 feature map, which is specifically used for the detection of large ships.

[0099] The feature extraction process of the SDC unit is as follows:

[0100] First, a 1×1 traditional convolution is used to adjust the number of channels of the input feature map, and a 3×3 traditional convolution is used to adaptively obtain the two-dimensional offset. Then, based on the two-dimensional offset, a 3×3 deformable convolution is used to extract ship features of any shape, size, and direction. Finally, a 1×1 traditional convolution is used to reduce the dimension, and a linear activation function is used to activate the features to generate an output feature map.

[0101] The SDC unit is:

[0102] Remote sensing images have a high resolution, which means that the model needs to occupy a large amount of video memory resources when training such images; if the batch size is set too large, the computer video memory will overflow and training will fail; compared with batch normalization, the calculation method of group normalization is optimized; the group normalization operation does not depend on the size of the batch size, and it calculates the mean and variance of each group in the channel direction; group normalization is more suitable for remote sensing image training; the batch size represents the number of samples selected for a single training; the batch normalization is expressed in English as BatchNormalization, abbreviated as BN; the group normalization is expressed in English as Group Normalization, abbreviated as GN;

[0103] The linear activation function is used in the SDC unit to ensure that the model retains more complete feature information; if the input features are located in a non-low-dimensional subspace of the input space, the nonlinear activation function will destroy the extracted features and affect the detection performance;

[0104] The SA module is:

[0105] The SA module suppresses the background noise of remote sensing images from multiple scales, reduces the interference of irrelevant features on the model, and suppresses the background noise to the greatest extent;

[0106] The SA module is based on the inherent multi-scale architecture of DAPNet itself, including SA2, SA3 and SA4 modules; the process of SA2 module adjusting attention is described as follows:

[0107]

[0108] V3=Flatten(Conv(F3))

[0109] A3=Grid(Sigmoid(V3))

[0110]

[0111] in, Indicates the fusion of feature maps of the same size; F3 indicates the fused feature map; Flatten indicates the flattening operation, which can transform the feature map into a one-dimensional vector V3; Sigmoid is an activation function, which can obtain the attention coefficient of each feature point of V3; Grid indicates the gridding operation, which transforms the one-dimensional attention coefficient into an attention coefficient map A3 of the same size as P3; Represents the multiplication of corresponding elements of two matrices;

[0112] The attention adjustment process of SA3 and SA4 is the same as that of SA2 module.

[0113] In step S2, the MFE module in MFENet is used, and the MFE module uses parallel Laplace convolution kernels and dilated convolution kernels; wherein the Laplace convolution kernel is used to sharpen the features of small targets; the dilated convolution kernel is used to enhance the receptive field of small targets, and the receptive field is the area where the network can perceive the input feature map, so that MFENet enhances the model's feature description of small target ships; after DAPNet fully extracts the features of ship targets in remote sensing images, the obtained feature maps are enhanced, and multi-scale ships are accurately detected, and the missed detection rate of small target ships is eliminated;

[0114] The enhancement process of the MFE module is as follows:

[0115]

[0116] Among them, GN(·) represents group normalization; LConv 3×3 Conv represents a Laplace convolution kernel of size 3×3, which is used to sharpen ship targets of different scales through Laplace features; 1×1Represents a standard convolution kernel of size 1×1, used to adjust the number of channels; DConv 3×3 represents a 3×3 dilated convolution, which extracts background information of small-scale targets by expanding the receptive field to improve the model's feature description of small target ships; Represents the feature fusion operation.

[0117] In step S3, Oriented RPN is used to generate region proposals of various scales and arbitrary directions; then a pooling operation is performed to generate a k×k fixed-size ROI region;

[0118] The MApooling strategy is based on a bilinear interpolation method, which can adaptively and uniformly acquire horizontal and vertical sampling points and aggregate them into multiple closed-loop rectangles to form a compact feature representation.

[0119] In MAPNet, the deep feature map has low resolution, large receptive field, and weak geometric information representation ability, which makes it easier to highlight the contour features of the ship, which is beneficial to improving the positioning ability of the model; the deep feature is represented as a closed loop rectangle close to the center point;

[0120] The shallow feature map has high resolution, a small receptive field, and strong geometric information representation capability, which is beneficial to improving the classification ability of the model; the shallow feature is represented as a closed loop rectangle away from the center point;

[0121] Finally, MAPNet generates a fixed-size feature map covering deep and shallow feature information through these closed loop rectangles.

[0122] In step S4, RPNet further identifies ships in the remote sensing image based on the ROI region and generates RBB to mark ships in any direction;

[0123] The RBB can be represented by five parameters, namely x, y, w, h, θ. (x, y) represents the coordinates of the center point of the RBB; w and h represent the width and height of the RBB respectively; θ represents the angle of the RBB w relative to the x-axis. The angle range is [-90°, 0°]; the regression process of the RBB is as follows:

[0124] t x =(xx a ) / w a ,t y =(yy a ) / h a ,t w =log(w / w a ),t h =log(h / h a ),t θ =θ-θ a

[0125] t′ x =(x′-x a ) / w a ,t′ y =(y′-y a ) / h a ,t′ w =log(w′ / w a ),t′ h =log(h′ / h a ),t′ θ =θ′-θ a

[0126] In the above formula, x, y, w, h, θ represent the parameters of the label bounding box; x a ,y a ,w a ,h a ,θ a represents the parameters of the auxiliary bounding box marked according to the region proposal; x', y', w', h', θ' represent the parameters of the rotation prediction box;

[0127] After RBB is generated, in order to better distinguish positive and negative samples, the skewIoU strategy is used to calculate the intersection-over-union ratio of the rotated bounding box; the skewIoU strategy is based on the triangulation idea and can calculate the intersection-over-union ratio more accurately;

[0128] Finally, the soft-rotate-NMS strategy is used to remove redundant rotation prediction boxes. When selecting the rotation prediction box with the highest confidence, the soft-rotate-NMS strategy updates the confidence of the remaining adjacent rotation prediction boxes according to the overlapping area, reasonably selects the rotation prediction box, and accurately marks the ship.

[0129] In step S5, the ship detection network is trained to optimize parameters to achieve remote sensing ship target detection: a multi-task loss function is used, including regression loss and classification loss;

[0130] The regression loss is used to adjust the position of the RBB, and the classification loss is used to determine whether the target is a ship. The definitions are as follows:

[0131]

[0132] Where i refers to the i-th auxiliary bounding box; L cls is the logarithmic loss of the foreground and background (the foreground is the ship, and the background is the non-ship target); L reg It is expressed as the regression loss of the target candidate box. In order to improve the robustness of the model and effectively solve the problem of imbalance between positive and negative samples, the present invention uses balanced L1 Loss as the regression loss; Pi Refers to the probability that each auxiliary bounding box belongs to the target ship; t i Represents the coordinate parameters of the predicted candidate box of the sample; Represents the coordinate parameters of the real candidate box of the sample; λ is the balance parameter of the weight;

[0133] Category L cls and regression L reg The loss is specifically expressed as:

[0134] L cls =-(1-y i )log(1-p i )-(y i )log(p i )

[0135]

[0136] Among them, y i Indicates the positive or negative of the sample. When y i When it is 1, it means the corresponding sample is a positive sample. i When y is 0, it means that the corresponding sample is a negative sample. In particular, when y i When it is 0, there is no regression loss. |x| γ is defined as

[0137] Embodiment 1:

[0138] The following is combined with Figure 1-8 The specific embodiments of the present invention are described to facilitate those skilled in the art to understand the present invention, but it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0139] An embodiment of a remote sensing ship target detection method based on a deformation attention pyramid in a DOTA data set comprises the following steps:

[0140] The S1 and DOTA datasets are one of the largest public datasets for remote sensing target detection. The remote sensing images are from Google Earth, GF-2, and JL-1 satellites. The dataset contains 15 types of targets (ships, ports, helicopters, etc.), and the image resolution ranges from about 800×800 to 4000×4000.

[0141] Since there are limited images containing ships in DOTA, the present invention uses data augmentation technology to expand them. A total of 13624 images (about 165K ship targets) are obtained, and 1 / 2 are randomly selected as training sets, 1 / 3 as test sets, and 1 / 6 as validation sets. Finally, in order to fairly compare the performance of HBB detector and RBB detector, the present invention uses HBB and RBB to annotate the training sets respectively, and the annotation tools used are LabelImg and RoLabelImg respectively, and the target label is ship.

[0142] S2. Input the DOTA dataset to be processed into DAPNet for processing. The network uses the self-designed ship deformation convolution unit as the basic feature extraction module, and uses the ship attention module to suppress background noise. The DAPNet structure diagram is shown in the figure Figure 3 shown.

[0143] S3, the multi-scale feature map processed by DAPNet is input into MFENet, and the multi-scale feature enhancement module is used to enhance its features. Figure 4 shown.

[0144] S4, then the data is input into MAPNet, MAPNet first uses Oriented RPN to generate multi-scale rotation region proposals, and then uses MA pooling strategy to obtain more accurate regions of interest. The MAPNet structure diagram is shown in Figure 5 As shown, the working principle of MA pooling strategy is as follows Figure 6 shown.

[0145] S5. Finally, RPNet generates RBB to mark ships in any direction, such as Figure 7 shown.

[0146] S6. Experimental environment and hyperparameter settings:

[0147] The basic backbone network used in the model is ResNet101 integrated with SDC units and SA modules, named SS-ResNet101, relying on PyTorch 1.1.0 deep learning framework, programming language is Python 3.6. All experiments were carried out on a computer equipped with NVIDIA GeForce RTX 3080Ti, and its deep learning acceleration environment is CUDA 10.0, cuDNN7.0. During training, the present invention uses MomentumOptimizer for optimization training, with a batch size of 16, a weight decay of 0.0001, and a momentum of 0.9. The total number of training iterations is 160K times, the initial learning rate is 0.003, and the learning rate will decay with the increase in the number of iterations, so that the model can fully learn the target features.

[0148] The model testing results are as follows Figure 8 As shown in the figure. (a)~(b) are scenes with crowded distribution of ships; (e)~(h) are scenes with small target ships; (i)~(l) are scenes with multi-scale ships; (m)~(p) are scenes with background noise.

Claims

1. A remote sensing ship target detection method based on deformation attention pyramid, characterized by: The specific steps are as follows: Step S1: Input the image to be detected into the deformation attention pyramid network for processing. The network uses the ship deformation convolution unit as the basic feature extraction module and uses the ship attention module to suppress background noise. The ship attention module is: The ship attention module suppresses the background noise of remote sensing images from multiple scales, reduces the interference of irrelevant features on the model, and suppresses the background noise to the greatest extent; The ship attention module is based on the inherent multi-scale architecture of the Deformable Attention Pyramid Network, including SA2, SA3 and SA4 modules; the process of adjusting attention in the SA2 module is described as follows: V3=Flatten(Conv(F3)) A3=Grid(Sigmoid(V3)) in, Indicates the fusion of feature maps of the same size; F3 indicates the fused feature map; Flatten indicates the flattening operation, which can transform the feature map into a one-dimensional vector V3; Sigmoid is an activation function, which can obtain the attention coefficient of each feature point of V3; Grid indicates the gridding operation, which transforms the one-dimensional attention coefficient into an attention coefficient map A3 of the same size as P3; Represents the multiplication of corresponding elements of two matrices; The attention regulation process of SA3 and SA4 is the same as that of the SA2 module; Step S2, inputting the multi-scale feature map processed by the deformation attention pyramid network into the multi-scale feature enhancement network, and using the multi-scale feature enhancement module therein to enhance its features; the multi-scale feature enhancement module uses parallel Laplacian convolution kernels and hollow convolution kernels; Step S3: input the data enhanced by the multi-scale feature enhancement network into the multi-scale adaptive pooling network. The multi-scale adaptive pooling network first uses the Oriented RPN to generate multi-scale rotation region suggestions, and then uses the multi-scale adaptive pooling strategy to obtain a more accurate region of interest. Step S4: The rotation prediction network further identifies ships in the remote sensing image based on the region of interest. First, a rotation bounding box is generated to mark ships in any direction. Then, the skew IoU strategy is used to calculate the intersection-over-union ratio of the rotation bounding box to better distinguish positive and negative samples. Finally, the soft-rotate-NMS strategy is used to reasonably select the rotation prediction box. Step S5, adopting a multi-task loss function to train the above-mentioned ship detection network, including regression loss and classification loss, the regression loss is used to adjust the position of the rotation bounding box, and the classification loss is used to determine whether the target to be detected is a ship; gradually train and optimize parameters to realize the detection of remote sensing ship targets.

2. The remote sensing ship target detection method based on deformation attention pyramid according to claim 1 is characterized in that: In step S1, the deformation attention pyramid network processing adopts two processing processes: bottom-up and top-down; The bottom-up processing process is: use the ship-shaped convolution unit to fully extract the features of the input image to form a feature pyramid {C2, C3, C4, C5} with increasing dimensions and decreasing resolution; The top-down processing process is: taking {C2, C3, C4, C5} as input, constructing a top-down feature pyramid {P2, P3, P4, P5} by upsampling and lateral connection; in the two processing processes, the ship attention module is integrated to suppress the background noise in the feature maps of different scales; When identifying a ship of complex size, the following processing is performed: For the detection of smaller ships, a feature map with higher resolution is used, and the semantic information in the low-resolution feature map {P4, P5} is integrated into the high-resolution feature map {P2, P3} by using multi-scale feature fusion; For larger ships, a downsampling operation with a step size of 2 is performed on the basis of the high-dimensional feature map P5 to generate the P6 feature map, which is specifically used for the detection of large ships.

3. The remote sensing ship target detection method based on deformation attention pyramid according to claim 2 is characterized in that: The feature extraction process of the ship deformation convolution unit: First, a 1×1 traditional convolution is used to adjust the number of channels of the input feature map, and a 3×3 traditional convolution is used to adaptively obtain the two-dimensional offset. Then, based on the two-dimensional offset, a 3×3 deformable convolution is used to extract ship features of any shape, size, and direction. Finally, a 1×1 traditional convolution is used to reduce the dimension, and a linear activation function is used to activate the features to generate an output feature map. The ship deformation convolution unit is: Remote sensing images have a high resolution, which means that the model needs to occupy a large amount of video memory resources when training such images. If the batch size is set too large, the computer video memory will overflow and training will fail. Compared with batch normalization, the calculation method of group normalization is optimized; the group normalization operation does not depend on the batch size, and it calculates the mean and variance of each group in the channel direction; group normalization is more suitable for remote sensing image training; A linear activation function is used in the ship deformation convolution unit to ensure that the model retains more complete feature information; if the input features are located in a non-low-dimensional subspace of the input space, the nonlinear activation function will destroy the extracted features and affect the detection performance.

4. The remote sensing ship target detection method based on deformation attention pyramid according to claim 1 is characterized in that: In step S2, a multi-scale feature enhancement module in a multi-scale feature enhancement network is used, wherein the Laplace convolution kernel is used to sharpen the features of small targets; the dilated convolution kernel is used to enhance the receptive field of small targets, so that the multi-scale feature enhancement network enhances the model's feature description of small target ships; after fully extracting the features of ship targets in remote sensing images through the deformable attention pyramid network, the obtained feature map is enhanced to accurately detect multi-scale ships and eliminate the missed detection rate of small target ships; The enhancement process of the multi-scale feature enhancement module is as follows: Among them, GN(·) represents group normalization; LConv 3×3 Conv represents a Laplace convolution kernel of size 3×3, which is used to sharpen ship targets of different scales through Laplace features; 1×1 Represents a standard convolution kernel of size 1×1, used to adjust the number of channels; DConv 3×3 represents a 3×3 dilated convolution, which extracts background information of small-scale targets by expanding the receptive field to improve the model's feature description of small target ships; Represents the feature fusion operation.

5. The remote sensing ship target detection method based on deformation attention pyramid according to claim 1 is characterized in that: In step S3, OrientedRPN is used to generate region proposals of various scales and arbitrary directions; a pooling operation is then performed to generate a region of interest of a fixed size of k×k; The multi-scale adaptive pooling strategy is based on a bilinear interpolation method, which can adaptively and uniformly obtain horizontal and vertical sampling points and aggregate them into multiple closed loop rectangles to form a compact feature representation; In the multi-scale adaptive pooling network, the deep feature map has low resolution, large receptive field, and weak geometric information representation ability, which makes it easier to highlight the contour features of the ship, which is beneficial to improving the positioning ability of the model; the deep feature is represented as a closed loop rectangle close to the center point; The shallow feature map has high resolution, a small receptive field, and strong geometric information representation capability, which is beneficial to improving the classification ability of the model; the shallow feature is represented as a closed loop rectangle away from the center point; The multi-scale adaptive pooling network generates fixed-size feature maps covering deep and shallow feature information through these closed loop rectangles.

6. The remote sensing ship target detection method based on deformation attention pyramid according to claim 1 is characterized in that: In step S4, the rotation prediction network further identifies ships in the remote sensing image based on the region of interest, and generates a rotation bounding box to mark ships in any direction; The rotated bounding box can be represented by five parameters, namely x, y, w, h, θ; (x, y) represents the coordinates of the center point of the rotated bounding box; w and h represent the width and height of the rotated bounding box respectively; θ represents the angle of w of the rotated bounding box relative to the x-axis; the angle range is [-90°, 0°]; the regression process of the rotated bounding box is as follows: t x =(x-x a ) / w a ,t y =(y-y a ) / h a ,t w =log(w / w a ),t h =log(h / h a ),t θ =θ-θ a t′ x =(x′-x a ) / w a ,t′ y =(y′-y a ) / h a ,t′ w =log(w′ / w a ),t′ h =log(h′ / h a ),t′ θ =θ′-θ a In the above formula, x, y, w, h, θ represent the parameters of the label bounding box; x a ,y a ,w a ,h a ,θ a represents the parameters of the auxiliary bounding box marked according to the region proposal; x', y', w', h', θ' represent the parameters of the rotation prediction box; After the rotation bounding box is generated, in order to better distinguish positive and negative samples, the skewIoU strategy is used to calculate the intersection-over-union ratio of the rotation bounding box; the skewIoU strategy is based on the triangulation idea and can calculate the intersection-over-union ratio more accurately; Finally, the soft-rotate-NMS strategy is used to remove redundant rotation prediction boxes. When selecting the rotation prediction box with the highest confidence, the soft-rotate-NMS strategy updates the confidence of the remaining adjacent rotation prediction boxes according to the overlapping area, reasonably selects the rotation prediction box, and accurately marks the ship.

7. The remote sensing ship target detection method based on deformation attention pyramid according to claim 1 is characterized in that: In step S5, the ship detection network is trained to optimize parameters to achieve remote sensing ship target detection: a multi-task loss function is used, including regression loss and classification loss; The regression loss is used to adjust the position of the rotation bounding box, and the classification loss is used to determine whether the target is a ship; the definitions are as follows: Where i refers to the i-th auxiliary bounding box; L cls is the logarithmic loss of the foreground and background. In this method, the foreground is the ship and the background is the non-ship target; L reg It is expressed as the regression loss of the target candidate box; P i Refers to the probability that each auxiliary bounding box belongs to the target ship; t i Represents the coordinate parameters of the predicted candidate box of the sample; Represents the coordinate parameters of the real candidate box of the sample; λ is the balance parameter of the weight; classification L cls and regression L reg The loss is specifically expressed as: L cls =-(1-y i )log(1-p i )-(y i )log(p i ) Among them, y i Indicates the positive or negative of the sample. When y i When it is 1, it means the corresponding sample is a positive sample. i When it is 0, it means that the corresponding sample is a negative sample; in particular, when y i When it is 0, there is no regression loss, and |x| represents γ is defined as 8. A remote sensing ship target detection device based on deformation attention pyramid for implementing the method of claim 1, characterized in that: It includes the following four sub-networks: deformation attention pyramid network, multi-scale feature enhancement network, multi-scale adaptive pooling network and rotation prediction network. The detected image passes through the above sub-networks in turn to gradually realize feature extraction, feature enhancement, generate regions of interest and mark ships. Firstly, the ship deformation convolution unit and the ship attention module are integrated into the deformation attention pyramid network to fully extract the features of ships in any direction; Then, a multi-scale feature enhancement module is used in the multi-scale feature enhancement network to extract the features of the ship, so as to enhance the features of the ship at different scales and enhance the target features of the ship. Thirdly, a multi-scale adaptive pooling strategy is adopted in the multi-scale adaptive pooling network to accurately obtain the region of interest of large aspect ratio ships and learn the multi-scale contextual information of the ships; Finally, a rotation detection box is generated through the rotation prediction network, which generates a rotation bounding box based on the ship target through parameter regression and foreground and background classification, which is used to accurately mark ships in any direction.

Images