Sea surface optical remote sensing image ship detection method based on coarse and fine double detection frames

Abstract

The invention relates to a sea surface optical remote sensing image ship detection method based on a coarse and fine double detection framework, and the method comprises the steps: carrying out the coarse detection of a plurality of to-be-detected remote sensing images at the same time, obtaining a plurality of initial ship saliency maps, splicing the plurality of initial ship saliency maps, and obtaining a ship total saliency map; and finally, performing fine detection on the ship total saliency map based on an improved Mask R-CNN network framework to obtain a target detection map. The improved Mask R-CNN network framework is a network framework obtained by replacing a backbone network in an original Mask R-CNN framework from ResNet to Swinin-Transform and adding a context enhancement module in front of a feature pyramid. In the rough detection, a to-be-detected remote sensing image is processed based on a full convolution transformer module and a wide focus module to obtain an initial ship saliency map. According to the method, the omission ratio and the false alarm rate are low, and effective instance segmentation and accurate contour description positioning can be carried out on the ships which are densely parked or partially overlapped.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image target detection, in particular to a ship detection method for sea surface optical remote sensing image based on a coarse and fine double detection framework. BACKGROUND

[0002] As an important carrier for sea transportation, the accurate and rapid detection and positioning of ships have strategic significance and application value for ensuring maritime safety, maintaining maritime rights and interests, and promoting the sustainable development of the marine economy. At present, the remote sensing means for obtaining ship information mainly includes synthetic aperture radar, infrared, and visible light imaging. The synthetic aperture radar imaging system has all-weather, all-day working ability and wide detection range, but its image inherently lacks rich color and texture features, which has limitations in the fine identification of ships. Although infrared imaging performs outstandingly in low light conditions, it is highly sensitive to environmental factors, often leading to the loss of target structure details. In contrast, visible light optical remote sensing images can capture the fine texture, structure, and color information of the target, providing key data for robust ship identification.

[0003] However, the direct application of existing target detection algorithms to the ship detection task in large-scale visible light optical remote sensing images faces many challenges. These challenges mainly include: the wide coverage of remote sensing images, which results in the ship target usually being small or even tiny; the extremely complex marine background, which contains clouds, sea clutter, islands, coastlines, and wake generated by ship navigation, etc. multiple interference elements; the diverse shape and large scale of ships, and the possibility of dense arrangement or partial occlusion. Under the combined action of these challenges, the existing target detection algorithms often have high miss detection rate and false alarm rate when applied to the ship detection task in visible light optical remote sensing images, and it is difficult to achieve accurate instance segmentation and individual positioning. SUMMARY

[0004] Therefore, it is necessary to provide a ship detection method for sea surface optical remote sensing image based on a coarse and fine double detection framework, which has low miss detection rate and false alarm rate and can effectively segment and accurately locate the outline of densely parked or partially overlapped ships.

[0005] The present application provides a ship detection method for sea surface optical remote sensing image based on a coarse and fine double detection framework, comprising the following steps:

[0006] Simultaneously performing coarse detection on multiple remote sensing images to be detected to obtain multiple initial ship saliency maps;

[0007] Splicing multiple initial ship saliency maps to obtain a total ship saliency map;

[0008] The total ship saliency map is finely detected based on the improved Mask R-CNN network framework to obtain a target detection map.

[0009] The improved Mask R-CNN network framework is obtained by replacing the backbone network in the original Mask R-CNN framework with a Swin-Transformer and adding a context enhancement module before a feature pyramid.

[0010] The method comprises the following steps of:

[0011] The to-be-detected remote sensing image is input into a data preprocessing module to obtain a compressed semantic map of the remote sensing image.

[0012] The to-be-detected remote sensing image is input into a full convolutional Transformer module to obtain a dense feature map of the compressed semantic map of the remote sensing image.

[0013] The compressed semantic map of the remote sensing image and the dense feature of the compressed semantic map of the remote sensing image are added by element-by-element addition to obtain a remote sensing image fusion feature map.

[0014] The remote sensing image fusion feature map is input into a wide focal point module to obtain an initial ship saliency map.

[0015] In one embodiment, the plurality of saliency maps are spliced in a mosaic enhancement manner.

[0016] In one embodiment, the total ship saliency map is finely detected based on the improved Mask R-CNN network framework to obtain a target detection map, which comprises the following steps:

[0017] The total ship saliency map is converted into a high-dimensional feature vector through a linear embedding operation.

[0018] The high-dimensional feature vector is input into a Swin-Transformer backbone network of the improved Mask R-CNN network framework, and the Swin-Transformer backbone network extracts multi-scale features of the high-dimensional feature vector through a sliding window attention mechanism and performs image splicing operation at each stage to obtain a plurality of hierarchical feature maps.

[0019] The deepest feature map in the plurality of hierarchical feature maps is input into a context enhancement module for global semantic information extraction.

[0020] The global semantic information of the deepest feature map is fused with the original features of the total ship saliency map to obtain a deep semantic and saliency guided fusion feature map.

[0021] The deep semantic and saliency guided fusion feature map is step by step up-sampled, and is sequentially fused with the remaining layer feature maps to obtain a multi-scale feature fusion high-resolution feature map.

[0022] The multi-scale feature fused high-resolution feature map is subjected to a convolution operation to obtain a target detection map.

[0023] In one of the embodiments, in the coarse detection process, the data preprocessing module, the full convolution transformer module and the wide focus module jointly constitute a coarse detection framework, and the total loss function of the coarse detection framework is:

[0024]

[0025] In the formula, α, β and γ all represent hyperparameters, L BCE represents binary cross entropy, L Huber represents Huber loss, L SSIM represents structural similarity index loss;

[0026] wherein,

[0027]

[0028] In the formula, P is a predicted initial ship saliency map, G is a real ship saliency map label, H and W are respectively height and width of a to-be-detected remote sensing image, i represents a row index of the to-be-detected remote sensing image, and j represents a column index of the to-be-detected remote sensing image;

[0029]

[0030] In the formula, δ represents a preset threshold, represents a Huber loss component when |e|≤δ, when |e|>δ, e=P ij -G ij , and e represents an error between the predicted initial ship saliency map and the real ship saliency map label;

[0031]

[0032] In the formula, M represents a total number of windows in the to-be-detected remote sensing image, x k represents pixel content of a kth window in the predicted initial ship saliency map, y k represents pixel content of a window corresponding to x k in the real initial ship saliency map;

[0033]

[0034] wherein, respectively are the mean of x k , y k , respectively are the variance of x k , y k , is the covariance of x k , y k , and c1 and c2 represent the stable constants.

[0035] In one of the embodiments, the improved Mask R-CNN network framework is trained by using a standard multi-task loss function.

[0036] In one of the embodiments, the remote sensing image to be detected includes a ship target to be detected, and the white area in the target detection image identifies the position of the detected ship target.

[0037] In one of the embodiments, the data preprocessing module sequentially includes a normalization layer, two two-dimensional convolution layers and a maximum pooling layer along the data transmission direction.

[0038] The beneficial effects of the present application are:

[0039] (1) The present application uses a double detection framework from coarse to fine, generates a high-quality saliency map in the coarse detection stage to guide the subsequent fine detection, effectively reduces the background interference, so that the fine detection stage can focus more on the potential ship area, and the final detection accuracy can be improved;

[0040] (2) In the coarse detection stage, the full convolution Transformer and the wide focal point module can effectively capture multi-scale context information and global dependency, improve the detection ability of different scale ships, especially small targets, and better distinguish ships from complex backgrounds such as sea waves, clouds, fog and land, thereby reducing the missed detection rate and false alarm rate;

[0041] (3) In the fine detection stage, the improved Mask R-CNN framework is adopted, the Swin-Transformer is introduced as the feature extraction backbone network, the feature representation ability is significantly improved, and the context enhancement module CEM can improve the understanding of the scene, so as to effectively separate the densely parked or partially overlapped ships and accurately outline the contour. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The whole process framework schematic diagram of the sea surface optical remote sensing image ship detection method based on the coarse and fine double detection framework provided by the embodiment of the present application is shown in the figure.

[0043] Figure 2 The structure schematic diagram of the full convolution Transformer module provided by the embodiment of the present application is shown in the figure.

[0044] Figure 3 The flow framework schematic diagram of the fine detection provided for the embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] In one embodiment, as shown in Figure 1 , Figure 1 The overall flow framework schematic diagram of the sea surface optical remote sensing image ship detection method based on the thick and thin double detection framework provided for the embodiment of the present application is shown. The method is applied to a computer device and includes the following steps:

[0047] S101, simultaneously performing coarse detection on multiple to-be-detected remote sensing images to obtain multiple initial ship saliency maps.

[0048] Specifically, the to-be-detected remote sensing image can be collected by a UAV and includes a to-be-detected ship target.

[0049] S102, splicing multiple initial ship saliency maps to obtain a ship total saliency map.

[0050] Specifically, in the present embodiment, the multiple saliency maps are spliced in a mosaic enhancement manner. Splicing multiple saliency maps can enhance the data and further improve the generalization ability of the detection framework and the robustness to target size and background diversity.

[0051] S103, performing fine detection on the ship total saliency map based on an improved Mask R-CNN network framework to obtain a target detection map.

[0052] The improved Mask R-CNN network framework is a network framework obtained after replacing the backbone network in the original Mask R-CNN framework from ResNet to Swin-Transformer and adding a context enhancement module before the feature pyramid.

[0053] Swin-Transformer is used as a feature extraction backbone network. With its powerful hierarchical feature representation capability and efficient modeling of local and global information, it significantly improves the detection and segmentation performance of ships of various sizes and appearances. The context enhancement module CEM aims to aggregate multi-scale context information from different levels of the backbone network, further enriching the feature maps input to the subsequent region proposal, classification, bounding box regression and mask prediction heads.

[0054] The coarse detection of each remote sensing image to be detected includes the following steps:

[0055] S201. Input the remote sensing image to be detected into the data preprocessing module to obtain the compressed semantic map of the remote sensing image.

[0056] Specifically, the data preprocessing module includes, sequentially along the data transmission direction, a normalization layer, two 2D convolutional layers, and a max-pooling layer. The normalization layer performs layer normalization on the remote sensing image to be detected, unifying the distribution across channels and enhancing training stability. The 2D convolutional layers extract local spatial features, such as edge and texture information, and introduce non-linear expressive power. The max-pooling layer performs downsampling, compressing spatial dimensions while preserving significant responses, improving the robustness and computational efficiency of the detection framework.

[0057] S202. Input the compressed semantic map of the remote sensing image to be detected into the fully convolutional Transformer module to obtain the dense feature map of the compressed semantic map of the remote sensing image.

[0058] like Figure 2 As shown in the upper part, in the fully convolutional Transformer module, the remote sensing image compressed semantic map, as input, is divided into multiple small region tokens (usually patches) to adapt to the input requirements of the Transformer structure. Each token is encoded within the dimension C×H×W, forming a local visual representation. The partitioned tokens are input to the global Transformer module, where global contextual information is extracted through a self-attention mechanism. The FCT module internally includes a multi-head attention mechanism and a feedforward network structure, allowing information interaction between different locations. After each stage of processing, important tokens are selected through a token filtering mechanism (e.g., through attention scores, weighted aggregation, etc.) and passed to the next layer to reduce redundant computation. This strategy is repeated along the "downsampling" path, thereby gradually compressing the number of tokens. The entire process consists of multiple stages ( Figure 2 The diagram shows three stages, each including steps such as local token partitioning, Transformer processing, and token filtering.

[0059] Figure 2 The lower half shows the specific structural design of the FCT module, which adopts an improved QKV mechanism and a spatial attention enhancement strategy.

[0060] Firstly, the input feature map to the FCT module has a channel number of C and a spatial size of HxW. In order to effectively extract local features and provide support for subsequent attention mechanisms, the input feature map is first processed by two parallel convolution modules. One is a depthwise convolution (DwConv) with a size of 5x5, which is used to extract the spatial local relationship within each channel. The other is a pointwise convolution (PwConv), which is mainly used for linear combination and information fusion between channels. The outputs of the two convolutions generate three groups of feature tensors, which are used to construct Q (Query), K (Key), and V (Value) in the Transformer.

[0061] Next, the Q and K feature tensors perform element-wise Hadamard product operation, thereby establishing fine-grained spatial correlation at the local feature level. After completing the Hadamard product, the obtained tensor is sent to a group of convolution operations for information integration in the spatial dimension, and a spatial gating tensor is generated through the Sigmoid function. The gating tensor acts on the Value feature to perform weighted modulation, thereby emphasizing the response of key areas while maintaining the integrity of the spatial features.

[0062] Finally, to improve the multi-channel information expression capability, the modulated Q, K, and V features are concatenated in the channel dimension to form a feature map with a dimension of 3C. The high-dimensional feature is then input to a group of 1x1 convolutions (i.e., pointwise convolutions) for channel compression, restoring to the initial channel number C to form the final output feature. It is worth emphasizing that all operations in the above process maintain the spatial size (HxW) unchanged, avoiding information loss and ensuring the expression capability of the network when processing high-resolution images or fine targets.

[0063] S203, the remote sensing image compression semantic map and the dense feature of the remote sensing image compression semantic map are added by element-wise addition to obtain a remote sensing image fusion feature map.

[0064] S204, inputting the remote sensing image fusion feature map into a wide focal point module to obtain an initial ship saliency map.

[0065] The wide focal point module processes the input feature map by parallel using multiple 3x3 dilation convolutions with different dilation rates to create multiple effective receptive fields. The features after concatenating the outputs of the multiple dilation convolutions are subjected to adaptive channel recalibration through channel attention mechanism (Channel Attention), and are fused with the input features through residual connection. The wide focal point module can effectively capture multi-scale context information, enhance the recognition of ship features, and suppress background noise.

[0066] In this embodiment, the coarse detection stage is completed based on a coarse detection framework, which is composed of a full convolutional transformer module (FCT) and a wide focus module (WFB). The FCT is used to extract high-level semantic features, and the WFB can work with it to capture multi-scale context while integrating a cross-attention module (CCAM) to model long-range dependencies between pixels, thereby improving the quality of the saliency map. In general, the use of full convolutional Transformer and wide focus module can effectively capture multi-scale context information and global dependencies, improve the detection capability of different scale ships, especially small targets, and better distinguish ships from complex backgrounds such as sea waves, clouds, fog, and land, thereby reducing the miss rate and false alarm rate.

[0067] In one embodiment, as shown in Figure 3 the improved Mask R-CNN network framework is used to perform fine detection on the ship total saliency map to obtain a target detection map, including the following steps:

[0068] S301, convert the ship total saliency map into a high-dimensional feature vector through a linear embedding operation.

[0069] S302, input the high-dimensional feature vector into the Swin-Transformer backbone network of the improved Mask R-CNN network framework. The Swin-Transformer backbone network extracts multi-scale features of the high-dimensional feature vector through a sliding window attention mechanism, and performs image stitching operation at each stage to obtain a plurality of hierarchical feature maps.

[0070] S303, input the deepest feature map in the plurality of hierarchical feature maps into a context enhancement module for global semantic information extraction.

[0071] S304, fuse the global semantic information of the deepest feature map with the original features of the ship total saliency map to obtain a deep semantic and saliency guided fusion feature map.

[0072] S305, perform step-by-step upsampling on the deep semantic and saliency guided fusion feature map, and sequentially fuse it with the remaining layer feature maps to obtain a multi-scale feature fusion high-resolution feature map.

[0073] Exemplarily, Figure 3The plurality of hierarchical feature maps include a shallow layer Feature S2', a middle layer Feature S3', and a deep layer Feature S4'. Subsequently, the deepest layer Feature S4' is sent to a context enhancement module (CEM), global semantic information of the Feature S4' is extracted through a pooling operation, and is fused with the original features of the ship overall saliency map to enhance the context expression capability thereof. The enhanced deep layer semantic and saliency guided fusion feature map is subjected to step-by-step up-sampling, and is sequentially fused with the middle layer Feature S3' and the shallow layer Feature S2' to obtain a multi-scale feature fusion high-resolution feature map, so as to restore the spatial resolution and improve the detection accuracy.

[0074] In S306, a convolution operation is performed on the multi-scale feature fused high-resolution feature map to obtain a target detection map. The white area in the target detection map is identified as the position of the detected ship target.

[0075] In one of the embodiments, in the coarse detection process, the data preprocessing module, the fully convolutional transformer module, and the wide focus module jointly constitute a coarse detection framework, and the total loss function of the coarse detection framework is as follows:

[0076]

[0077] In the formula, α, β and γ represent hyperparameters, L BCE represents binary cross entropy, L Huber represents Huber loss, L SSIM represents structural similarity index loss;

[0078] wherein,

[0079]

[0080] In the formula, P is a predicted initial ship saliency map, G is a real ship saliency map label, H and W are the height and width of the to-be-detected remote sensing image respectively, i represents the row index of the to-be-detected remote sensing image, and j represents the column index of the to-be-detected remote sensing image.

[0081]

[0082] In the formula, δ represents a preset threshold, represents Huber loss component, when |e|≤δ, when |e|>δ, e=P ij -G ij represents the error of the predicted initial ship saliency map and the real ship saliency map label.

[0083]

[0084] where M denotes the total number of windows in the remote sensing image to be detected, x k denotes the pixel content of the kth window in the predicted initial ship saliency map, y k denotes the pixel content of the corresponding window in the true initial ship saliency map; k

[0085]

[0086] where, are the mean of x k , y k , respectively, are the variance of x k , y k , respectively, is the covariance of x k , y k , and c1 and c2 denote the stable constants.

[0087] In this embodiment, a composite loss function including binary cross-entropy loss, Huber loss and structural similarity index loss is designed for saliency map prediction in the coarse detection stage, which can optimize the quality of the saliency map from multiple angles such as pixel-level accuracy, robustness to outliers and structure information preservation.

[0088] In one embodiment, the improved Mask R-CNN network framework is trained using a standard multi-task loss function, which usually combines the classification loss of region proposal, the bounding box regression loss and the binary cross-entropy loss of mask prediction.

[0089] In a specific embodiment, the method of the present application is compared with YOLOv7, CornerNet, FCOS and SSD300 through comparative experiments. The data set used in the experiment is a total data set composed of Airbus Ship Detection (ASD) Challenge data set, HRSC2016 data set and DOTA data set, which covers a variety of scenes, image resolutions, annotation styles and environmental challenges.

[0090] The experimental environment is: Windows 10 operating system (version number is 22H2, operating system internal version is 19045.3570), CPU model is Intel(R) Core(TM) i5-9400@2.90GHz, running memory is 8GB, and software is PyCharm2021.

[0091] ​The total data set is divided into a training set, a test set and a validation set, and the method of the application and YOLOv7, CornerNet, FCOS and SSD300 are trained and tested. Among them, for the method of the application, the weight hyperparameter of the coarse detection framework compound loss function is set to a = 1.0, b = 1.0, g = 0.5, and the Huber loss is set to d = 1.0 during the training process. The Adam optimizer is used, and the initial learning rate is 1x10 -4 The cosine annealing learning rate scheduling strategy is used, and the batch size is 8. The parameters of the fine detection stage are adjusted on the validation set.

[0092] The different performances of different methods obtained in the experiment are shown in Table 1. As can be seen from Table 1, the weighted average value F1 of the precision and recall of the method of the application, the average value mAP of the target detection accuracy when the IoU threshold is 0.5 0.5 , the precision and recall are better than those of other comparative methods, and the method of the application has high detection accuracy.

[0093] Table 1 Different performances of the method of the application and comparative methods

[0094]

[0095] The above-described embodiments only express several embodiments of the application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which are all within the protection scope of the application. Therefore, the protection scope of the application should be subject to the appended claims.

Claims

1. A ship detection method based on a thick-thin double bounding box framework for sea surface optical remote sensing images, characterized in that, The method comprises the following steps: obtaining a plurality of initial ship saliency maps through coarse detection on a plurality of to-be-detected remote sensing images simultaneously; splicing the plurality of initial ship saliency maps to obtain a total ship saliency map; performing fine detection on the total ship saliency map based on an improved Mask R-CNN network framework to obtain a target detection map; the improved Mask R-CNN network framework is a network framework obtained after replacing the backbone network in the original Mask R-CNN framework with Swin-Transformer and adding a context enhancement module before a feature pyramid; wherein, the coarse detection on each to-be-detected remote sensing image comprises the following steps: inputting the to-be-detected remote sensing image into a data preprocessing module to obtain a remote sensing image compressed semantic map; inputting the to-be-detected remote sensing image compressed semantic map into a full convolution transformer module to obtain a dense feature map of the remote sensing image compressed semantic map; adding the remote sensing image compressed semantic map and the dense feature map of the remote sensing image compressed semantic map through element-by-element addition to obtain a remote sensing image fusion feature map; inputting the remote sensing image fusion feature map into a wide focal point module to obtain an initial ship saliency map.

2. The sea surface optical remote sensing image ship detection method based on the thick-thin double detection frame according to claim 1, characterized in that, The plurality of saliency maps are spliced in a mosaic enhancement mode.

3. The sea surface optical remote sensing image ship detection method based on the thick-thin double detection frame according to claim 2, characterized in that, The fine detection on the total ship saliency map based on the improved Mask R-CNN network framework to obtain the target detection map comprises the following steps: transforming the total ship saliency map into a high-dimensional feature vector through a linear embedding operation; inputting the high-dimensional feature vector into a Swin-Transformer backbone network of the improved Mask R-CNN network framework, the Swin-Transformer backbone network extracts multi-scale features of the high-dimensional feature vector through a sliding window attention mechanism, and performs image splicing operation at each stage to obtain a plurality of hierarchical feature maps; inputting the deepest feature map in the plurality of hierarchical feature maps into a context enhancement module for global semantic information extraction; fusing the global semantic information of the deepest feature map with original features of the total ship saliency map to obtain a deep semantic and saliency guided fusion feature map; performing step-by-step upsampling on the deep semantic and saliency guided fusion feature map, and sequentially fusing it with the remaining layer feature maps to obtain a multi-scale feature fusion high-resolution feature map; performing convolution operation on the multi-scale feature fused high-resolution feature map to obtain the target detection map.

4. The sea surface optical remote sensing image ship detection method based on the thick-thin double detection frame according to claim 3, characterized in that, In the coarse detection process, the data preprocessing module, the full convolution transformer module and the wide focal point module jointly constitute a coarse detection framework, and a total loss function of the coarse detection framework is: where α, β and γ are hyperparameters, L BCE denotes binary cross-entropy, L Huber denotes Huber loss, L SSIM denotes structural similarity index loss; wherein, in the formula, P is a predicted initial ship saliency map, G is a real ship saliency map label, H and W are height and width of the to-be-detected remote sensing image respectively, i represents a row index of the to-be-detected remote sensing image, and j represents a column index of the to-be-detected remote sensing image; wherein δ represents a preset threshold value, represents a Huber loss component when |e|≤δ, when |e|>δ, e=P ij -G ij , e represents the error between the predicted initial ship saliency map and the true ship saliency map annotation. In the formula, M represents the total number of windows in the remote sensing image to be detected, and x k y represents the pixel content of the k-th window in the initial ship saliency map of the prediction. k Represents the true initial ship saliency graph with x k The pixel content of the corresponding window; wherein are the mean values of x k , y k , are the variances of x k , y k , is the covariance of x k , y k , and c1and c2represent the stability constants.

5. The sea surface optical remote sensing image ship detection method based on the thick-thin double detection frame according to claim 3, characterized in that, the improved Mask R-CNN network framework is trained by using a standard multi-task loss function.

6. The sea surface optical remote sensing image ship detection method based on the thick-thin double detection frame according to claim 4 or 5, characterized in that, the to-be-detected remote sensing image comprises a to-be-detected ship target, and a white region in the target detection map identifies a detected ship target position.

7. The sea surface optical remote sensing image ship detection method based on the thick-thin double detection frame according to claim 6, characterized in that, The data preprocessing module comprises, in sequence along a data transmission direction, a normalization layer, two two-dimensional convolution layers and a maximum pooling layer.

Images