A SAR image ship detection method based on convolutional neural network

Abstract

The application provides a SAR image ship detection method based on a convolutional neural network, comprising the following steps: S1, obtaining at least one high-resolution SAR image containing a plurality of ocean ships output by a synthetic aperture radar; S2, respectively pre-processing each high-resolution SAR image to obtain a corresponding pre-processed SAR image; S3, for each pre-processed SAR image, inputting the pre-processed SAR image into a pre-constructed multi-scale feature pyramid network model to detect and instance segment a multi-scale ocean ship, and obtaining a feature result image containing ocean ships of different scales. The beneficial effect is that the application proposes a multi-scale feature pyramid model, which can detect and instance segment small objects from a complex background, and can adaptively select important multi-scale feature maps for detecting and segmenting multi-scale ocean ships.

Description

Technical Field

[0001] This invention relates to the field of radar remote sensing technology, and more specifically, to a method for ship detection in SAR images based on convolutional neural networks. Background Technology

[0002] Synthetic Aperture Radar (SAR) is an active microwave remote sensing imaging radar that can operate around the clock and in all weather conditions. It has a wide range of important applications in military and civilian fields such as maritime surveillance, resource exploration, key area monitoring, and maritime situational awareness, playing a role that cannot be replaced by other remote sensing methods such as optics and infrared.

[0003] In recent years, Convolutional Neural Networks (CNNs) have become a potential solution to the complex multi-scale detection problem in SAR images because they are capable of representing and learning features at multiple scales. Some CNN-based methods for detecting marine vessels have shown good performance on SAR images. In 2017, Lin et al. proposed Feature Pyramid Network (FPN), which has become a standard solution for multi-scale marine vessel detection in SAR images. FPN can use some reasonable semantic features extracted by its backbone network to detect vessels with different sizes and resolutions, resulting in good performance and thus attracting widespread attention.

[0004] However, FPN cannot adaptively learn and select prominent feature scales for multi-scale ship detection in SAR images, leading to some errors and missed detections. It only re-corrects the weights of the input feature map at a single scale, and the single-scale attention mechanism cannot effectively capture all important feature result maps with different scales. Summary of the Invention

[0005] The problem this invention aims to solve is to provide a SAR image ship detection method based on a convolutional neural network that can re-correct the weights of multi-scale feature maps and capture feature result maps of all scales through a multi-scale attention mechanism.

[0006] To address the above problems, this invention provides a SAR image ship detection method based on convolutional neural networks, comprising:

[0007] Step S1: Acquire a high-resolution SAR image containing multiple ocean ships, output by at least one synthetic aperture radar.

[0008] Step S2: Preprocess each of the high-resolution SAR images to obtain a corresponding preprocessed SAR image;

[0009] Step S3: For each preprocessed SAR image, the preprocessed SAR image is input into a pre-constructed multi-scale feature pyramid network model to perform multi-scale ocean ships detection and instance segmentation, and a feature result map containing ocean ships of different scales is obtained.

[0010] Preferably, the preprocessing method in step S2 is data cleaning and data noise reduction.

[0011] Preferably, before performing step S3, a model building process is further included. This model building process obtains the multi-scale feature pyramid network model by embedding multiple convolutional pyramid modules and multiple multi-scale attention mechanism modules into the feature pyramid network. The model building process includes:

[0012] Step A1: Set up four convolutional pyramid modules to perform feature extraction in the first to fourth stages on the preprocessed SAR image to obtain the corresponding feature information;

[0013] Step A2: Connect the output channels of the four convolutional pyramid modules to the feature pyramid network so that the feature pyramid network performs feature fusion on each feature information to obtain a feature fusion image;

[0014] Step A3: Set up four multi-scale attention mechanism modules and connect the output channels of the four multi-scale attention mechanism modules to the output channel of the feature pyramid network to perform multi-scale ocean ships detection and instance segmentation on the feature fusion image, and obtain the feature result map containing ocean ships of different scales.

[0015] Preferably, in step A1, the input channels of each convolutional pyramid module are set to 256, 512, 1024 and 2048 respectively for feature extraction at different stages, and the output channels of each convolutional pyramid module are all set to 256 so that the output channels of each convolutional pyramid module are aligned with the input channels of the feature pyramid network.

[0016] Preferably, each convolutional pyramid module contains three parallel 3×3 convolutional layers and one 1×1 convolutional layer. The expansion rates of each convolutional layer are 1, 2, and 4 to achieve different receptive fields. In step S3, the multi-scale feature pyramid network model extracts features from the preprocessed SAR image under different receptive fields through the 3×3 convolutional layers in the top convolutional pyramid module to obtain the corresponding feature information. After channel dimensionality reduction by the 1×1 convolutional layer, the information is input into the lower convolutional pyramid module for the next step of feature extraction. After each convolutional pyramid module has completed feature extraction, the feature information is input into the feature pyramid network for feature fusion.

[0017] Preferably, in step A3, each of the multi-scale attention mechanism modules contains a different number of cascaded 3×3 convolutional layers. The original cascaded 3×3 convolutional layers with n channels are divided into multiple cascaded small convolutional groups with w channels, and the cascaded small convolutional groups are connected in a hierarchical residual manner. The output channels of each cascaded small convolutional group are sequentially connected to two fully connected layers. In step S3, the multi-scale feature pyramid model performs instance segmentation and feature extraction on the feature fusion image through each of the cascaded small convolutional groups to obtain feature maps of different scales. Then, the attention weights of the feature maps of each cascaded small convolutional group are recalibrated through two fully connected layers and pooling methods, and multiple calibrated feature maps of different scales are obtained based on the attention weights. The calibrated feature maps are then stitched together to obtain the feature result map containing ocean ships of different scales.

[0018] Preferably, step S3 further includes:

[0019] For each of the cascaded small convolutional groups, the feature maps obtained from the cascaded small convolutional groups are subjected to average pooling and global pooling respectively to obtain the corresponding average pooling results and global pooling results. After nonlinear transformation through two fully connected layers and the behavior function, the results are added together to obtain the attention weights corresponding to the cascaded small convolutional groups.

[0020] Preferably, the average pooling result and the global pooling result are obtained using the following calculation formulas:

[0021]

[0022]

[0023] in,

[0024] avg i This represents the average pooling result;

[0025] maxi This represents the global pooling result;

[0026] y i This represents the feature map output by the i-th cascaded small convolutional group;

[0027] H represents the width of the feature map;

[0028] W represents the height of the feature map;

[0029] m represents the preset parameter;

[0030] n represents the preset parameter.

[0031] Preferably, the attention weights and the calibrated feature map are obtained using the following formula:

[0032] att i =w avgi +w maxi

[0033] Y i =y i ×att i

[0034] in,

[0035] att i This represents the attention weight of the cascaded small convolutional group in the i-th group;

[0036] w avgi This represents the nonlinear transformation result of the average pooling result;

[0037] w maxi This represents the nonlinear transformation result of the global pooling result;

[0038] Y i This represents the calibrated feature map of the i-th group of cascaded small convolutional groups;

[0039] y i This represents the feature map output by the i-th cascaded small convolutional group.

[0040] This invention has the following beneficial effects: Based on the FPN model, this invention adds a convolutional pyramid module and a multi-scale attention mechanism module to form a novel multi-scale feature pyramid model. The convolutional pyramid module performs shallow multi-scale feature map extraction on the preprocessed SAR image, and the multi-scale mechanism module performs detection and instance segmentation on the feature maps. Based on multi-scale attention weights, important feature maps are adaptively selected for stitching to obtain the final feature result map. This feature result map can clearly and accurately display ocean ships at different scales. Attached Figure Description

[0041] Figure 1 This is a flowchart of the steps of the present invention;

[0042] Figure 2 This is a flowchart of the model construction process of the present invention;

[0043] Figure 3 This is a schematic diagram of two cascaded 3×3 convolutional layers of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] In a preferred embodiment of the present invention, based on the above-mentioned problems existing in the prior art, a SAR image ship detection method based on a convolutional neural network is provided, such as... Figure 1 As shown, it includes:

[0046] Step S1: Acquire a high-resolution SAR image containing multiple ocean ships, output by at least one synthetic aperture radar.

[0047] Step S2: Preprocess each high-resolution SAR image to obtain a corresponding preprocessed SAR image;

[0048] Step S3: For each preprocessed SAR image, the preprocessed SAR image is input into a pre-constructed multi-scale feature pyramid network model to perform multi-scale ocean ships detection and instance segmentation, and a feature result map containing ocean ships of different scales is obtained.

[0049] Specifically, in this embodiment, considering that Synthetic Aperture Radar (SAR) is an important form of radar widely used in the field of remote sensing for capturing two-dimensional images or creating three-dimensional reconstructions of objects, such as marine vessels and natural landscapes, SAR imaging, as an active imaging sensor type using microwaves, is superior to traditional passive imaging sensors, such as infrared and optical sensors, in many ways because it is less affected by environmental factors such as weather, visible light, and clouds. In marine affairs management, SAR imaging plays an important role because it has the ability to detect hidden objects and can operate in all-day and all-weather environments. With the rapid development of space-based and airborne SAR, such as TerraSAR-X and RADARSAT-2, SAR imaging has been routinely used for marine monitoring, fisheries management, marine traffic control, and marine emergency rescue. All of these inevitably require the participation of marine vessels. Therefore, SAR data analysis related to marine vessels, especially the detection and segmentation of marine vessels, has become an important research direction in the field of remote sensing and is being actively studied in recent years.

[0050] Detecting and segmenting ships from SAR images, specifically detecting their location and delineating their accurate shape, is a challenging task. This challenge stems primarily from the multi-scale characteristics of ships in SAR datasets and the accompanying complex backgrounds. Maritime ships in SAR datasets can be very small and vary in size due to their different categories and dimensions, as well as inherent imaging parameters of SAR imaging, such as resolution and incident angle. The number of maritime ships present in two SAR images can also differ significantly. Furthermore, the presence of near-shore buildings sometimes makes the background in some SAR images very complex, making the analysis of near-shore maritime ships more challenging than that of near-shore maritime ships. In addition, small-sized maritime ships constitute the majority in some well-known SAR datasets, which are derived from high-resolution SAR image datasets. Therefore, the performance of detecting and segmenting small maritime ships is crucial.

[0051] Preferably, to improve the detection of multi-scale marine vessels, some pioneering works have adopted instance segmentation to assist in vessel detection. Su et al. proposed a high-resolution feature extraction network, which was originally designed for instance segmentation of general remote sensing images, while SAR images were only used as test data. It achieved relatively good performance in segmenting marine vessels. However, this model was not specifically designed for SAR image analysis, so it did not take into account the multi-scale characteristics of marine vessels and could not accurately segment multi-scale marine vessels from complex backgrounds. Wei et al. demonstrated a high-resolution SAR image dataset (HRSID) for detecting and segmenting marine vessels, but there was no corresponding instance segmentation method. In order to improve the segmentation performance of multi-scale vessels, attention mechanisms have also been applied to SAR image segmentation. Gao et al. introduced the CBAM module in the feature fusion process of FPN to extract salient features at different scales, thereby enhancing the feature representation ability and reducing the interference of irrelevant information in complex backgrounds. Although CBAM can improve segmentation performance by applying different weights to the input features, as mentioned above, CBAM is a single-scale attention mechanism and cannot effectively solve the problem of multi-scale vessel segmentation in SAR images.

[0052] Preferredly, some pioneering works have attempted to address the problem of simultaneous detection and segmentation of multi-scale marine vessels in SAR images, but there is still considerable room for further improvement. Specifically, Zhang et al. proposed a model specifically for vessel detection and instance segmentation in SAR images, which captures salient contextual information at different background levels by embedding a contextual SENet module in the FPN. However, two aspects limit the performance of this model and other existing models in simultaneous detection and instance segmentation of multi-scale marine vessels. First, as mentioned above, SENet is a single-scale attention mechanism. Second, these FPN-based models mainly use deep semantic information, while shallow high-resolution feature maps contain richer details, which are crucial for detecting small boats in complex backgrounds. Therefore, in this embodiment, a novel multi-scale feature pyramid network model is constructed, which can effectively extract shallow and deep feature information and combine them to achieve multi-scale representation of marine vessels. It also has a novel multi-scale attention mechanism that can adaptively learn and select important features, which will greatly promote the detection and segmentation of multi-scale marine vessels in SAR images.

[0053] Preferably, the MS-FPN (Multi-Scale Feature Pyramid Network) model in this invention is a variant of the FPN model. The FPN model is designed in a top-down manner, using lateral connections to obtain fine-grained feature pyramids. Due to its ability to fuse multi-level features of the backbone network, the FPN model architecture has been widely used in many computer vision applications, including marine vessel detection and SAR image segmentation. However, a significant weakness of FPN and existing FPN-based models is that deep semantic features are used more thoroughly than shallow features, leading to inaccurate detection of small boats. This is because the features of small objects have been erased in the deep layers by pooling operations. Compared to the abstract features extracted in the deep layers, the features extracted by the shallow filters contain more specific feature information, such as edges, textures, and spots, which are more useful for detecting and segmenting small boats. Furthermore, small vessels have more pixels in the shallow feature map than in the deep layers, meaning that more features of small vessels are available for analysis. Therefore, this invention combines the shallow high-resolution feature map with the deep low-resolution feature map using the MS-FPN model, which is of great significance for the detection and segmentation of vessels of different sizes in SAR images.

[0054] In a preferred embodiment of the present invention, the preprocessing method in step S2 is data cleaning and data noise reduction.

[0055] In a preferred embodiment of the present invention, a model building process is further included before executing step S3. This model building process obtains a multi-scale feature pyramid network model by embedding multiple convolutional pyramid modules and multiple multi-scale attention mechanism modules into the feature pyramid network. The model building process is as follows: Figure 2As shown, it includes:

[0056] Step A1: Set up four convolutional pyramid modules to perform feature extraction from the first to the fourth stage on the preprocessed SAR image to obtain the corresponding feature information;

[0057] Step A2: Connect the output channels of the four convolutional pyramid modules to the feature pyramid network so that the feature pyramid network can perform feature fusion on each feature information to obtain a feature fusion image;

[0058] Step A3: Set up four multi-scale attention mechanism modules and connect the output channels of the four multi-scale attention mechanism modules to the output channel of the feature pyramid network to perform multi-scale detection and instance segmentation of ocean ships on the feature fusion image, and obtain feature result maps containing ocean ships of different scales.

[0059] In a preferred embodiment of the present invention, in step A1, the input channels of each convolutional pyramid module are set to 256, 512, 1024 and 2048 respectively for feature extraction at different stages, and the output channels of each convolutional pyramid module are all set to 256 so that the output channels of each convolutional pyramid module are aligned with the input channels of the feature pyramid network.

[0060] Specifically, in this embodiment, the MS-FPN model of the present invention aims to combine shallow and deep feature maps and adaptively select important feature maps from multi-scale feature maps, specifically for the accurate detection and separation of multi-scale ships in SAR images. The MS-FPN model consists of a Convolutional Pyramid (ACP) module and a Multi-Scale Attention Mechanism (MSAM) module. The ACP module extracts feature information of each stage between stage 1 and stage 4. The input channels of each stage of the ACP module are set to [256, 512, 1024, 2048] for each stage of the backbone network. The number of input channels is aligned, and the number of output channels is set to 256 for the top-down input channel alignment of the FPN model.

[0061] Preferably, the feature information extracted by each convolutional pyramid module is added to the top-down FPN model for feature fusion. Finally, the input and output channels of the MSAM module are set to 256 to keep in line with the output channels of the top-down FPN model. For example, the ACP module is used to extract the shal-low feature map of the second stage. After upsampling the second layer in the feature pyramid, additive fusion is performed at the first layer scale, and then the MSAM module is used for final prediction.

[0062] Preferably, the MS-FPN model has three advantages over the FPN model: First, by utilizing the ACP module, which has the ability to extract multi-scale features, it can extract as much feature information as possible from shallow high-resolution images to improve the detection and segmentation of small ships; Second, the MSAM module can assign greater weights to certain feature maps that are important for ship detection and segmentation, thereby helping to eliminate interference from complex backgrounds in SAR images; Third, both the ACP and MSAM modules are multi-scale modules, which can better handle geometric problems such as scale transformation, which will further improve the ability of multi-scale ship detection and segmentation.

[0063] In a preferred embodiment of the present invention, each convolutional pyramid module includes three parallel 3×3 convolutional layers and one 1×1 convolutional layer. The expansion rates of each convolutional layer are 1, 2 and 4 to achieve different receptive fields. In step S3, the multi-scale feature pyramid network model extracts features from the preprocessed SAR image under different receptive fields through the 3×3 convolutional layers in the top convolutional pyramid module to obtain the corresponding feature information. After channel dimensionality reduction through the 1×1 convolutional layer, the information is input into the lower convolutional pyramid module for the next step of feature extraction. After each convolutional pyramid module has completed feature extraction, the feature information is input into the feature pyramid network for feature fusion.

[0064] Specifically, in this embodiment, since the FPN model only uses 1×1 convolutional layers to extract shallow feature maps, the visual field is not large enough, and the shallow features are not fully utilized. In this embodiment, the ACP module is used to obtain complex contextual information from the last layer of each stage for detecting small boats. This is achieved by using convolutional kernels with different dilation rates. The core idea of ​​the ACP module is to use parallel branches of multi-scale receptive fields, represented by different dilation rates, to extract multi-scale contextual information. The ACP module has three parallel 3×3 convolutional layers with dilation rates of 1, 2, and 4, which mean different receptive fields. They add multi-scale background information for detecting small objects. According to the characteristics of small boats in SAR images, using different dilation rates will reduce the background noise that may be caused by using only a large dilation rate.

[0065] Preferably, these three parallel 3×3 convolutional layers can be represented as follows:

[0066]

[0067] out = Conv 1×1,d＝1 (y)

[0068] Where d represents the expansion rate, It is a concatenation operation, where x removes the input image and out represents the output image. By using three parallel 3×3 convolutional layers with different dilation rates, the ACP module can increase the resensory field of the proposed model, which can significantly improve the performance of detecting small marine vessels. Then, these features are combined and refined by adding the elements of the three parallel 3×3 convolutional layers. The final 1×1 convolutional layer performs channel dimensionality reduction to keep the output with the same channel dimension as the input of the next step.

[0069] Preferably, as can be seen from the above, using the ACP module has two advantages: (1) it increases the receptive field without sacrificing any detail information; (2) using three parallel channels can refine the low-level feature representation and transfer background information from the backbone to the FPN model, which is very useful for detecting and segmenting small boats.

[0070] In a preferred embodiment of the present invention, in step A3, each multi-scale attention mechanism module contains a different number of cascaded 3×3 convolutional layers. The original cascaded 3×3 convolutional layers with n channels are divided into multiple cascaded small convolutional groups with w channels, and the cascaded small convolutional groups are connected in a hierarchical residual manner. The output channels of each cascaded small convolutional group are sequentially connected to two fully connected layers. In step S3, the multi-scale feature pyramid model performs instance segmentation and feature extraction on the feature fusion image through each cascaded small convolutional group to obtain feature maps of different scales. Then, the attention weights of the feature maps of each cascaded small convolutional group are recalibrated through two fully connected layers and pooling methods, and multiple calibrated feature maps of different scales are obtained based on the attention weights. The calibrated feature maps are then stitched together to obtain a feature result map containing ocean ships of different scales.

[0071] Specifically, in this embodiment, another problem with the FPN model is that after extracting multi-scale feature information, it is unclear which layers of feature maps are more useful for detecting and segmenting multi-scale ocean vessels. Single-scale attention mechanism models, such as SENet and CBAM, cannot distinguish feature maps obtained from different scales. However, multi-scale feature maps are very important for the detection and segmentation of multi-scale vessels in SAR images. Multi-scale feature maps can be used to improve the performance of vessel quantification in two ways: 1) distinguishing vessels from the background by using a larger receptive field; 2) perceiving information about small vessels by using a smaller receptive field. For example, the ocean is a useful background containing background information that can be used to determine whether a small object placed above the ocean is a ship or a building. Therefore, in order to learn and select the most important features from multi-scale feature maps, designing a multi-scale attention mechanism is particularly important for the detection and segmentation of ocean vessels in SAR images.

[0072] Specifically, in this embodiment, in order to solve the problem of segmenting multi-scale ships in complex backgrounds in SAR images and the problem that the current single attention mechanism model is not efficient in utilizing multi-scale information, this embodiment proposes the MSAM module, a multi-scale attention mechanism module. By adopting a hierarchical structure similar to residuals and two fully connected layers, it effectively utilizes multi-scale spatial information and improves the detection and segmentation of multi-scale ships.

[0073] Preferred, such as Figure 3 As shown, the receptive field of a single 5×5 convolution is equivalent to that of two cascaded 3×3 convolutions. The number of parameters in these two types of convolutions is determined by the following factors:

[0074] params = C in ×k 2 ×C out

[0075] Where C out C represents the number of output channels. in represents the number of input channels, and k represents the size of the convolution kernel. For the same receptive field, two 3×3 convolutions have far fewer parameters than a 5×5 convolution. Therefore, cascaded 3×3 convolutions can be used instead of large convolution kernels to obtain the same receptive field. Similar to two cascaded 3×3 convolutions, the receptive field of three cascaded 3×3 convolutions is equivalent to that of a 7×7 convolution.

[0076] In a preferred embodiment of the present invention, step S3 further includes:

[0077] For each cascaded small convolutional group, the feature maps obtained from the cascaded small convolutional group are subjected to average pooling and global pooling respectively to obtain the corresponding average pooling results and global pooling results. After nonlinear transformation through two fully connected layers and the behavior function, the results are added together to obtain the attention weights corresponding to the cascaded small convolutional group.

[0078] In a preferred embodiment of the present invention, the average pooling result and the global pooling result are obtained by the following calculation formulas:

[0079]

[0080]

[0081] in,

[0082] avg i This represents the average pooling result;

[0083] max i This represents the result of global pooling;

[0084] y iThis represents the feature map output by the i-th cascaded mini-convolutional group;

[0085] H represents the width of the feature map;

[0086] W represents the height of the feature map;

[0087] m represents the preset parameter;

[0088] n represents the preset parameter.

[0089] In a preferred embodiment of the present invention, the attention weights and calibrated feature maps are obtained using the following calculation formula:

[0090] att i =w avgi +w maxi

[0091] Y i =y i ×att i

[0092] in,

[0093] att i This represents the attention weights of the i-th cascaded small convolutional group;

[0094] w avgi This represents the nonlinear transformation result of the average pooling result;

[0095] w maxi This represents the non-linear transformation result of global pooling.

[0096] Y i This represents the calibrated feature map of the i-th group of cascaded small convolutional groups;

[0097] y i This represents the feature map output by the i-th cascaded small convolutional group.

[0098] Specifically, in this embodiment, the operation is divided into three steps:

[0099] The first step involves extracting multi-scale feature maps by using different numbers of cascaded mini-convolutional groups. The original 3×3 convolutional layer with n channels is divided into several cascaded mini-convolutional groups with w channels (e.g., n = s × w). These cascaded mini-convolutional groups are connected in a hierarchical residual manner. The output of each cascaded mini-convolutional group can be represented as:

[0100]

[0101] Among them, y i Let x represent the feature map output by the i-th cascaded small convolutional group. i Let k represent the input of the i-th cascaded small convolutional group.i This represents the kernel size of the i-th cascaded small convolutional group. In each cascaded small convolutional group, due to the different number of 3×3 convolutional kernels, the input feature map of the same receptive field will eventually produce an equivalent multi-scale feature map.

[0102] The second step involves recalibrating the weights of the feature maps of each cascaded small convolutional group using two fully connected layers and pooling methods to obtain the channel correlation between multi-scale features. Average pooling and global pooling are then performed on the feature maps of each cascaded small convolutional group to obtain the global receptive field, and finally, the calibrated feature maps are obtained.

[0103] The third step is to stitch together the multi-scale feature maps to obtain a feature map of ocean-going vessels at different scales, which can be expressed as follows:

[0104] out = concat([Y0,Y1,...,Y...) s-1 ]).

[0105] By using the MSAM module, feature maps at different scales can be extracted to obtain more background information about multi-scale marine vessels.

[0106] Example 1:

[0107] We used the publicly available, accurately annotated dataset HRSID to train and test the performance of the MS-FPN model on ship detection and instance segmentation. HRSID was chosen because it is the first SAR dataset to support in-station segmentation of marine vessels. This dataset contains 136 panoramic SAR images with resolutions between 1 and 5 meters, from three different satellites: Sentinel-1B, Ter-raSAR-X, and TanDEM. These panoramic SAR images were cropped to 800x800 pixels with an overlap of 25%. The dataset contains a total of 5,604 cropped synthetic aperture radar images and 16,951 annotated ships.

[0108] Example 2:

[0109] The performance of the MS-FPN model was compared with several classic models, including the FPN model, the FPN-CARAFE model, the HRFPN model, and the PAFPN model, using the same environment configuration to ensure fairness in the comparison. Table 1 summarizes the quantitative comparison results of detection and segmentation performance. From these results, we found that the FPN model performs the worst in multi-scale marine vessel detection and segmentation. The HRFPN model with deep high-resolution representation learning, the PAFPN model with path enhancement, and the CARAFE model with content-aware reconstruction algorithm show limited improvement in vessel detection and segmentation, and they cannot incorporate semantic information from different layers. The MS-FPN model outperforms the other models overall because the MS-FPN model can fully utilize the semantic and detailed information of all layers deeper than a certain layer, and can obtain high-resolution feature maps with more detailed semantic information.

[0110] Visually, the proposed MS-FPN model also demonstrates state-of-the-art performance in detecting and segmenting multi-scale marine vessels in complex backgrounds on the SAR HRSID dataset. Compared to ground-based data, the FPN model exhibits some false detections, missed detections, and poor segmentation. The FPN-CARAFE model also shows a reasonable number of missed detections, with land clutter being incorrectly detected as a marine vessel. Similarly, the HRFPN and PAFPN models also show false detections, missed detections, and imperfect segmentation. Compared to other models, the proposed MS-FPN model exhibits fewer false detections and missed detections, and its segmentation is more accurate with smoother contours. Only the MS-FPN model can detect the position of the vessel and correctly segment its contour. Therefore, the proposed MS-FPN model not only performs better in detecting and segmenting large marine vessels but also in detecting and segmenting small vessels surrounded by complex backgrounds. The performance improvement of the MS-FPN model is due to the effective extraction of low-level feature maps by the ACP module and the fusion of these feature maps by the MSAM module. These feature maps can be combined with deep feature maps, resulting in significant improvements.

[0111] Table 1. Detection and segmentation performance of FPN, FPN-Carafe, HRFPN, DAFPN, and MS-FPN models.

[0112]

[0113] Example 3:

[0114] To further evaluate which module (ACP or MSAM) contributes more to the improvement of the proposed MS-FPN model, Mask R-CNN was used as the base model, and the performance of models without ACP and MSAM modules, models using only ACP module, models using only MSAM module, and models using both ACP and MSAM modules were compared. Table 2 summarizes the quantitative comparison results of ship detection and segmentation. Compared with models without ACP and MSAM modules, models using only ACP or MSAM modules showed better performance in both detection and segmentation. In addition, models using both ACP and MSAM modules showed better performance than the other three models mentioned above. This result indicates that using ACP or MSAM modules alone can improve the model performance. The MSAM module contributes slightly more to the proposed MS-FPN model, and their combination can make full use of shallow feature information and select multi-scale feature maps that are more conducive to ship detection and segmentation, thus further improving the model performance.

[0115] To visually compare the effectiveness of each module, ablation experiments were conducted on the ACP module, the MSAM module, and the ACP & MSAM module. Compared to ground-level data, the model using only the ACP module showed false positives and false negatives, the model using only the MSAM module showed a reasonable number of false positives and inaccurate segmentation, and the model using both the ACP and MSAM modules showed fewer false positives and false negatives, more accurate segmentation, and smoother contours. Overall, these results demonstrate that the combination of the ACP and MSAM modules can better utilize the network's multi-scale feature maps.

[0116] Table 2 shows the performance of models with MSAM or ACP modules.

[0117]

[0118] Example 4:

[0119] A detailed performance evaluation of each module (ACP module and MSAM module) was conducted. First, the ACP module was compared with the classic multi-scale feature extraction structure ASPP. The results are summarized in Table 3. It can be seen that the AP value of ASPP is lower than that of ACP in both detection and isolation. At the same time, the AP values ​​of ACP module and ASPP are higher than those of the base model Mask R-CNN. This comparison shows that: 1) shallow high-resolution features are useful for the detection and segmentation of multi-scale ships in SAR images; 2) ACP has better performance than ASPP. The fundamental reason is that ASPP uses a larger dilation rate than ACP, so ASPP has a larger receptive field, which may lead to a higher background noise level. This also confirms that although increasing the receptive field can improve the performance of the base model, an excessively large receptive field will negatively affect the performance of multi-scale ship detection and segmentation in SAR images.

[0120] Table 3 Comparison of ACP and ASPP modules

[0121]

[0122] Example 5:

[0123] The proposed MSAM module is compared with several classic attention mechanisms, including ECA-Net, SENet, CBAM, and CA. Table 4 summarizes the quantitative comparison results of marine vessel detection and segmentation. Compared with models that do not use attention mechanisms, models using one of the four attention mechanisms (ECA-Net, SENet, CBAM, and CA) show improved performance in detection and segmentation. The model using MSAM shows better performance than models using other attention mechanisms. This comparison leads to two conclusions: first, connecting the attenuation mechanism to the pyramid network is effective for detecting multi-scale vessels in SAR images; second, compared with the aforementioned single-scale attention mechanisms, the multi-scale attention mechanism MSAM is more powerful in detecting and segmenting multi-scale marine vessels in SAR images.

[0124] Table 4 Comparison of MSAM and Classical Attention Mechanisms

[0125]

[0126] Example 6:

[0127] To explore the impact of receptive field on model detection and segmentation performance, different scale factors were used to test the performance. These scale factors can obtain a receptive field of a specific size during convolution, as shown in Table 5. The size of the receptive field is important for model detection and segmentation. The ship detection and segmentation performance at scale 4 is the best, at 0.4 AP and 0.2 AP respectively, which is higher than that at scale 2, at 0.6 AP and 0.2 AP respectively, still higher than the baseline. Compared with scale 4, scale 8 shows less improvement, which may be due to the additional noise introduced by the excessively large receptive field in the SAR image.

[0128] Table 5. Impact of Scale Changes on Model Performance

[0129]

[0130] Example 7:

[0131] Since pooling is an important component of the MSAM module, several different types of pooling functions were used to further investigate the impact of pooling on model performance, as shown in Table 6. Pooling functions are important for model performance because models with pooling functions show performance improvements to varying degrees. Compared with the base model, which has neither attention mechanisms nor pooling functions, using the max pooling function did not show significant improvement, but using the average pooling function improved detection and segmentation by 1.4 AP and 0.9 AP, respectively. The hybrid pooling of max and average functions had the highest improvements in detection and segmentation, at 1.4 AP and 1.0 AP, respectively. This comparison shows that hybrid pooling can obtain more useful information about multi-scale ships than single pooling.

[0132] Table 6. Impact of Pooling Functions on Model Performance

[0133]

[0134] While the disclosure is as stated above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the protection scope of this invention.

Claims

1. A method for ship detection in SAR images based on convolutional neural networks, characterized in that, include: Step S1: Acquire a high-resolution SAR image containing multiple ocean ships, output by at least one synthetic aperture radar. Step S2: Preprocess each of the high-resolution SAR images to obtain a corresponding preprocessed SAR image; Step S3: For each preprocessed SAR image, the preprocessed SAR image is input into a pre-constructed multi-scale feature pyramid network model to perform multi-scale ocean ships detection and instance segmentation, and a feature result map containing ocean ships of different scales is obtained. Before performing step S3, a model building process is also included. This model building process obtains the multi-scale feature pyramid network model by embedding multiple convolutional pyramid modules and multiple multi-scale attention mechanism modules into the feature pyramid network. The model building process includes: Step A1: Set up four convolutional pyramid modules to perform feature extraction in the first to fourth stages on the preprocessed SAR image to obtain the corresponding feature information; Step A2: Connect the output channels of the four convolutional pyramid modules to the feature pyramid network so that the feature pyramid network performs feature fusion on each feature information to obtain a feature fusion image; Step A3: Set up four multi-scale attention mechanism modules and connect the output channels of the four multi-scale attention mechanism modules to the output channel of the feature pyramid network to perform multi-scale ocean ships detection and instance segmentation on the feature fusion image, and obtain the feature result map containing ocean ships of different scales.

2. The SAR image ship detection method according to claim 1, characterized in that, The preprocessing methods in step S2 are data cleaning and data noise reduction.

3. The SAR image ship detection method according to claim 1, characterized in that, In step A1, the input channels of each convolutional pyramid module are set to 256, 512, 1024 and 2048 respectively for feature extraction at different stages, and the output channels of each convolutional pyramid module are all set to 256 so that the output channels of each convolutional pyramid module are aligned with the input channels of the feature pyramid network.

4. The SAR image ship detection method according to claim 1, characterized in that, Each of the convolutional pyramid modules contains three parallel 3×3 convolutional layers and one 1×1 convolutional layer. The expansion rates of each convolutional layer are 1, 2, and 4 to achieve different receptive fields. In step S3, the multi-scale feature pyramid network model extracts features from the preprocessed SAR image under different receptive fields through the 3×3 convolutional layers in the top convolutional pyramid module to obtain the corresponding feature information. After channel dimensionality reduction by the 1×1 convolutional layer, the information is input into the lower convolutional pyramid module for the next step of feature extraction. After each convolutional pyramid module has completed feature extraction, the feature information is input into the feature pyramid network for feature fusion.

5. The SAR image ship detection method according to claim 1, characterized in that, In step A3, each of the multi-scale attention mechanism modules contains a different number of cascaded 3×3 convolutional layers. The original cascaded 3×3 convolutional layers with n channels are divided into multiple cascaded small convolutional groups with w channels, and the cascaded small convolutional groups are connected in a hierarchical residual manner. The output channels of each cascaded small convolutional group are sequentially connected to two fully connected layers. In step S3, the multi-scale feature pyramid model performs instance segmentation and feature extraction on the feature fusion image through each of the cascaded small convolutional groups to obtain feature maps of different scales. Then, the attention weights of the feature maps of each cascaded small convolutional group are recalibrated through two fully connected layers and pooling methods, and multiple calibrated feature maps of different scales are obtained based on the attention weights. The calibrated feature maps are then stitched together to obtain the feature result map containing ocean ships of different scales.

6. The SAR image ship detection method according to claim 5, characterized in that, Step S3 further includes: For each of the cascaded small convolutional groups, the feature maps obtained from the cascaded small convolutional groups are subjected to average pooling and global pooling respectively to obtain the corresponding average pooling results and global pooling results. After nonlinear transformation through two fully connected layers and the behavior function, the results are added together to obtain the attention weights corresponding to the cascaded small convolutional groups.

7. The SAR image ship detection method according to claim 6, characterized in that, The average pooling result and the global pooling result are obtained using the following formulas: ； in, This represents the average pooling result; This represents the global pooling result; Indicates the first The feature map output by the cascaded small convolutional groups; This represents the width of the feature map; Indicates the height of the feature map; Indicates preset parameters; This indicates the preset parameters.

8. The SAR image ship detection method according to claim 6, characterized in that, The attention weights and the calibrated feature map are obtained using the following formulas: ； in, Indicates the first The attention weights of the cascaded small convolutional groups; This represents the nonlinear transformation result of the average pooling result; This represents the nonlinear transformation result of the global pooling result; Indicates the first The calibrated feature map of the cascaded small convolutional groups; Indicates the first The feature map output by the cascaded small convolutional groups.

Images