SAR image ship instance segmentation method based on cross-scale attention

Abstract

The application discloses a SAR image ship instance segmentation method based on cross-scale attention, takes an image and real categories, real coordinates and a real target mask of a target in the image as training samples, extracts multiple groups of training samples from a data set, constructs a CARSNet network structure based on cross-scale attention, inputs the training samples into the CARSNet network structure, obtains a trained CARSNet network, and performs SAR image ship instance segmentation through the trained CARSNet network structure. In the segmentation process, the cross-scale attention module is used to supplement stable detailed information for high-scale features, and the segmentation effect of the algorithm on the contour of a ship target is improved. The application adopts a positive sample sampling method based on Gaussian distribution, the method can adaptively generate more positive samples containing a ship head and tail according to the geometric characteristics of the ship target, and is helpful to the learning of the network on the overall characteristics of the ship.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a method for segmenting ship instances in SAR images based on cross-scale attention. Background Technology

[0002] As an active microwave imaging sensor, Synthetic Aperture Radar (SAR) coherently images targets by emitting electromagnetic pulses and receiving the echoes. Utilizing electromagnetic microwave transmission, it is virtually unaffected by weather, cloud cover, or time of day, enabling the acquisition of clear images of ground targets around the clock. Furthermore, its longer wavelength than optical waves provides stronger penetration, allowing it to penetrate clouds, vegetation, and smoke. Due to these advantages, SAR is widely used in both military and civilian fields. In civilian applications, SAR can be used for monitoring illegal vessels at sea, environmental monitoring, urban planning, and mapping. In the military field, SAR can detect key military targets, providing precise coordinate information for weapon systems. Because of SAR's unique advantages, countries have long used SAR satellites to observe naval vessels. Meanwhile, instance segmentation technology can depict clearer outline details of targets, thus finding wide application in autonomous driving, medical image analysis, facial recognition, video analysis, industrial automation, and remote sensing image analysis. Compared to target detection, ship instance segmentation in SAR images can obtain more accurate ship location information and contour details. Therefore, ship instance segmentation in SAR images has significant practical value and broad application prospects in both civilian and military fields.

[0003] The SRNet algorithm uses only a single layer of feature map from the FPN network for segmentation in the segmentation head. Due to the lack of detailed information in the high-scale feature map, the contour segmentation effect of the large and medium targets generated by the algorithm is very coarse, which results in the inability to provide the precise location of this type of target. Summary of the Invention

[0004] The purpose of this invention is to provide a SAR image ship instance segmentation method based on cross-scale attention, which solves the problem that the lack of detailed information on the feature map leads to the inability to provide the accurate location of this type of target, thus affecting the segmentation effect.

[0005] The technical solution adopted in this invention is a SAR image ship instance segmentation method based on cross-scale attention, which is implemented according to the following steps: Step 1: Take the image and the true class of the target in the image, the true coordinates of the target, and the true target mask as a set of training samples, and extract multiple sets of training samples from the dataset; Step 2: Construct the CARSNet network structure based on cross-scale attention; Step 3: Input the training samples into the CARSNet network structure based on cross-scale attention to obtain the trained CARSNet network; Step 4: Segment ship instances in SAR images using the trained CARSNet network structure.

[0006] The invention is further characterized by: In step 1, the target's true coordinates include the coordinates of the target's center point in the image, the target's width and height, and the target's rotation angle coordinates.

[0007] The dataset in step 1 is either the Instance-RSDD dataset or the SSDD dataset.

[0008] The CARSNet network structure based on cross-scale attention in step 2 includes: The ResNet network is used to extract low-scale feature maps from the input image. Feature extraction networks are used to extract multi-scale feature maps from input images; A rotating target detection network is used to predict the center point coordinates, rotation angle, width and height, and class score of a rotating target bounding box using multi-scale feature maps. The instance segmentation network includes a cross-scale attention module and a segmentation head, which samples the feature map within the rotated detection box in 14×14 format and supplements the sampled feature map with detailed information from the low-scale feature map. The supplemented feature map is then input into the segmentation head to predict the segmentation mask.

[0009] The 14×14 sampling of the rotating detection frame is performed using a bilinear interpolation algorithm.

[0010] The specific process for extracting low-scale feature maps from the input image is as follows: After the image is input into the ResNet network, the outputs of stage 0 and stage 1 of the ResNet network are used as features C0 and C1, respectively. The output features C0 and C1 are fused using formula (1) to obtain the fused feature C. F The fusion formula is: (1) Where Conv represents a 1×1 convolution and DeConv represents a 4×4 transposed convolution; Fusing feature C F As a low-scale feature map of the image.

[0011] Step 3 is as follows: Step 3.1: Input the training samples into the ResNet network and output the low-scale feature maps of the training sample images; Step 3.2: Input the training samples into the feature extraction network and output the multi-scale feature map of the training sample image; Step 3.3: Input the multi-scale feature map of the training sample image into the rotating target detection network, convert the ground truth bounding box of the target into a two-dimensional Gaussian distribution, and use the positive sample sampling method based on Gaussian distribution to select positive sample anchor boxes and negative sample anchor boxes from the anchor boxes, and predict the rotating detection box of the target, the width and height of the rotating detection box, and the rotating detection box category score. Step 3.4: Select the single-layer feature map of the information to be supplemented from the multi-scale feature map according to the width and height of the rotation detection box. Input the single-layer feature map of the information to be supplemented and the rotation detection box into the instance segmentation network. The single-layer feature map in the rotation detection box is sampled in 14×14 through the cross-scale attention module. The detailed information on the low-scale feature map is supplemented into the sampled single-layer feature map to obtain the feature map of the supplemented information. Step 3.5: Input the feature map of the supplementary information into the segmentation head to predict the segmentation mask; Step 3.6: Introduce three loss functions: a classification prediction loss function that measures the difference between the predicted class score and the true class, a regression prediction loss function that measures the difference between the predicted rotated bounding box and the true target coordinates, and a mask prediction loss function that measures the difference between the predicted segmentation mask and the true target mask. Add the values ​​obtained from these three loss functions together and return to step 3.1. Reduce the total loss value using the stochastic gradient descent algorithm until, after 36 training rounds, the CARSNet network parameters corresponding to the minimum total loss value are obtained, thus obtaining the trained CARSNet network.

[0012] Step 3.3 The formula for converting the target's ground truth bounding box into a two-dimensional Gaussian distribution is: (2) in, express p Gaussian distribution values ​​of the points Represents the center point coordinates of the ship target , , , p This represents the coordinates of the center point of an anchor frame. Represents the inverse transformation of a matrix. This indicates finding the determinant of a matrix. This represents the target's rotation angle coordinates.

[0013] The specific process of selecting a single-layer feature map from the multi-scale feature map based on the width and height of the rotated detection box is as follows: Select a single-layer feature map from the multi-scale feature map based on the width and height of the rotated detection box. F kThe hierarchy of a single-layer feature map in a multi-scale feature map k The calculation expression is: (3) in, k 0 indicates the lowest scale of the multi-scale feature map. w , h These represent the width and height of the rotating detection frame, respectively. Select k Corresponding single-layer feature map F k This serves as a single-layer feature map for information to be supplemented.

[0014] Step 3.4 is as follows: Based on the width and height of the rotation detection box, select a single-layer feature map of the information to be supplemented from the multi-scale feature map. Input the single-layer feature map of the information to be supplemented and the rotation detection box into the instance segmentation network. Use the bilinear interpolation algorithm to perform 14×14 feature sampling on the low-scale feature map of the image and the single-layer feature map of the information to be supplemented within the rotation detection box of the target, and obtain the high-scale feature map F. H and detailed information feature map F L The high-scale feature map F is processed through a fully connected network with a cross-scale attention module. H Mapped to a query matrix Q H , to extract detailed information feature map F L Mapped to the bond matrix K respectively L Sum matrix V L By querying matrix Q H With bond matrix K L The high-scale feature map F is obtained by multiplying the matrices. H and detailed information feature map F L The similarity matrix, the similarity matrix and the value matrix V L Multiplication yields a stable feature map F containing detailed information. D The expression is: (4) Where Linear represents a fully connected network used for dimension transformation, and d represents the dimension of the query matrix; The feature map with supplementary detail information is obtained by adding the stable detail information feature map FD to the high-scale feature map FH element by element.

[0015] The beneficial effects of this invention are: This invention employs a cross-scale attention module to supplement stable detail information for high-scale features, thereby improving the algorithm's segmentation effect on ship target contours.

[0016] This invention employs a positive sample sampling method based on Gaussian distribution. This method can adaptively generate more positive samples containing the bow and stern of the ship according to the geometric characteristics of the ship itself, which helps the network learn the overall features of the ship. Attached Figure Description

[0017] Figure 1 This is a block diagram of the CARSNet network structure based on cross-scale attention in this invention. Figure 2 This is a schematic diagram of the ResNet network used in this invention to extract low-scale feature maps of the input image; Figure 3 This is a schematic diagram of the feature extraction network used in this invention to extract multi-scale feature maps; Figure 4 This is a block diagram of the instance segmentation network structure used in this invention; Figure 5 This is a structural block diagram of the cross-scale attention module in this invention; Figure 6 This is a map showing the true annotation results of SAR images in the SSDD dataset; Figure 7 The image shows the segmentation results from SRNet. Figure 8 This is a diagram showing the segmentation result of the present invention.

[0018] Figure 9 The image shows the ground truth annotation results for SAR images in the Instance-RSDD dataset; Figure 10 The image shows the segmentation results from SRNet. Figure 11 This is a diagram showing the segmentation result of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 This invention relates to a SAR image ship instance segmentation method based on cross-scale attention, which is implemented according to the following steps: Step 1: Take the image and the true class of the target in the image, the true coordinates of the target, and the true target mask as a set of training samples, and extract multiple sets of training samples from the dataset; The true coordinates of the target include the coordinates of the target's center point in the image, the target's width and height, and the target's rotation angle coordinates.

[0021] The dataset is either the Instance-RSDD dataset or the SSDD dataset.

[0022] Step 2: Construct the CARSNet network structure based on cross-scale attention; the CARSNet network structure based on cross-scale attention is as follows: Figure 1 As shown, it includes: The ResNet network is used to extract low-scale feature maps from the input image; the specific process is as follows: After the image is input into the ResNet network, the outputs of stage 0 and stage 1 of the ResNet network are used as features C0 and C1, respectively. The output features C0 and C1 are fused using formula (1) to obtain the fused feature C. F The fusion formula is: (1) Where Conv represents a 1×1 convolution and DeConv represents a 4×4 transposed convolution; Fusing feature C F As a low-scale feature map of the image.

[0023] Feature extraction networks are used to extract multi-scale feature maps from input images; A rotating target detection network is used to predict the center point coordinates, rotation angle, width and height, and class score of a rotating target bounding box using multi-scale feature maps. Instance segmentation networks include cross-scale attention modules and segmentation heads such as... Figure 4 As shown, a 14×14 sampling method is used to sample the feature map within the rotated detection box, and the detailed information on the low-scale feature map is added to the sampled feature map. The feature map with the added information is then input into the segmentation head to predict the segmentation mask.

[0024] The 14×14 sampling of the rotating detection frame is performed using a bilinear interpolation algorithm.

[0025] Step 3: Input the training samples into the CARSNet network structure based on cross-scale attention to obtain the trained CARSNet network; the specific process is as follows: Step 3.1: Input the training samples into the ResNet network and output the low-scale feature maps of the training sample images; Step 3.2: Input the training samples into the feature extraction network and output the multi-scale feature map of the training sample image; Step 3.3: Input the multi-scale feature map of the training sample image into the rotating target detection network, convert the ground truth bounding box of the target into a two-dimensional Gaussian distribution, and use the positive sample sampling method based on Gaussian distribution to select positive sample anchor boxes and negative sample anchor boxes from the anchor boxes, and predict the rotating detection box of the target, the width and height of the rotating detection box, and the rotating detection box category score. During the training phase of the rotating target detection network, this invention designs a positive sample sampling method based on Gaussian distribution, which is used to sample the true bounding box coordinates of a ship target. The distribution is transformed into a two-dimensional Gaussian distribution using the following formula (2): (2) in, express p Gaussian distribution values ​​of the points Represents the coordinates of the center point of the ship target. , , , p This represents the coordinates of the center point of an anchor frame. Represents the inverse transformation of a matrix. This indicates finding the determinant of a matrix. This represents the target's rotation angle coordinates.

[0026] For each anchor box, calculate the value of its center point on the 2D Gaussian distribution of the ground truth box, and then retain the anchor boxes with the top k largest values. Calculate the IoU between these anchor boxes and the ground truth box, and calculate the mean m and variance g of these IoUs. Set the IoU threshold for selecting positive samples: t = m + g. Finally, determine whether an anchor box is a positive sample based on whether its IoU value with the ground truth box is greater than t.

[0027] Step 3.4: Select the single-layer feature map from the multi-scale feature map based on the width and height of the rotated detection box. The specific process is as follows: Select a single-layer feature map from the multi-scale feature map based on the width and height of the rotated detection box. F k The hierarchy of a single-layer feature map in a multi-scale feature map k The calculation expression is: (3) in, k 0 indicates the lowest scale of the multi-scale feature map. w , h These represent the width and height of the rotating detection frame, respectively. Select k Corresponding single-layer feature map F k This serves as a single-layer feature map for information to be supplemented.

[0028] like Figure 5As shown, the single-layer feature map of the information to be supplemented and the rotation detection box are input into the instance segmentation network. A cross-scale attention module performs 14×14 sampling on the single-layer feature map within the rotation detection box, and details from the low-scale feature map are added to the sampled single-layer feature map to obtain the supplemented information feature map. Specifically, the single-layer feature map of the information to be supplemented is selected from the multi-scale feature map based on the width and height of the rotation detection box. The single-layer feature map of the information to be supplemented and the rotation detection box are input into the instance segmentation network. A bilinear interpolation algorithm is used to perform 14×14 feature sampling on the low-scale feature map of the image and the single-layer feature map of the information to be supplemented within the rotation detection box of the target, resulting in the high-scale feature map F. H and detailed information feature map F L The high-scale feature map F is processed through a fully connected network with a cross-scale attention module. H Mapped to a query matrix Q H , to extract detailed information feature map F L Mapped to the bond matrix K respectively L Sum matrix V L By querying matrix Q H With bond matrix K L The high-scale feature map F is obtained by multiplying the matrices. H and detailed information feature map F L The similarity matrix, the similarity matrix and the value matrix V L Multiplication yields a stable feature map F containing detailed information. D The expression is: (4) Where Linear represents a fully connected network used for dimension transformation, and d represents the dimension of the query matrix; The feature map with supplementary detail information is obtained by adding the stable detail information feature map FD to the high-scale feature map FH element by element.

[0029] Step 3.5: Input the feature map of the supplementary information into the segmentation head to predict the segmentation mask; Step 3.6: Introduce three loss functions: a classification prediction loss function that measures the difference between the predicted class score and the true class, a regression prediction loss function that measures the difference between the predicted rotated bounding box and the true target coordinates, and a mask prediction loss function that measures the difference between the predicted segmentation mask and the true target mask. Add the values ​​obtained from these three loss functions together and return to step 3.1. Reduce the total loss value using the stochastic gradient descent algorithm until, after 36 training rounds, the CARSNet network parameters corresponding to the minimum total loss value are obtained, thus obtaining the trained CARSNet network.

[0030] Step 4: Segment ship instances in SAR images using the trained CARSNet network structure.

[0031] The framework of this invention consists of three parts: a feature extraction network, a rotating target detection network, and an instance segmentation network. This invention designs a cross-scale attention module to supplement high-scale features with stable low-scale details, improving the algorithm's segmentation performance on target contours. This invention also designs a positive sample sampling method based on Gaussian distribution, approximating the entire ship as a two-dimensional Gaussian distribution. Positive samples are selected based on the Gaussian distribution, which helps increase the number of positive samples at the ship's bow and stern, improving the performance in the rotating target detection stage.

[0032] Example 2 The method of this invention is used to segment ships in SAR images of the SSDD dataset, and the SRNet method is used for comparison. The comparison results are shown in Table 1. Table 1

[0033] As shown in Table 1, the AP results of the present invention and the comparison method SRNet under the SSDD dataset instance segmentation benchmark show that the detection accuracy of the present invention exceeds that of SRNet in all AP metrics, indicating that the present invention has a better detection effect on ship targets in SAR images.

[0034] Figure 6 , Figure 7 , Figure 8 For SAR image simulation results in the SSDD dataset, Figure 6 This is the actual labeled result image for this scenario. Figure 7 The segmentation results of SRNet, Figure 8 This is a diagram showing the segmentation result of the present invention.

[0035] Example 3 The method of this invention is used to segment ships in SAR images of the Instance-RSDD dataset, and the SRNet method is used for comparison. The comparison results are shown in Table 2. Table 2

[0036] As shown in Table 2, the AP results of the present invention and the comparison method SRNet under the instance segmentation benchmark of the Instance-RSDD dataset show that the detection accuracy of the present invention exceeds that of SRNet in all AP metrics, indicating that the present invention has a better detection effect on ship targets in SAR images.

[0037] Figure 9 , Figure 10 , Figure 11 The simulation results of SAR images on the Instance-RSDD dataset, Figure 9 This is the actual labeled result image for this scenario. Figure 10 The segmentation results of SRNet, Figure 11 This is a diagram showing the segmentation result of the present invention.

[0038] As shown in Tables 1 and 2 of Examples 2 and 3, the detection accuracy of the present invention surpasses that of SRNet in all AP metrics, indicating that the present invention has a better detection effect on ship targets in SAR images. Figure 7 , Figure 8 Comparison and Figure 10 , Figure 11 The comparison shows that SRNet produces detection boxes that cannot completely surround the ship target, affecting the final segmentation result; while the Ga-ATSS proposed in this invention significantly improves the accuracy of the detection boxes, which is beneficial to subsequent segmentation; at the same time, SRNet is not very good at segmenting the details of the ship outline, and the cross-scale attention module proposed in this invention greatly improves this problem and enhances the segmentation effect of the ship.

Claims

1. A method for ship instance segmentation in SAR images based on cross-scale attention, characterized in that, The specific steps are as follows: Step 1: Take the image and the true class of the target in the image, the true coordinates of the target, and the true target mask as a set of training samples, and extract multiple sets of training samples from the dataset; Step 2: Construct the CARSNet network structure based on cross-scale attention; The CARSNet network structure based on cross-scale attention includes: The ResNet network is used to extract low-scale feature maps from the input image. Feature extraction networks are used to extract multi-scale feature maps from input images; A rotating target detection network is used to predict the center point coordinates, rotation angle, width and height, and class score of a rotating target bounding box using multi-scale feature maps. The instance segmentation network includes a cross-scale attention module and a segmentation head, which is used to sample the feature map within the rotated detection box in 14×14, and supplement the sampled feature map with detailed information from the low-scale feature map. The feature map with the supplemented information is then input into the segmentation head to predict the segmentation mask. Step 3: Input the training samples into the CARSNet network structure based on cross-scale attention to obtain the trained CARSNet network; Step 4: Segment ship instances in SAR images using the trained CARSNet network structure.

2. The SAR image ship instance segmentation method based on cross-scale attention as described in claim 1, characterized in that, The target's true coordinates mentioned in step 1 include the coordinates of the target's center point in the image, the target's width and height, and the target's rotation angle coordinates.

3. The SAR image ship instance segmentation method based on cross-scale attention according to claim 1, characterized in that, The dataset mentioned in step 1 is either the Instance-RSDD dataset or the SSDD dataset.

4. The SAR image ship instance segmentation method based on cross-scale attention according to claim 1, characterized in that, The 14×14 sampling of the feature map within the rotated detection box is performed using a bilinear interpolation algorithm.

5. The SAR image ship instance segmentation method based on cross-scale attention according to claim 4, characterized in that, The specific process for extracting low-scale feature maps from the input image is as follows: After the image is input into the ResNet network, the outputs of stage 0 and stage 1 of the ResNet network are used as features C0 and C1, respectively. The output features C0 and C1 are fused using formula (1) to obtain the fused feature C. F The fusion formula is: （1） Where Conv represents a 1×1 convolution and DeConv represents a 4×4 transposed convolution; Fusion feature C F As a low-scale feature map of the image.

6. The SAR image ship instance segmentation method based on cross-scale attention according to claim 4, characterized in that, Step 3 is as follows: Step 3.1: Input the training samples into the ResNet network and output the low-scale feature maps of the training sample images; Step 3.2: Input the training samples into the feature extraction network and output the multi-scale feature map of the training sample image; Step 3.3: Input the multi-scale feature map of the training sample image into the rotating target detection network, convert the ground truth bounding box of the target into a two-dimensional Gaussian distribution, and use the positive sample sampling method based on Gaussian distribution to select positive sample anchor boxes and negative sample anchor boxes from the anchor boxes, and predict the rotating detection box of the target, the width and height of the rotating detection box, and the rotating detection box category score. Step 3.4: Select the single-layer feature map of the information to be supplemented from the multi-scale feature map according to the width and height of the rotation detection box. Input the single-layer feature map of the information to be supplemented and the rotation detection box into the instance segmentation network. The single-layer feature map in the rotation detection box is sampled in 14×14 through the cross-scale attention module. The detailed information on the low-scale feature map is supplemented into the sampled single-layer feature map to obtain the feature map of the supplemented information. Step 3.5: Input the feature map of the supplementary information into the segmentation head to predict the segmentation mask; Step 3.6: Introduce three loss functions: a classification prediction loss function that measures the difference between the predicted class score and the true class, a regression prediction loss function that measures the difference between the predicted rotated bounding box and the true target coordinates, and a mask prediction loss function that measures the difference between the predicted segmentation mask and the true target mask. Add the values ​​obtained from these three loss functions together and return to step 3.

1. Reduce the total loss value using the stochastic gradient descent algorithm until, after 36 training rounds, the CARSNet network parameters corresponding to the minimum total loss value are obtained, thus obtaining the trained CARSNet network.

7. The SAR image ship instance segmentation method based on cross-scale attention according to claim 6, characterized in that, The formula for converting the target's ground truth bounding box into a two-dimensional Gaussian distribution, as described in step 3.3, is: （2） in, express p Gaussian distribution values ​​of the points Represents the coordinates of the center point of the ship target. , , , p This represents the coordinates of the center point of an anchor frame. Represents the inverse transformation of a matrix. This indicates finding the determinant of a matrix. This represents the target's rotation angle coordinates.

8. The SAR image ship instance segmentation method based on cross-scale attention according to claim 6, characterized in that, The specific process of selecting the single-layer feature map to be supplemented from the multi-scale feature map based on the width and height of the rotated detection box is as follows: Select a single-layer feature map from the multi-scale feature map based on the width and height of the rotated detection box. F k The hierarchy of a single-layer feature map in a multi-scale feature map k The calculation expression is: （3） in, k 0 indicates the lowest scale of the multi-scale feature map. w , h These represent the width and height of the rotating detection frame, respectively. Select k Corresponding single-layer feature map F k This serves as a single-layer feature map for information to be supplemented.

9. The SAR image ship instance segmentation method based on cross-scale attention according to claim 6, characterized in that, Step 3.4 is as follows: Based on the width and height of the rotation detection box, select a single-layer feature map of the information to be supplemented from the multi-scale feature map. Input the single-layer feature map of the information to be supplemented and the rotation detection box into the instance segmentation network. Use the bilinear interpolation algorithm to perform 14×14 feature sampling on the low-scale feature map of the image and the single-layer feature map of the information to be supplemented within the rotation detection box of the target, and obtain the high-scale feature map F. H and detailed information feature map F L The high-scale feature map F is processed through a fully connected network with a cross-scale attention module. H Mapped to a query matrix Q H , to extract detailed information feature map F L Mapped to the bond matrix K respectively L Sum matrix V L By querying matrix Q H With bond matrix K L The high-scale feature map F is obtained by multiplying the matrices. H and detailed information feature map F L The similarity matrix, the similarity matrix and the value matrix V L Multiplication yields a stable feature map F containing detailed information. D The expression is: （4） Where Linear represents a fully connected network used for dimension transformation, and d represents the dimension of the query matrix; The stable detailed information feature map F D With high-scale feature map F H The feature map is obtained by adding elements together to obtain additional detailed information.

Images