SAR image ship target segmentation method based on diffusion model

Abstract

The invention discloses an SAR image ship target segmentation method based on a diffusion model, and solves the problems that the existing segmentation technology is insufficient in precision and wastes time and labor. According to the method, a diffusion model is introduced into original SAR ship instance segmentation, a ship instance segmentation task is processed from a generative perspective for the first time, and various effective improvements such as a space context joint enhanced feature pyramid network, a focusing intersection-union ratio loss function and mask representation based on instance perception are further provided, so that excellent instance segmentation precision is ensured. Compared with a traditional method, the method has the advantages that the model calculation complexity is moderate, the space complexity is the highest, and higher segmentation precision can be brought to a slightly complex model in an actual application scene. Therefore, the SAR image ship target instance segmentation method can be widely applied to SAR image ship target instance segmentation tasks, and the model complexity can be further improved in the future; the optimal average precision of SAR image ship target segmentation in offshore and offshore scenes is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of synthetic aperture radar (SAR) image interpretation, and relates to a SAR image ship target segmentation method based on a diffusion model. Background Art

[0002] Synthetic aperture radar (SAR) is an active microwave sensor that can work all day and all weather. Compared with optical sensors, SAR can penetrate clouds and fog and can complete observation tasks under adverse weather conditions. SAR has become an important means of earth observation today, and has been increasingly widely used in national economy such as terrain image generation, target detection and reconnaissance, land and resources exploration, and natural disaster monitoring. In particular, SAR is also widely used in the marine field, especially SAR image ship detection is of great significance in the field of remote sensing. For details, see the literature "Zhang Qingjun, Han Xiaolei, Liu Jie. Progress and development trend of spaceborne synthetic aperture radar remote sensing technology [J]. Spacecraft Engineering, 2017, 26(06): 1-8.".

[0003] With the massive acquisition of SAR ship data and the rapid development of SAR ship intelligent interpretation technology, SAR ship instance segmentation has become an advanced target interpretation method. In particular, as the main link of ship monitoring, SAR ship target instance segmentation can effectively realize pixel-level ship target detection and refined ocean information perception. For details, see the document "Gao, F., Huo, Y., Wang, J., Hussain, A., Zhou, H., 2021. Anchor-free SAR ship instancesegmentation with centroid-distance based loss. IEEE J. Sel. Topics Appl. EarthObserv. Remote Sens. 14, 11352–11371.".

[0004] At present, the SAR ship instance segmentation technology based on deep learning has become mature. A fully automatic, efficient and high-precision network model can be designed to achieve data-driven SAR ship instance segmentation. However, the existing SAR ship instance segmentation technology based on deep learning often starts from the perspective of regression optimization, constantly adjusts the prior hyperparameters, and iteratively optimizes the network model, so that the accuracy of SAR ship instance segmentation still has room for further improvement. It mainly faces two challenges: the preset target prior is difficult to perfectly match the shape and size of the ship target, and the prior parameter setting is time-consuming and labor-intensive.

[0005] Therefore, in order to solve the above problems, the present invention proposes a SAR image ship target segmentation method based on a diffusion model. This method proposes a new diffusion model, which represents the SAR ship instance segmentation task as a denoising process from a noise box to a target box and a reconstruction process from a target box to a ship instance. Innovatively, from a generative perspective, it avoids the hyperparameter difficulties of the preset target prior and the time-consuming and labor-intensive hyperparameter adjustment process of the traditional segmentation method, while ensuring the best average segmentation accuracy. Summary of the invention

[0006] The present invention discloses a method for SAR image ship target segmentation based on a diffusion model, which is used to solve the problems of insufficient precision and time-consuming and labor-intensive existing segmentation technologies. Based on the diffusion model theory, the method represents the SAR ship instance segmentation task as a denoising process from a noise box to a target box and a reconstruction process from a target box to a ship instance, and provides a spatial context joint enhanced feature pyramid network to enhance spatial position and multi-level context information and promote multi-scale ship feature extraction; provides a focused intersection-over-union loss function to alleviate the problem of imbalance between positive and negative samples; provides an instance-aware mask representation to adaptively obtain SAR ship instances from the denoised target box. The present invention achieves the best average accuracy of SAR image ship target segmentation in offshore and nearshore scenarios.

[0007] In order to facilitate the description of the present invention, the following terms are first defined:

[0008] Definition 1: SAR image

[0009] In remote sensing, SAR uses airborne or satellite-borne platforms to obtain images of the earth's surface. This process is accomplished by transmitting phase-encoded pulses along a direction nearly perpendicular to the sensor's motion vector, and receiving and recording the echoes reflected from the earth's surface. In order to form an image, intensity measurements are required in two mutually orthogonal axes. For details on SAR images, please refer to the document "Principles of Synthetic Aperture Radar Imaging", compiled by Pi Yiming et al., published by the University of Electronic Science and Technology of China Press.

[0010] Definition 2: SSDD dataset

[0011] SSDD dataset refers to SAR ship detection dataset. This dataset is the first open dataset widely used to study advanced ship detection technology based on deep learning (DL) synthetic aperture radar (SAR) images. The official version of SSDD was released at the BIGSARDATA conference on December 1, 2017. The official version of SSDD covers three types: BBox-SSDD, RBox-SSDD and PSeg-SSDD, containing 1160 SAR images with an image size of about 480*330 pixels, which is convenient for scholars to use according to different task requirements. The detailed information of the SSDD dataset can be obtained from the reference “Zhang, T., Zhang, X., Li, J., Xu, X., Wang, B., Zhan, X., Xu, Y., Ke, X., Zeng, T., Su, H., Ahmad, I., Pan, D., Liu, C., Zhou, Y., Shi, J., Wei, S., 2021. SAR ship detection dataset (SSDD): Official release and comprehensive data analysis. Remote Sens. 13 (18), 3690.”

[0012] Definition 3: HRSID dataset

[0013] The HRSID dataset refers to a high-resolution SAR image dataset. This dataset is constructed to solve the problems of small SAR image size, limited training samples, and inappropriate annotations in existing SAR ship datasets. Not only object detection can be performed on HRSDI, but also instance segmentation can be achieved. The dataset crops 136 panoramic SAR images with resolutions ranging from 1m to 5m into 800*800 pixel SAR images at an overlap rate of 25%. HRSID consists of 5604 cropped SAR images and 16951 ship instances. The detailed information of the HRSID dataset can be obtained from the reference "Wei, S., Zeng, X., Qu, Q., Wang, M., Su, H., Shi, J., 2020.HRSID: A high-resolution SAR images dataset for ship detection and instance segmentation. IEEE Access 8, 120234–120254."

[0014] Definition 4: Classic Resnet-101 network

[0015] Resnet-101 is a residual network with strong feature extraction capability and moderate model complexity. It is used to map the original image to a high-level deep feature space for subsequent decoder use. Resnet-101 is a variant of the Resnet architecture with 101 layers. It is designed to learn abstract features at different levels and has been proven to have good generalization and robustness in various SAR intelligent interpretation tasks. For details of the Resnet-101 model, please refer to the literature "He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778.".

[0016] Definition 5: Traditional Deformable Convolution Method

[0017] The deformable convolution method introduces two modules, deformable convolution and deformable region of interest pooling, into the classic convolutional neural network to enhance the conversion modeling ability of CNN. The spatial sampling position in the module is enhanced by an additional offset, and the offset is learned from the target task without additional supervision. The new module can easily replace the ordinary module in the existing CNN and can be easily trained end-to-end through standard back-propagation to produce a deformable convolutional network. For details of the traditional deformable convolution method, please refer to the literature "Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y., 2017. Deformable convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 764–773.".

[0018] Definition 6: Classic Convolutional Neural Network

[0019] The classic convolutional neural network (CNN) refers to a type of feedforward neural network that includes convolution calculations and has a deep structure. Convolutional neural networks are constructed by imitating the visual perception mechanism of organisms and can perform supervised learning and unsupervised learning. The convolution kernel parameter sharing in the hidden layer and the sparsity of the inter-layer connection enable the convolutional neural network to extract features with a smaller amount of computation. Convolutional neural networks have made rapid progress in computer vision, natural language processing, speech recognition and other fields. For details of the classic convolutional neural network method, please refer to the literature "Zhang Suofei, Feng Ye, Wu Xiaofu. Progress in target detection algorithms based on deep convolutional neural networks [J / OL]. Journal of Nanjing University of Posts and Telecommunications (Natural Science Edition), 2019(05):1-9.".

[0020] Definition 7: Classic CNN feature extraction method

[0021] Classic CNN feature extraction, that is, feature extraction of the original input image through CNN. In short, the original input image is converted into a series of feature maps through convolution operations of different features. In CNN, the convolution kernel in the convolution layer continuously slides on the image. At the same time, the maximum pooling layer is responsible for taking the maximum value of each local block on the inner product result. Therefore, CNN implements the image feature extraction method through convolution layer and maximum pooling layer. For details of classic CNN feature extraction, please refer to the website "https: / / blog.csdn.net / qq_30815237 / article / details / 86703620".

[0022] Definition 8: Classic convolutional layer

[0023] The convolution layer is composed of several convolution units, and the parameters of each convolution unit are optimized through the back propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolution layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features. For details of the classic convolution layer, please refer to the website "https: / / www.zhihu.com / question / 49376084".

[0024] Definition 9: Traditional recombinant nucleus generation method

[0025] Reassembly Kernel is a convolution kernel generation method for dynamic upsampling, which is mainly used for content-aware feature reorganization. This method encodes the content of the input feature map and dynamically generates convolution kernels suitable for different positions according to the context, which is used to upsample the feature map to a higher resolution. First, the number of channels of the input feature map is reduced through a 1×1 convolution layer to reduce the computational complexity; the compressed feature map is further convolved to generate a dynamic convolution kernel. Each generated convolution kernel is adjusted according to the contextual features of a specific position; the convolution kernel is normalized using the Softmax function to ensure that the sum of the convolution kernel weights is 1, thereby achieving smooth upsampling similar to the interpolation method. For details on the traditional recombinant core generation method, please refer to the literature “Wang, J., Chen, K., Xu, R., Liu, Z., Loy, CC, Lin, D., 2019. CARAFE: Content-aware reassembly of features. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3007–3016.”.

[0026] Definition 10: Classic Feature Pyramid Network

[0027] Feature Pyramid Network (FPN) is a top-down architecture with lateral connections, which is used to construct high-level semantic feature maps of all scales. FPN mainly solves the multi-scale problem in object detection. On the basis of basically not increasing the computational complexity of the original model, it greatly improves the detection performance of small objects by simply changing the network connection. For details of Feature Pyramid Network, please refer to the literature "T.-Y.Lin, P.Dollár, R.Girshick, K.He, B.Hariharan and S.Belongie,"Feature Pyramid Networks for Object Detection,"2017IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Honolulu, HI, USA, 2017, pp.936-944, doi:10.1109 / CVPR.2017.106.". Definition 11: Traditional content-aware feature reconstruction method

[0028] The traditional content-aware feature reconstruction (CARAFE) method has the ability of large receptive field information aggregation and adaptive content recognition, and includes a kernel prediction module and a content-aware reconstruction module. Specifically, CARAFE can generate a lower-level feature layer P1 from the original lowest-level feature layer P2, thereby providing rich spatial location information and improving the feature representation ability of small-scale targets. Using CARAFE to generate an additional feature layer P1 can be described as:

[0029] P1=CARAFE ×2 (P2)

[0030] Among them, CARAFE ×2 Indicates a double upsampling operation using CARAFE. When using the CARAFE module, an input feature map and an upsampling ratio must be given first, and CARAFE can generate an output feature map. First, the kernel prediction module predicts and reassembles an adaptive secondary upsampling kernel K, whose size is n×n, representing the pixel neighborhood of position I with a size of n×n. The convolution layer is used for channel compression and generates a channel number of n 2 ×2 2 The reassembly kernel is then used. The content-aware reassembly module provides the reassembly feature map generated by the convolutional layer, thereby obtaining the upsampled feature map P1. The traditional content-aware feature reassembly (CARAFE) method is described in the literature “Wang, J., Chen, K., Xu, R., Liu, Z., Loy, CC, Lin, D., 2019. CARAFE: Content-aware reassembly of features. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3007–3016.”.

[0031] Definition 12: Classical two-stage ACmix

[0032] ACmix is ​​a fusion method of convolution and self-attention mechanisms, which can be used to refine the contextual representation of different feature levels of the integrated feature map, and ultimately improve the multi-scale feature representation. ACmix breaks the barriers between the two attention mechanisms and fuses them together by exploring the similarities of convolution and self-attention in 1×1 convolution operations. ACmix consists of two stages. The final output is the weighted sum of the convolution path and the self-attention path:

[0033] F out =αF conv +βF self

[0034] Among them, F outis the output feature, F conv is the convolution path output feature, F self is the output feature of the self-attention path, and α and β are the learnable weight coefficients involved in network training. For details of the classic two-stage ACmix, please refer to the literature "Pan, X., Ge, C., Lu, R., Song, S., Chen, G., Huang, Z., Huang, G., 2022. On the integration of self-attention and convolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 805-815.".

[0035] Definition 13: Traditional intermediate feature set generation method

[0036] In the first stage of ACmix, the input features need to be projected through three 1×1 convolutions and reshaped into N blocks respectively to obtain an intermediate feature set including 3x N feature maps. The core idea of ​​using 1x1 convolution for projection is to use a 1x1 convolution kernel to linearly transform or adjust the number of channels (dimensions) of the input feature map without changing the spatial dimensions (width and height). For details on the traditional intermediate feature set generation method, please refer to the literature "Pan, X., Ge, C., Lu, R., Song, S., Chen, G., Huang, Z., Huang, G., 2022. On the integration of self-attention and convolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 805-815.".

[0037] Definition 14: Traditional convolution path method

[0038] In the second stage of ACmix, one of the paths is called the convolution path. It refers to generating a feature map through a lightweight fully connected layer, and then convolving the generated feature map through shift and sum operations, and extracting the contained information from the perspective of the local receptive field. For details of the traditional convolution path method, please refer to the literature "Pan, X., Ge, C., Lu, R., Song, S., Chen, G., Huang, Z., Huang, G., 2022. On the integration of self-attention and convolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 805-815.". Definition 15: Traditional self-attention path method

[0039] In the second stage of ACmix, another path is called the self-attention path. It means that the intermediate features generated in the first stage are divided into N groups, each of which contains three feature maps, and then the three feature maps are processed in a multi-head self-attention manner through attention and aggregation operations. The three feature maps are used as queries, keys, and values ​​respectively. For details of the traditional self-attention path method, please refer to the literature "Pan, X., Ge, C., Lu, R., Song, S., Chen, G., Huang, Z., Huang, G., 2022. On the integration of self-attention and convolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 805-815.".

[0040] Definition 16: Traditional Cascade Neural Network Method

[0041] The traditional cascade neural network method is a structure that cascades multiple neural network models according to specific rules. Cascade neural networks are usually composed of multiple independent sub-networks, each of which is responsible for completing a specific task or extracting specific features. By connecting multiple sub-networks in series, the cascade neural network can gradually extract and refine the features of the input data, thereby improving the performance of the overall model. For details of the traditional cascade neural network method, please visit the website "https: / / www.kepuchina.cn / article / articleinf o?business_type=100&ar_id=249206".

[0042] Definition 17: Traditional Cascade Random Box Filling Strategy Method

[0043] The cascade random box filling strategy refers to gradually generating random boxes with noise starting from the labeled box of the data (groundtruth box). The random boxes generated at each level superimpose different degrees of Gaussian noise on the original groundtruth box. In each level of cascade, the intensity of Gaussian noise gradually increases, causing disturbances in the position, size or shape of the box. The traditional cascade random box filling strategy method can be obtained from the literature "Song, J., Meng, C., & Ermon, S., 2020.Denoisingdiffusion implicit models.arXiv preprint arXiv:2010.02502."

[0044] Definition 18: Traditional Diffusion Model Approach

[0045] The diffusion model is a likelihood-based latent variable model that maps data to a latent space using a Markov chain. The diffusion model is divided into two steps: the forward diffusion process and the reverse denoising process. The former is to gradually add noise to the data at each time step until completely random noise is obtained; the latter is to gradually denoise the data at each sampling step until the true value is obtained. Specifically, given an input image distribution x0~q(x0), the detailed implementation formula of the forward diffusion process is defined as:

[0046]

[0047] Among them, x t represents a noise sample with a time step of t∈{0,1,…,T}; β t represents the variance table when Gaussian noise is gradually added to x0, β t It is set as a linearly increasing variable from 0.0004 to 0.02, and the upper limit of the time step T is 1000.

[0048] Therefore, given an original input x0, by gradually adding a Gaussian noise Until a completely random noise sample is obtained (when t = T), as follows:

[0049]

[0050] in,

[0051] During the training process, the network f θ (x t ,t) will try to extract the noise samples x by minimizing the l2 norm loss tFitting the original target sample x0:

[0052]

[0053] During inference, the network f θ (x t ,t) Try to get the noise sample x t The original target sample x0 is gradually reconstructed in , and there is an update rule at each sample step, namely X T →X T-Δ →X0.

[0054] In ship instance segmentation, the diffusion model is formulated as a denoising process from noise box to target box and a reconstruction process from target ship box to instance.

[0055] The traditional diffusion model method can be obtained from the literature “Song, J., Meng, C., & Ermon, S., 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.”

[0056] Definition 19: Classical Intersection-over-Union loss function

[0057] The intersection over union (IoU) is defined as the ratio of the intersection and union of the predicted box (bounding box) and the ground truth box (ground truth), and its calculation formula is:

[0058]

[0059] Among them, A represents the predicted box, B represents the real box, ∩ represents the intersection operation, and ∪ represents the union operation. Usually in the process of training the model, the IoU value is used to judge whether the result is a good result.

[0060] When the intersection-over-union ratio is used as the loss function, the commonly used calculation formula is:

[0061]

[0062] The intersection over union loss function can better reflect the degree of overlap and is scale invariant; however, when the predicted box and the true box have no intersection, the IoU Loss cannot be calculated, resulting in a gradient of 0 and no optimization. Detailed information on the classic intersection over union loss function can be obtained from the website "https: / / blog.csdn.net / weixin_43482623 / article / details / 119574564".

[0063] Definition 20: Traditional training total loss calculation method

[0064] During training, the total training loss is composed of detection loss and segmentation loss:

[0065]

[0066] in, represents the detection loss, represents the segmentation loss, λ bal represents the balance hyperparameter between detection and segmentation. Specifically, and It can be expressed as:

[0067]

[0068] Among them, λ cls represents the classification weight, λ reg represents the regression weight, λ IoU represents the intersection-and-union weight; represents the classification loss, α t represents the balance factor, γ represents the modulation factor, and p t represents the predicted category probability; represents regression loss, n represents the number of samples, y i represents the predicted value of the i-th sample, f(X i ) represents the true value of the i-th sample; represents the focused intersection-over-union loss; m represents the predicted ship mask, m gt Indicates the actual ship mask.

[0069] Definition 21: Traditional denoising diffusion implicit model method

[0070] The denoising diffusion implicit model (DDIM) is proposed based on the denoising diffusion probability model. The training process is the same as that of the denoising diffusion probability model, and samples are generated by simulating the Markov chain. The difference is that DDIM is a more efficient iterative implicit probability model, in which a class of non-Markov diffusion processes is constructed, and the reverse process of training the same target can sample faster. For details of the traditional denoising diffusion implicit model method, please refer to the literature Song, J., Meng, C., & Ermon, S., 2020.Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502. ".

[0071] Definition 22: Reconstructed ship instance mask

[0072] During inference, starting from a noise box with a Gaussian distribution, a groundtruth box that conforms to the target distribution is gradually generated through a denoising process. Among them, a denoising model is used to estimate the detection box for the next sampling step, and a box update strategy is used to better align inference with training. Box update means that after each sampling step, low-scoring boxes are first filtered out, and then the remaining boxes are connected with new random boxes sampled from a Gaussian distribution. Then instance-specific filters are established, and finally the ship instance mask m can be reconstructed:

[0073] b0=f(...(f(b T-s ,Ts))),s={0,...,T}

[0074] θ0=η(b0)

[0075] m=φ(F mask ;θ0)

[0076] Among them, b0 is the real box that conforms to the target distribution and is gradually generated through the denoising process. T-s is a noise box conforming to a Gaussian distribution, f(·) is the denoising network, η(·) is a fully connected layer, and θ0 is an instance-specific filter.

[0077] Definition 23: Traditional Polarized Self-Attention Module Method

[0078] Self-Attention (SA), also known as internal attention mechanism, is an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. The polarized self-attention module (PSA) consists of two branches, namely the channel self-attention branch and the spatial self-attention branch. It achieves high-quality pixel regression through two key designs of polarized filtering and enhancement, which can ensure effective information integration. Specifically, given an input feature map The process can be described as:

[0079] F ‘’ =f psa (F′)

[0080] F′=f re (concat(conv 1×3 (F),conv 3×3 (F),conv 3×1 (F)))

[0081] Among them, conv 1×3 Represents a 1×3 convolution operation, conv 3×3 Represents a 3×3 convolution operation, conv 3×1 represents a 3×1 convolution operation; f psa(·) indicates the PSA operator; concat(·) indicates the concatenation operator; f re (·) represents a feature refinement operation, such as a 1×1 convolutional layer. The traditional polarized self-attention module method is detailed in the paper "Liu, H., Liu, F., Fan, X., Huang, D., 2021. Polarized self-attention: Towards high-quality pixel-wise regression. arXiv preprint arXiv: 2107.00782." Definition 24: Conditional convolution method for traditional instance segmentation

[0082] Conditional convolution for instance segmentation (CondInst) uses a dynamic instance-aware network conditioned on instances. The fully convolutional network solves instance segmentation without ROI cropping and feature alignment. In addition, since the capacity of dynamically generated conditional convolutions is greatly improved, the mask header can be very compact, significantly speeding up the inference speed. For details on conditional convolutions for traditional instance segmentation, see the paper "Tian, ​​Z., Shen, C., Chen, H., 2020. Conditional convolutions for instancesegmentation. In: Proceedings of the IEEE European Conference on Computer Vision. pp. 282–298.".

[0083] Definition 25: Classic AdamW optimization method

[0084] The classic Adam optimization method is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can iteratively update the weights of the neural network based on the training data. Adam designs independent adaptive learning rates for different parameters by calculating the first-order moment estimate and the second-order moment estimate of the gradient. AdamW decouples the weight decay from the optimization step of the loss function through a simple modification to restore the original formula of weight decay regularization, greatly improving the generalization performance of Adam. For details of the classic Adam optimization method, please refer to the literature "Loshchilov, I., Hutter, F., 2019. Decoupled weight decay regularization. In: Proceedings of the IEEE International Conference on Learning Representations. pp. 1–19.".

[0085] Definition 26: Standard network testing method

[0086] The standard network testing method refers to performing a final test on the model on the test set to obtain the test results of the model on the test set. The standard network testing method is detailed in the literature "C. Lu, and W. Li, "Ship Classification in High-Resolution SAR Images via Transfer Learning with Small Training Dataset," Sensors, vol. 19, no. 1, pp. 63, 2018.".

[0087] Definition 27: Classic COCO metric method

[0088] The COCO metric is used as the evaluation indicator. The core criterion of the COCO metric is the average precision (AP), which is the average precision under 10 IoU thresholds, ranging from 0.50 to 0.95 with an interval of 0.05. 50 It represents the average precision when the IoU threshold is 0.50; AP 75 It represents the average precision when the IoU threshold is 0.75; AP S Indicates the average accuracy of small pixel targets (ships <32 2 pixels); AP M The average precision of medium pixel objects (32 2 <Ship<96 2 pixels); AP L Indicates the average accuracy of large pixel targets (ships>96 2 Pixels). In ship instance segmentation, mask average precision (maskAP) is used to represent the average precision AP value. In addition, the number of parameters (#Para) is used to measure the computational complexity of the algorithm, and the number of floating-point operations (FLOPs) is used to measure the spatial complexity of the algorithm. The smaller the values ​​of the two, the lighter the network. For details of the standard COCO metric, please refer to the literature "Zhang, T., Zhang, X., Ke, X., Zhan, X., Shi, J., Wei, S., Pan, D., Li, J., Su, H., Zhou, Y., Kumar, D., 2020. LS-SSDD-v1.0: Adeep learning dataset dedicated to small ship detection from large-scale sentinel-1SAR images. Remote Sens. 12 (18), 2997.".

[0089] Definition 28: Nine traditional methods with similar functions to the present invention

[0090] They are:

[0091] 1. Traditional Mask Object Instance Segmentation Framework Method (Mask R-CNN), see the reference "He, K., Gkioxari, G., Dollár, P., & Girshick, R., 2017. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2980–2988." for details;

[0092] 2. Traditional multi-stage object detector method (Cascade Mask R-CNN), see the reference "Cai, Z., Vasconcelos, N., 2019. Cascade R-CNN: High quality object detection and instancesegmentation. IEEE Trans. Pattern Anal. Mach. Intell. 43(5), 1483–1498." for details;

[0093] 3. Traditional real-time instance segmentation model (YOLACT) method, see the reference "Bolya, D., Zhou, C., Xiao, F., Lee, Y.

[0094] J., 2019. YOLACT: Real-time instance segmentation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 9156–9165.";

[0095] 4. Traditional conditional convolution method for instance segmentation (CondInst), see the reference "Tian, ​​Z., Shen, C., Chen, H., 2020. Conditional convolutions for instance segmentation. In: Proceedings of the IEEE European Conference on Computer Vision. pp. 282–298.";

[0096] 5. Traditional efficient real-time target detector method (RTMDet), see the reference “Lyu, C., Zhang, W., Huang, H.,

[0097] Zhou,Y.,Wang,Y.,Liu,Y.,Zhang,S.,Chen,K.,2022.RTMDet: An empirical study of designing realtime object detectors.arXiv preprint arXiv:2212.07784.";

[0098] 6. Traditional instance segmentation model method based on rotation bounding box (SRNet), see the reference "Yang, X., Zhang, Q.,

[0099] Dong,Q.,Han,Z.,Luo,X.,Wei,D.,2023.Ship Instance Segmentation BasedonRotated Bounding Boxes for SAR Images.Remote Sens.15(5),pp.1324.";

[0100] 7. Traditional instance segmentation model method based on Gaussian heat map (BFSS-Inst), see the reference “Gao, F., Zhong, F., Sun,

[0101] J.,Hussain,A.,Zhou,H.,2024.BBox-free SAR ship instance segmentationmethodbased on gaussian heatmap.IEEE Trans.Geosci.Remote Sens.62,1–18.";

[0102] 8. High-quality instance segmentation network method for traditional remote sensing images (HQ-ISNet), see the reference “Su, H., Wei, S., Liu, S.,

[0103] Liang,J.,Wang,C.,Shi,J.,Zhang,X.,2020.HQ-ISNet: High-quality instance segmentation for remote sensing imagery.Remote Sens.12(6),989.";

[0104] 9. Traditional instance segmentation model method based on centroid distance loss (AFSS-Inst), see the reference “Gao, F., Huo, Y.,

[0105] Wang,J.,Hussain,A.,Zhou,H.,2021.Anchor-free SAR ship instancesegmentationwith centroid-distance based loss.IEEE J.Sel.Topics Appl.EarthObserv.Remote Sens.

[0106] 14,11352–11371.”.

[0107] The present invention provides a SAR image ship target segmentation method based on a diffusion model, which comprises the following steps:

[0108] Step 1: Prepare the dataset

[0109] For the SSDD dataset provided in Definition 2, the ratio of the training set to the test set is set to 8:2, and the training set is denoted as SSDD train , the test set is denoted as SSDD test ;

[0110] For the HRSID dataset provided in Definition 3, the ratio of training set to test set is set to 13:7, and the training set is denoted as HRSID train , the test set is denoted as HRSID test .

[0111] Step 2: Encoder Construction

[0112] Step 2.1: Backbone network construction

[0113] For the classic Resnet-101 network model in Definition 4, the traditional deformable convolution method in Definition 5 is used to process it, and the backbone classic convolutional neural network of the SAR image ship target segmentation network in Definition 6 is obtained, which is denoted as Backbone.

[0114] Using the classic CNN feature extraction method in Definition 7, the training set SSDD obtained in step 1 train or HRSID train Any SAR image in the image is processed on the backbone classical convolutional neural network Backbone obtained in the previous step, and the feature output is recorded as F in ;

[0115] Step 2.2: Spatial context joint enhancement feature pyramid network construction

[0116] Set the size of the sampling kernel n to 5, and use the traditional recombinant kernel generation method in Definition 9 to process the classic convolution layer in Definition 8, generating 5 channels. 2 ×22 of recombinant nuclei;

[0117] Output the features obtained in step 2.1 to F in As input, the traditional content-aware feature reorganization method in Definition 11 is used to process the classic feature pyramid network in Definition 10 to generate additional low-level feature maps, and obtain the spatial information enhancement module, denoted as SIE. The output feature of the SIE module is denoted as F CARAFE ;

[0118] For the classic two-stage ACmix in Definition 12, in the first stage, the feature output F obtained above is CARAFE As input, the traditional intermediate feature set generation method in Definition 13 is used to obtain the intermediate feature set, denoted as F m ;

[0119] In the second stage, the intermediate feature set F obtained in the first stage is m As input, the traditional convolution path method in Definition 14 and the traditional self-attention path method in Definition 15 are used to obtain the semantic information enhancement module, denoted as CIE. The weighted sum of the features of the two paths output by the CIE module is denoted as F out .

[0120] At this point, the SIE module and CIE module obtained in the first two steps are cascaded using the traditional cascade neural network method in Definition 16 to obtain a spatial context joint enhanced feature pyramid network, denoted as SCJE-FPN.

[0121] Output the features obtained in step 2.1 to F in Input to the spatial context joint enhancement feature pyramid network SCJE-FPN, the spatial context joint enhancement feature pyramid network SCJE-FPN outputs the deep feature F after the feature is jointly enhanced with the spatial context out , the encoder part of the SAR image ship target segmentation network based on the diffusion model is completed.

[0122] Step 3: Ship target segmentation network construction

[0123] During the training process, the deep feature F in step 2 is out , using the traditional cascade random box filling strategy method in Definition 17, we get the noise box B t ;

[0124] For the noise frame B obtained in the previous step t , using the traditional diffusion model method defined in Definition 18, we obtain the instance-specific filter θ0 = η(f(B t ,t)), where f(·) is the denoising network and η(·) is the fully connected layer;

[0125] For the classic intersection-over-union loss function in Definition 19, by adding a new focusing factor α, we get the focused intersection-over-union loss function, denoted as FIoU:

[0126] Specifically, given a predicted box B and a real box B gt , the FIoU loss function is defined as follows:

[0127]

[0128] Among them, p1 is the coordinate of the point on the upper left of the prediction box, and p2 is the coordinate of the point on the lower right of the prediction box. is the coordinate of the upper left point of the real box, is the coordinate of the point on the lower right of the real box, (·) is the Euclidean distance, c is the diagonal length of the minimum bounding box of the predicted box and the real box, is the intersection over union ratio, α is the focusing factor with a value of 0.95, and IoU is the intersection over union ratio;

[0129] For the focused intersection-over-union loss function FIoU obtained in the previous step, the traditional training total loss calculation method in Definition 20 is used to obtain the total loss function in the training process: in represents the detection loss, represents the segmentation loss, λ bal represents the balance hyperparameter between detection and segmentation, and its value is 5.

[0130] During the inference process, the noise box B in the training process t After the network is trained using the traditional denoising diffusion implicit model method in Definition 21, the reconstructed ship instance mask m in Definition 22 is obtained.

[0131] At this point, the decoder module of the SAR image ship target segmentation network based on the diffusion model has been completed.

[0132] Output the features in step 2 to F out As input, the traditional polarized self-attention module method in Definition 23 is used for information integration to obtain a multi-asymmetric convolutional self-attention module, denoted as MACSA;

[0133] For the MACSA module obtained in the previous step, the conditional convolution method of traditional instance segmentation in Definition 24 is used to process it to obtain the instance-aware mask representation module, denoted as IAMR.

[0134] At this point, the construction of the SAR image ship target segmentation network based on the diffusion model is completed, denoted as DiffSARShipInst.

[0135] Step 4: Build a ship segmentation model

[0136] The training set SSDD obtained in step 1 train or HRSID train As input, the SAR image ship target segmentation network DiffSARShipInst based on the diffusion model completed in step 3 is optimized and trained using the classic AdamW optimization method in Definition 25. After the training is completed, a ship segmentation model is obtained, which is recorded as DiffSARShipInst_Model.

[0137] Step 5: Test the ship segmentation model

[0138] Use the test set SSDD obtained in step 1 test or HRSID test , the ship segmentation model DiffSARShipInst_Model obtained in step 4 is tested using the standard network testing method in definition 26 to obtain the test result of the test set on DiffSARShipInst_Model, which is recorded as Result.

[0139] Step 6: Evaluate the ship segmentation model

[0140] Taking the test result Result of the ship segmentation model obtained in step 5 as input, the classic COCO measurement method in Definition 27 is used to obtain the mask average precision maskAP.

[0141] This completes the entire method.

[0142] The innovation of the present invention is that it introduces a diffusion model into the original SAR ship instance segmentation, processes the ship instance segmentation task from a generative perspective for the first time, and also provides effective improvements such as spatial context joint enhanced feature pyramid network, focused intersection-over-union loss function, and instance-aware mask representation to ensure superior instance segmentation accuracy. The experimental results of the present invention on the SAR Ship Detection Dataset (SSDD) and the High Resolution SAR Image Dataset (HRSID) have achieved the best average mask accuracy in both offshore and nearshore scenes.

[0143] The advantage of the present invention is that the SAR ship instance segmentation task is represented as a denoising process from a noise box to a target box and a reconstruction process from a target box to a ship instance, thereby avoiding the difficulty of presetting target prior hyperparameters in traditional segmentation methods and the time-consuming and labor-intensive hyperparameter adjustment process. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 Schematic diagram of the process of SAR image ship target segmentation method in the present invention

[0145] Figure 2 This is a numerical comparison table of the average accuracy of the SAR image ship target segmentation method in the present invention and different ship instance segmentation methods on the SSDD dataset.

[0146] Where: AP stands for Average Precision, the best model is marked in bold, and the sub-optimal model is underlined.

[0147] Figure 3 This is a numerical comparison table of the average accuracy of the SAR image ship target segmentation method in the present invention and different ship instance segmentation methods on the HRSID dataset.

[0148] Where: AP stands for Average Precision, the best model is marked in bold, and the sub-optimal model is underlined.

[0149] Figure 4 Table 1 is a comparison table of model complexity between the SAR image ship target segmentation method of the present invention and different ship instance segmentation methods.

[0150] in: Represents the FLOP calculated based on SSDD; Represents the FLOP calculated based on HRSID; Represents #Para calculated according to SSDD / HRSID, the best complexity evaluation index value is marked in bold, and the suboptimal complexity evaluation index value is marked with a subscript. DETAILED DESCRIPTION

[0151] The following is combined with Figure 1 The present invention is described in further detail.

[0152] Step 1: Prepare the dataset

[0153] For the SSDD dataset provided in Definition 2, the ratio of the training set to the test set is set to 8:2, and the training set is denoted as SSDD train , the test set is denoted as SSDD test ;

[0154] For the HRSID dataset provided in Definition 3, the ratio of training set to test set is set to 13:7, and the training set is denoted as HRSID train , the test set is denoted as HRSID test .

[0155] Step 2: Encoder Construction

[0156] Step 2.1: Backbone network construction

[0157] For the Resnet-101 network model in Definition 4, the deformable convolution method in Definition 5 is used to obtain the backbone convolutional neural network of the SAR image ship target segmentation network in Definition 6, which is denoted as Backbone.

[0158] Using the classic CNN feature extraction method in Definition 7, the training set SSDD obtained in step 1 train or HRSID train A SAR image in the image is processed on the backbone convolutional neural network Backbone obtained in the previous step, and the feature output is recorded as F in ;

[0159] Step 2.2: Spatial context joint enhancement feature pyramid network construction

[0160] For the classic convolutional layer in Definition 8, the size of the sampling kernel n is set to 5, and the traditional recombinant kernel generation method in Definition 9 is used to generate 5 channels. 2 ×2 2 of recombinant nuclei;

[0161] For the classic feature pyramid network in Definition 10, the feature output F obtained in step 2.1 in As input, the content-aware feature reconstruction technique in Definition 11 is used to generate additional low-level feature maps to obtain the spatial information enhancement module, denoted as SIE. The output feature of the SIE module is denoted as F CARAFE ;

[0162] Output the features of the previous step F CARAFE As input, in the first stage of the classic two-stage ACmix in Definition 12, the traditional intermediate feature set generation method in Definition 13 is used to process and obtain the intermediate feature set, denoted as F m ;

[0163] The intermediate feature set F obtained in the first stage m As input, for the second stage of the classic two-stage ACmix in Definition 12, the traditional convolution path method in Definition 14 and the traditional self-attention path method in Definition 15 are used to obtain a semantic information enhancement module, denoted as CIE, and the weighted sum of the features of the two paths output by the CIE module is denoted as F out .

[0164] At this point, the SIE module and CIE module obtained in the first two steps are cascaded using the cascade neural network in Definition 16 to obtain a spatial context joint enhanced feature pyramid network, denoted as SCJE-FPN.

[0165] Output the features obtained in step 2.1 to F inAs input to the spatial context joint enhancement feature pyramid network SCJE-FPN, the output is the deep feature F after the feature is jointly enhanced with the spatial context out , the encoder part of the SAR image ship target segmentation network based on the diffusion model is completed.

[0166] Step 3: Ship target segmentation network construction

[0167] During the training process, the feature output F in step 2 is out , using the cascade random box filling strategy in Definition 17, we get the noise box B t ;

[0168] For the noise frame B obtained in the previous step t , the denoising network in the diffusion model of Definition 18 is used to obtain an instance-specific filter θ0 = η(f(B t ,t)), where f(·) is the denoising network and η(·) is the fully connected layer;

[0169] For the intersection-over-union loss function in Definition 19, by adding a new focusing factor α, we get the focused intersection-over-union loss function, denoted as FIoU. Specifically, given a predicted box B and a true box B gt , FIoU loss is defined as follows:

[0170]

[0171] Among them, p1 is the coordinate of the point on the upper left of the prediction box, and p2 is the coordinate of the point on the lower right of the prediction box. is the coordinate of the upper left point of the real box, is the coordinate of the point on the lower right of the real box, (·) is the Euclidean distance, c is the diagonal length of the minimum bounding box of the predicted box and the real box, is the intersection over union ratio, α is the focusing factor with a value of 0.95, and IoU is the intersection over union ratio;

[0172] For the focused intersection-over-union loss function obtained in the previous step, the total training loss calculation method in Definition 20 is used to obtain the total loss function during training: in represents the detection loss, represents the segmentation loss, λ bal represents the balance hyperparameter between detection and segmentation, and its value is 5.

[0173] During the inference process, the noise box B in the training process t , after network training using the denoising diffusion implicit model in Definition 21, the reconstructed ship instance mask m in Definition 22 is obtained.

[0174] At this point, the decoder module of the SAR image ship target segmentation network based on the diffusion model has been completed.

[0175] Output the features in step 2 to F out As input, the polarized self-attention module in Definition 23 is used for information integration to obtain a multi-asymmetric convolutional self-attention module, denoted as MACSA;

[0176] For the MACSA module obtained in the previous step, the conditional convolution method of instance segmentation in Definition 24 is used to obtain the instance-aware mask representation module, denoted as IAMR.

[0177] At this point, the construction of the SAR image ship target segmentation network based on the diffusion model is completed, denoted as DiffSARShipInst.

[0178] Step 4: Build a ship segmentation model

[0179] The training set SSDD obtained in step 1 train or HRSID train As input, the SAR image ship target segmentation network DiffSARShipInst based on the diffusion model completed in step 3 is optimized and trained using the classic AdamW algorithm in Definition 25. After the training is completed, a ship segmentation model is obtained, which is recorded as DiffSARShipInst_Model.

[0180] Step 5: Test the ship segmentation model

[0181] Use the test set SSDD obtained in step 1 test or HRSID test , the ship segmentation model DiffSARShipInst_Model obtained in step 4 is tested using the standard network testing method in definition 26 to obtain the test result of the test set on DiffSARShipInst_Model, which is recorded as Result.

[0182] Step 6: Evaluate the ship segmentation model

[0183] Taking the test result Result of the ship segmentation model obtained in step 5 as input, the COCO measurement method in Definition 27 is used to obtain the mask average precision maskAP.

[0184] This completes the entire method.

[0185] It can be seen from the above specific implementations that the present invention can effectively perform SAR image ship target instance segmentation. Figure 2As shown in FIG. 1 , the AP value of the present invention achieved on the SSDD dataset is 70.6% for offshore scenes and 56.2% for coastal scenes. Figure 3 As shown in FIG. 1 , the AP value of the present invention in the offshore scene on the HRSID dataset is 70.9%, and the AP value of the nearshore scene is 42.6%, both of which are the best values ​​among all the compared methods. Figure 4 As shown in the figure, compared with nine traditional methods with similar functions, the present invention has moderate model calculation complexity and the highest spatial complexity; in practical application scenarios, it can bring advantages such as higher segmentation accuracy for slightly complex models. Therefore, the present invention can be widely used in SAR image ship target instance segmentation tasks, and the model complexity can be further improved in the future.

Claims

1. A method for ship target segmentation in SAR images based on a diffusion model, characterized in that The following steps are involved: Step 1: Prepare the dataset For the SSDD dataset, the ratio of the training set to the test set is set to 8:2, and the training set is denoted as SSDD. train , the test set is denoted as SSDD test ; For the HRSID dataset, the ratio of the training set to the test set is set to 13:7, and the training set is denoted as HRSID. train , the test set is denoted as HRSID test ; Step 2: Encoder Construction Step 2.1: Backbone network construction For the classic Resnet-101 network model, the traditional deformable convolution method is used to obtain the backbone classic convolutional neural network of the SAR image ship target segmentation network, which is denoted as Backbone; Using the classic CNN feature extraction method, the training set SSDD obtained in step 1 train or HRSID train Any SAR image in the image is processed on the backbone classical convolutional neural network Backbone obtained in the previous step, and the feature output is recorded as F in ; Step 2.2: Spatial context joint enhancement feature pyramid network construction The size of the sampling kernel n is set to 5, and the traditional recombinant kernel generation method is used to process the classic convolution layer, generating 5 channels. 2 ×2 2 of recombinant nuclei; Output the features obtained in step 2.1 to F in As input, the traditional content-aware feature reorganization method is used to process the classic feature pyramid network to generate additional low-level feature maps, and the spatial information enhancement module is obtained, denoted as SIE. The output feature of the SIE module is denoted as F CARAFE ; For the classic two-stage ACmix, in the first stage, the feature output F obtained above is CARAFE As input, the traditional intermediate feature set generation method is used to obtain the intermediate feature set, denoted as F m ; In the second stage, the intermediate feature set F obtained in the first stage is m As input, the traditional convolution path method and the traditional self-attention path method are used for processing to obtain the semantic information enhancement module, denoted as CIE. The weighted sum of the features of the two paths output by the CIE module is denoted as F out ; At this point, the SIE module and CIE module obtained above are cascaded using the traditional cascade neural network method to obtain a spatial context joint enhanced feature pyramid network, denoted as SCJE-FPN; Output the features obtained in step 2.1 to F in The input is sent to the spatial context joint enhancement feature pyramid network SCJE-FPN, and the spatial context joint enhancement feature pyramid network SCJE-FPN outputs the deep feature F after the feature is jointly enhanced with the spatial context. out , the encoder part of the SAR image ship target segmentation network based on the diffusion model is built; Step 3: Ship target segmentation network construction During the training process, the deep feature F in step 2 is out , using the traditional cascade random box filling strategy method, we get the noise box B t ; For the noise frame B obtained in the previous step t , processed using the traditional diffusion model method, we obtain the instance-specific filter θ0 = η(f(B t ,t)), where f(·) is the denoising network and η(·) is the fully connected layer; For the classic intersection-over-union loss function, by adding a new focusing factor α, we get the focused intersection-over-union loss function, denoted as FIoU: Specifically, given a predicted box B and a real box B gt , the FIoU loss function is defined as follows: Among them, p1 is the coordinate of the point on the upper left of the prediction box, and p2 is the coordinate of the point on the lower right of the prediction box. is the coordinate of the upper left point of the real box, is the coordinate of the point on the lower right of the real box, ρ(·) is the Euclidean distance, c is the diagonal length of the minimum bounding box of the predicted box and the real box, is the intersection over union ratio, α is the focusing factor with a value of 0.95, and IoU is the intersection over union ratio; For the focused intersection-over-union loss function FIoU obtained in the previous step, the traditional training total loss calculation method is used to obtain the total loss function in the training process in represents the detection loss, represents the segmentation loss, λ bal represents the balance hyperparameter between detection and segmentation, and its value is 5; During the inference process, the noise box B in the training process t ,After network training using the traditional denoising diffusion implicit model method, the reconstructed ship instance mask m is obtained; At this point, the decoder module of the SAR image ship target segmentation network based on the diffusion model has been completed; Output the features in step 2 to F out As input, the traditional polarized self-attention module method is used for information integration to obtain a multi-asymmetric convolutional self-attention module, denoted as MACSA; The MACSA module obtained in the previous step is processed using the traditional conditional convolution method for instance segmentation to obtain the instance-aware mask representation module, denoted as IAMR. At this point, the construction of the SAR image ship target segmentation network based on the diffusion model is completed, denoted as DiffSARShipInst; Step 4: Build a ship segmentation model The training set SSDD obtained in step 1 train or HRSID train As input, the SAR image ship target segmentation network DiffSARShipInst based on the diffusion model completed in step 3 is optimized and trained using the classic AdamW optimization method. After the training is completed, a ship segmentation model is obtained, which is recorded as DiffSARShipInst_Model; Step 5: Test the ship segmentation model Use the test set SSDD obtained in step 1 test or HRSID test , the ship segmentation model DiffSARShipInst_Model obtained in step 4 is tested using a standard network testing method to obtain the test result of the test set on DiffSARShipInst_Model, which is recorded as Result; Step 6: Evaluate the ship segmentation model Take the test result Result of the ship segmentation model obtained in step 5 as input, and use the classic COCO measurement method to obtain the mask average precision maskAP; This completes the entire method.

Images