A SAR image ship segmentation method and system based on visual basic model

Abstract

The present invention discloses a SAR image ship segmentation method and system based on a visual basic model, which relates to the field of image processing, wherein the method steps include: collecting ship SAR images to be segmented; constructing a SARSAM model, the SARSAM model includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basic SAM model; the adaptive wavelet soft threshold module is used to remove coherent spot noise in the ship SAR image, the morphological adapter is used to reduce the computing resources of the SARSAM model, and the prompter is used to solve the problem of target non-smoothness; and the SARSAM model is used to complete the segmentation of the ship SAR image. The present invention solves the problem of performance degradation of the SAM model when it is applied in the SAR field through innovative design, and at the same time shows better performance than other SAR image ship segmentation methods.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a SAR image ship segmentation method and system based on a visual basic model. Background Art

[0002] The purpose of instance segmentation is to locate the position of each target, obtain the category and give the corresponding mask. Simply put, instance segmentation can be regarded as a combination of target detection task and semantic segmentation task. At present, instance segmentation based on deep learning includes two types of methods: single-stage and two-stage. The former is continuously developed on the basis of MASK R-CNN, while the latter is developed on the basis of two-stage target detector. The difference between the two is that the single-stage method has the advantages of fast speed and small model, but the segmentation result is not fine enough; while the two-stage method can get more fine segmentation results, but the model size is often large and the reasoning speed is slow. For now, considering the actual application, affected by hardware resources and real-time performance, the single-stage method is more mainstream. For example, YOLOv8, Yolact, etc., these methods have good instance segmentation capabilities and fast reasoning speed. In addition, with the widespread application of transformers, instance segmentation methods based on visual transformers (ViT) have gradually attracted the attention of researchers, such as Cell-DETR, Mask Transfiner, etc. However, most of the instance segmentation methods are currently proposed for tasks in natural scenes. Since they do not take into account the particularity of synthetic aperture radar images, their performance will degrade when applied to the field of combined synthetic aperture radar. At present, the general visual basis model is becoming a new research hotspot, which can be migrated to various downstream tasks, providing a new solution for instance segmentation tasks. Summary of the invention

[0003] In order to solve the technical problems in the above background, the present invention proposes the following solutions:

[0004] A SAR image ship segmentation method based on a visual basic model, comprising the following steps:

[0005] Collect the ship SAR image to be segmented;

[0006] A SARSAM model is constructed, wherein the SARSAM model includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basis of a basic SAM model; the adaptive wavelet soft threshold module is used to remove coherent spot noise in a ship SAR image, the morphological adapter is used to reduce the computing resources of the SARSAM model, and the prompter is used to solve the problem of non-smoothness of the target;

[0007] The SARSAM model is used to segment the ship SAR image.

[0008] Preferably, the adaptive wavelet soft threshold module is based on a deep residual shrinkage network, uses an attention mechanism to automatically obtain a soft threshold, and combines discrete wavelet transform to perform soft threshold denoising on characteristic subbands in different frequency ranges after decomposition.

[0009] Preferably, the discrete wavelet transform method comprises:

[0010] Firstly, the ship SAR image to be segmented is used in the SAM encoder to generate shallow features with 768 channels through a large convolution kernel in a patch embedding layer to obtain a feature map;

[0011] Then, a two-dimensional discrete wavelet transform is performed on the feature map. Specifically, the feature map is down-sampled along the row direction after passing through a low-pass filter and a high-pass filter respectively to obtain a low-frequency component and a high-frequency component; then, the obtained low-frequency component and the high-frequency component are down-sampled along the column direction after passing through a low-pass filter and a high-pass filter respectively, so that the feature map finally obtains four sub-bands.

[0012] Preferably, the soft threshold denoising method includes: performing global pooling on the feature map obtained in each sub-band, taking the absolute value and generating a vector of size C*1*1; then, performing channel attention on the obtained vector to generate a vector of the same size, and then performing a dot product between the two to obtain the final threshold required for each channel.

[0013] Preferably, the morphological adapter adopts parameter efficiency fine-tuning technology, freezes the original parameters of the SAM model, and introduces an Adapter containing learnable parameters; at the same time, morphological processing is further introduced on the basis of the Adapter to form an M-Adapter, and the M-Adapter is inserted into the multi-head attention rear and MLP structure of each layer of the SAM model encoder to enhance the edge features of the target while fine-tuning.

[0014] Preferably, the YOLOv8 algorithm is used as the prompter and fed into the SARSAM model. The prompter is expressed as follows:

[0015] ,

[0016] ,

[0017] ,

[0018] ,

[0019] in, , , , denote detector, image encoder, hint encoder, and mask encoder respectively; I represents the input image; Represents the output detection result of the detector; Represents the output features of the image encoder; Represents the sparse embedding obtained by the detection box; Represents the final output mask.

[0020] The present invention also provides a SAR image ship segmentation system based on a visual basic model, the system is used to implement the above method, and includes: an acquisition module, a construction module and a segmentation module;

[0021] The acquisition module is used to acquire the ship SAR image to be segmented;

[0022] The construction module is used to construct a SARSAM model, and the SARSAM model includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basis of a SAM model; the adaptive wavelet soft threshold module is used to remove coherent spot noise in a ship SAR image, the morphological adapter is used to reduce the computing resources of the SARSAM model, and the prompter is used to solve the problem of non-smoothness of the target;

[0023] The segmentation module is used to complete the segmentation of the ship SAR image using the SARSAM model.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention solves the problem of performance degradation when the SAM model is applied in the SAR field through innovative design, and at the same time shows better performance than other SAR image ship segmentation methods. The adaptive wavelet soft threshold denoising module is used to decompose the shallow features into feature sub-bands with different frequencies and directions, and then the required threshold is automatically obtained to perform soft thresholding on each feature sub-band to achieve the denoising process of the features. Secondly, in order to save computing resources as much as possible, the present invention adopts efficient parameter fine-tuning technology, introduces morphological operations based on the standard adapter Adapter, and only a small amount of parameter training is required to achieve model fine-tuning on the SAR data set.

[0026] The present invention can be trained on a public SAR data set and the trained weights can be saved. The trained model can then be deployed to the edge device with weights. When a ship is equipped with this device, it can accurately locate nearby ships in the sea area, so as to better understand the dynamics of other ships and the distribution of ships in the current sea area. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of an adaptive wavelet soft threshold module according to an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the M-Adapter structure and its morphological construction process according to an embodiment of the present invention; wherein: Figure 2 Part (a) shows the structure of M-Adapter; Figure 2 Part (b) shows the construction process of morphological operation;

[0030] Figure 3 It is a structural schematic diagram of the SARSAM model of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Embodiment 1

[0034] This embodiment proposes a SAR image ship segmentation method based on a visual basic model, the steps comprising:

[0035] S1. Collect the ship SAR image to be segmented.

[0036] Synthetic aperture radar (SAR) is an all-weather, all-day active earth observation system. Compared with optical imaging technology, it is not affected by weather and lighting conditions, has strong penetration, and can obtain high-resolution remote sensing images. Therefore, it is widely used in ocean monitoring, geographic mapping and military fields.

[0037] As a method that combines target detection and segmentation, instance segmentation helps to accurately locate and classify single target instances in remote sensing images while highlighting their contours. Some instance segmentation methods proposed initially are mainly aimed at optical scenes. However, synthetic aperture radar is affected by coherent interference during the imaging process. There are a lot of scattered noise, side lobes generated by scattered imaging, and complex background interference in the image.

[0038] S2. Construct the SARSAM model.

[0039] Visual basic models have shown strong capabilities in various visual tasks. As a basic segmentation model, it is of great value to migrate SAM to SAR image ship segmentation. SAM is a model trained under massive natural scene optical data. Since optical images are very different from SAR images, directly migrating it to SAR tasks will greatly reduce the segmentation performance. Usually, migrating the model to downstream tasks requires fine-tuning the entire model, but the SAM model is huge and completely fine-tuning it requires high computing resources. Secondly, SAR images usually have a large amount of coherent speckle noise, side lobes generated by scattered imaging, and complex background interference, which will directly affect the segmentation of ship targets. In addition, since the targets in SAR images are composed of some scattered points, the edge contours of the targets are mostly discrete points and are not smooth, which is also not conducive to segmentation.

[0040] Therefore, the SARSAM model proposed in this embodiment includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basic SAM model; the adaptive wavelet soft threshold module is used to remove the coherent spot noise in the ship SAR image, the morphological adapter is used to reduce the computing resources of the SARSAM model, and the prompter is used to solve the problem of non-smooth targets. Specifically, the adaptive wavelet soft threshold module uses secondary wavelet decomposition to decompose the shallow features into different frequency sub-bands (four sub-bands in this embodiment), and then performs soft threshold processing on each sub-band respectively through the learned threshold to achieve the purpose of denoising; in order to solve the problem of complete fine-tuning consuming computing resources, this embodiment introduces two morphological adapters in each layer of the encoder, freezes the original parameters of the model during the training process, and only updates a small number of parameters of the morphological adapter, which greatly reduces the computing resources. The opening operation in the morphological processing can realize the smoothing of the contours of objects in the image, remove protruding parts and small depressions. Therefore, in order to solve the problem of non-smooth target edges, this embodiment adds morphological operations on the basis of the adapter. By reparameterizing the grouped convolution, the grouped convolution can be approximated as an opening operation and the appropriate structural elements can be automatically learned during the training process.

[0041] The detailed structures and specific applications of the three modules proposed in this embodiment will be described in detail below.

[0042] Adaptive wavelet soft threshold module: Aperture radar imaging is an interference imaging technology, and its image often contains a large amount of coherent spot noise, which makes the target in the image blurred and has an adverse effect on the segmentation task. In the image denoising task, threshold denoising is a commonly used denoising method, which is mainly divided into hard threshold denoising and soft threshold denoising. Compared with hard threshold denoising, the image after soft threshold denoising is smoother, but both methods require the selection of a reasonable threshold. Due to the complexity of the frequency characteristics of scattering noise, it is mainly concentrated in the high-frequency part of the image, and also contains low-frequency components. Therefore, it is obviously more appropriate to denoise the image separately in different frequency ranges. To this end, this embodiment is based on a deep residual shrinkage network, uses an attention mechanism to automatically obtain a soft threshold, and combines discrete wavelet transform to perform soft threshold denoising on the feature subbands in different frequency ranges after decomposition. Its specific structure is as follows Figure 1 shown.

[0043] The above discrete wavelet transform process specifically includes:

[0044] First, the input image (the ship SAR image to be segmented) is generated into a shallow feature with 768 channels in the SAM encoder through a large convolution kernel in the patch embedding layer to obtain a feature map; then the feature map is subjected to a two-dimensional discrete wavelet transform. Specifically, given a feature map , firstly, downsample the low-frequency component and high-frequency component along the row direction through a low-pass and high-pass filter respectively; then downsample the low-frequency component and high-frequency component along the column direction through a low-pass and high-pass filter respectively, and finally obtain four sub-bands in the feature map: , , , ; The specific calculation formula is as follows:

[0045] , (1)

[0046] in, represents the real number space; C Indicates the number of channels; H Indicates the height of the feature; W Indicates the width of the feature; Represents pixel coordinates; Represents the value of the point in the feature graph whose coordinate row and column are even numbers; Represents the value of the point in the feature graph whose coordinate row is even and column is odd; Represents the value of the point in the feature graph whose coordinate row is odd and column is even; Represents the value of the point in the feature map whose coordinate row and column are both odd numbers.

[0047] The above soft threshold denoising process includes:

[0048] Since the scattered noise contains high-frequency and low-frequency components, it is necessary to perform soft threshold denoising on the four sub-bands in all frequency ranges. It is worth noting that each sub-band has a certain number of channels, and these channels are not exactly the same. It is obviously unreasonable to use the same threshold denoising for all channels. Therefore, the AWST module proposed in this embodiment adopts an attention mechanism to automatically obtain a suitable threshold during the network training process. First, the feature map in the sub-band is globally pooled, and the absolute value is taken to generate a size of C *1*1. Then, the vector is passed through the channel attention to generate a vector of the same size, and then the two are multiplied to obtain the final threshold required for each channel. Since the hard threshold will produce discontinuity, this paper uses a soft threshold function to process the decomposed feature subbands. The function expression is shown in Formula 2. Finally, the feature subbands are reconstructed through IWT to complete the entire threshold denoising process;

[0049] , (2)

[0050] In the formula, represents the soft threshold function; ζ Represents pixel value; τ Indicates the threshold value.

[0051] Morphological Adapter: In visual tasks, many state-of-the-art models are built on ViT (e.g., SwinTransformer, PVT, VITdet), and their model sizes are often very large. SAM's image encoder is a VIT structure with three different model sizes: ViT-H, VIT-L, and VIT-B. For downstream tasks, the general measure is to fully fine-tune the model on the downstream dataset. However, for the smallest model in SAM, the number of parameters is close to 90 million, and full fine-tuning will consume a lot of computing resources. Therefore, this embodiment adopts a parameter-efficient fine-tuning technique, that is, freezing the original parameters of the SAM model and introducing an Adapter containing a small number of learnable parameters. However, due to the particularity of synthetic aperture radar interferometric imaging, the targets in the image are composed of a large number of scattered points, and the target edge contours are mostly discrete points and not smooth, which is extremely unfavorable for segmentation tasks. This embodiment further introduces morphological processing on the basis of Adapter to smooth the edge of the target, and designs a new M-Adapter, which is inserted into the multi-head attention and MLP structure of each layer of the SAM model encoder to enhance the edge features of the target while fine-tuning. The specific structure is as follows: Figure 2 As shown in part (a), its morphological construction process Figure 2 As shown in part (b) of .

[0052] The specific process of the above adaptive morphological operation is as follows:

[0053] Morphological processing is a commonly used tool in image processing, including operations such as dilation, erosion, opening and closing. Among them, the opening operation can achieve the smoothing of the target contour. However, the ship target in the synthetic aperture radar image has the characteristics of multi-scale and multi-directionality, and the classic opening operation requires a fixed structural element, which is difficult to achieve the best effect. Therefore, this embodiment simulates the morphological operation through two-bit convolution and automatically learns the required structural element shape during the network training process. First, the size of the morphological processing template is determined to be 3*3. Construct a 3*3 two-dimensional grouped convolution, the number of groups is the number of input channels, and ensure that the number of output channels is 9 times the number of input channels. In morphological processing, the dilation operation takes the maximum value of the pixel in the structural element, while the corrosion operation takes the minimum value. The opening operation corrodes first and then dilates. The shape of the structural element can be arbitrary, and its corresponding value in the template is 1. For this reason, it is necessary to reconstruct the parameters of the two-dimensional convolution. The Gumbel_softmax function is used to reconstruct the weights of the two-dimensional grouped convolution. The function definition is shown in formulas (3) and (4). When it is closer to 0, it is closer to one-hot encoding, that is, the weight of the two-dimensional convolution kernel has only one value 1 after reparameterization. Therefore, the 9 convolution kernels in each group of grouped convolutions correspond to any position in the 3*3 area, completing the construction of structural elements of arbitrary shapes.

[0054] , (3)

[0055] , (4)

[0056] in, z Represents the set of parameter values ​​after reparameterization; z i Indicates parameter value; N Indicates the number of parameters; , represents gumbel noise; Indicates the original i parameter values; Indicates the original j parameter values; T Represents the temperature coefficient.

[0057] Prompter: The target segmentation effect of SAM is heavily dependent on the quality of the prompts. Higher-quality prompts will produce better segmentation effects. This is because the ship target in the synthetic aperture radar image is a collection of scattered points, which will appear discontinuous. In addition, in the near-shore scene, the edges between the ship target and other targets are not clear. Unlike visible light images, some important targets in SAR images are difficult to identify. It is difficult for some personnel in professional fields to give reasonable prompts when using SAM to interactively segment ship targets, resulting in incorrect segmentation results. In order to help most people use SARSAM to complete ship segmentation in the SAR field, this embodiment uses the YOLOv8 algorithm as a prompter to provide relevant personnel with reliable box prompts, which are sent to the SARSAM model to achieve ship segmentation. This part is defined as follows:

[0058] , (5)

[0059] , (6)

[0060] , (7)

[0061] , (8)

[0062] in, , , , denote detector, image encoder, hint encoder, and mask encoder respectively; I represents the input image; Represents the output detection result of the detector; Represents the output features of the image encoder; Represents the sparse embedding obtained by the detection box; Represents the final output mask.

[0063] The final constructed SARSAM model is as follows Figure 3 shown.

[0064] S3. Use the SARSAM model to complete the segmentation of ship SAR images.

[0065] Finally, the constructed SARSAM model is used to complete the segmentation of ship SAR images.

[0066] Embodiment 2

[0067] In order to verify the effectiveness of the model constructed in this embodiment, this embodiment conducted experiments on two SAR public datasets (SSDD and HRSID), and compared them with other SAR ship segmentation methods.

[0068] The results of SARSAM on the SSDD validation set are shown in Table 1. The SARSAM model constructed in this implementation shows the best performance among all methods. and They reached 88.48% and 98.95% respectively;

[0069] Table 1

[0070] .

[0071] The results of SARSAM on the HRSID validation set are shown in Table 2. The SARSAM model constructed in this example shows the best performance among all methods. and They reached 90.29% and 96.56% respectively;

[0072] Table 2

[0073] .

[0074] To verify the effectiveness of the innovative features of this method, this embodiment conducted a series of ablation experiments on the SSDD and HRSID datasets. The results are shown in Tables 3 and 4. Table 3 is the ablation experiment of SARSAM on the SSDD dataset, and Table 4 is the ablation experiment of SARSAM on the HRSID dataset.

[0075] Table 3

[0076] .

[0077] Table 4

[0078] .

[0079] (1) Analysis of the effectiveness of adaptive wavelet soft thresholding (AWST): In order to explore the effectiveness of the AWST module on the designed method SARSAM, this embodiment adds the AWST module on the basis of the baseline, and then trains on the SSDD and HRSID datasets respectively. As shown in Tables 3 and 4, the introduction of AWST improves the segmentation effect of the ship target by this method, which increases by 9.78% and 2.35%, and by 5.35% and 1.59%, respectively. These improvements are due to the fact that the AWST module eliminates the influence of coherent speckle noise, which provides important support for the subsequent operation of the model.

[0080] (2) Analysis of the effectiveness of the morphological adapter (M-Adapter): Fully fine-tuning the SAM model on the SAR dataset requires a lot of computing resources, which is not friendly to researchers. The adapter is an efficient parameter fine-tuning technology that can achieve fine-tuning of the model while only updating a small number of parameters. Secondly, the opening operation in the morphological processing can smooth the target contours in the SAR image and eliminate some prominent scattering points. As shown in Tables 3 and 4, during the training phase, we further introduced the M-Adapter after introducing the AWST module in the baseline. This module further improved the performance of SARSAM for ship target segmentation, which increased by 7.29% and 11.17%, and by 0.15% and 4.74%, respectively. Compared with the ordinary adapter, the morphological adapter of this embodiment is more effective.

[0081] Embodiment 3

[0082] This embodiment also provides a SAR image ship segmentation system based on a visual basic model, including: an acquisition module, a construction module and a segmentation module; the acquisition module is used to acquire the ship SAR image to be segmented; the construction module is used to construct a SARSAM model, and the SARSAM model includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basic SAM model; the adaptive wavelet soft threshold module is used to remove coherent spot noise in the ship SAR image, the morphological adapter is used to reduce the computing resources of the SARSAM model, and the prompter is used to solve the problem of target non-smoothness; the segmentation module is used to complete the segmentation of the ship SAR image using the SARSAM model.

[0083] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for ship segmentation in SAR images based on a visual basic model, characterized in that the steps include: Collect the ship SAR image to be segmented; A SARSAM model is constructed, and the SARSAM model includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basic SAM model; the adaptive wavelet soft threshold module is used to remove the coherent spot noise in the ship SAR image, and the adaptive wavelet soft threshold module is based on a deep residual shrinkage network, and uses the attention mechanism to automatically obtain the soft threshold, and combines the discrete wavelet transform to perform soft threshold denoising on the feature subbands of different frequency ranges after decomposition; the morphological adapter is used to reduce the computing resources of the SARSAM model, and the morphological adapter adopts parameter efficiency fine-tuning technology to freeze the original parameters of the SAM model, and introduces the Adapter containing learnable parameters; at the same time, morphological processing is further introduced on the basis of the Adapter to form an M-Adapter, and the M-Adapter is inserted into the multi-head attention rear and MLP structure of each layer of the SAM model encoder to enhance the edge features of the target while fine-tuning; the prompter is used to solve the problem of non-smoothness of the target, and the YOLOv8 algorithm is used as the prompter to be sent to the SARSAM model, and the prompter is expressed as follows: ， ， ， ， in, , , , They represent detector, image encoder, hint encoder and mask encoder respectively; I represents input image; Represents the output detection result of the detector; Represents the output features of the image encoder; Represents the sparse embedding obtained by the detection box; Represents the final output mask; The SARSAM model is used to segment the ship SAR image.

2. The SAR image ship segmentation method based on the visual basic model according to claim 1 is characterized in that: The discrete wavelet transform method comprises: Firstly, the ship SAR image to be segmented is used in the SAM encoder to generate shallow features with 768 channels through a large convolution kernel in a patch embedding layer to obtain a feature map; Then, a two-dimensional discrete wavelet transform is performed on the feature map. Specifically, the feature map is down-sampled along the row direction after passing through a low-pass filter and a high-pass filter respectively to obtain a low-frequency component and a high-frequency component; then, the obtained low-frequency component and the high-frequency component are down-sampled along the column direction after passing through a low-pass filter and a high-pass filter respectively, so that the feature map finally obtains four sub-bands.

3. The SAR image ship segmentation method based on the visual basic model according to claim 2 is characterized in that: The soft threshold denoising method includes: performing global pooling on the feature map in each sub-band, taking the absolute value and generating a vector of size C*1*1; then, performing channel attention on the obtained vector to generate a vector of the same size, and then performing a dot multiplication on the two to obtain the final threshold required for each channel.

4. A SAR image ship segmentation system based on a visual basic model, the system is used to implement the method according to any one of claims 1 to 3, characterized in that: include: Acquisition module, construction module and segmentation module; The acquisition module is used to acquire the ship SAR image to be segmented; The construction module is used to construct a SARSAM model, and the SARSAM model includes: introducing an adaptive wavelet soft threshold module, a morphological adapter and a prompter on the basis of a SAM model; The adaptive wavelet soft threshold module is used to remove the coherent spot noise in the ship SAR image, the morphological adapter is used to reduce the computing resources of the SARSAM model, and the prompter is used to solve the problem of target non-smoothness; The segmentation module is used to complete the segmentation of the ship SAR image using the SARSAM model.

Images