An Infrared Ship Semantic Segmentation Method and System Based on SAM Enhancement Domain Adaptation

Abstract

The application discloses an infrared ship semantic segmentation method and system based on SAM enhanced domain self-adaption, five front layers of ResNet-101 are used as a backbone network to extract features of an image, a segmentation loss is calculated for a source domain image, meanwhile, a main prediction graph output by a main classifier, an auxiliary prediction graph output by an auxiliary classifier are used for respectively performing adversarial learning on the main classifier and a main discriminant network, the auxiliary classifier and an auxiliary discriminant network, common knowledge of the source domain and a target domain is extracted, and a pseudo label of the target domain is obtained; secondly, prompt information is extracted from the pseudo label, and then the prompt information and the target domain image are input into a SAM model to obtain an enhanced pseudo label; finally, fine tuning of a network is performed on the enhanced pseudo label and the target domain image, and then the segmentation accuracy of an infrared image is improved. The method based on the SAM enhanced domain self-adaption model can effectively improve the average intersection over union of infrared ship image semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer vision semantic segmentation technology, specifically to an infrared ship semantic segmentation method and system based on SAM-enhanced domain adaptation. Background Technology

[0002] Ships serve military defense and strategic deterrence. With the rapid development of modern technology, infrared imaging has become an important visual extension tool, providing a reliable means of observation by capturing the thermal radiation emitted by objects. However, the low contrast and high noise characteristics of infrared images make semantic segmentation of infrared ships a challenging task. Meanwhile, visible light ship images obtained under good weather conditions have prominent targets and clear edges, making pixel-level annotation relatively easy. Therefore, using domain adaptation techniques to train a segmentation network on visible light ship images to achieve good segmentation accuracy on infrared ship images is a promising approach to improving the segmentation accuracy of infrared ship images. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an infrared ship semantic segmentation method and system based on SAM-enhanced domain adaptation.

[0004] This invention discloses an infrared ship semantic segmentation method based on SAM enhancement domain adaptation, comprising:

[0005] Step 1: Obtain the visible light ship image dataset X s and the corresponding label Y S Obtain the infrared ship image dataset X t , will X t Divided into training set X train and test set X test The visible light ship image dataset serves as the source domain, while the infrared ship image dataset serves as the target domain.

[0006] Step 2: Construct an adversarial domain adaptation segmentation model based on DeepLab. The adversarial domain adaptation segmentation model consists of a segmentation network and a discriminant network. The segmentation network includes a backbone network, a main classifier, and an auxiliary classifier. The backbone network consists of the first 5 convolutional layers of ResNet-101. The main classifier is connected after the fifth convolutional layer, and the auxiliary classifier is connected after the fourth convolutional layer. The discriminant network includes a main discriminant network and an auxiliary discriminant network. The main discriminant network is connected after the main classifier, and the auxiliary discriminant network is connected after the auxiliary classifier.

[0007] Step 3: Transfer the visible light ship image dataset X s and infrared ship image dataset X tThe input is fed into the backbone network to obtain feature maps. The feature maps are then fed into the main classifier and the auxiliary classifier to obtain the main and auxiliary semantic segmentation prediction maps. The main and auxiliary semantic segmentation prediction maps are then fed into the main discriminant network and the auxiliary discriminant network to obtain the judgment results of the segmentation prediction maps.

[0008] Step 4: Calculate the source domain segmentation loss based on the main and auxiliary semantic segmentation prediction graphs, and calculate the discrimination loss based on the judgment results of the prediction graphs;

[0009] Step 5: Extract the required Prompt information from the segmentation prediction map of the target domain image; wherein, the extraction of the required Prompt information includes: extracting positive cues using sliding window correction mask and point clustering on Canny edges, and extracting negative cues using geometric offset;

[0010] Step 6: Input the obtained positive and negative prompts and infrared ship images into SAM to obtain the infrared ship mask map; perform semantic assignment on the mask map without semantic information output by SAM to obtain pseudo-labels of target domains with semantic categories after SAM enhancement.

[0011] Step 7: Continue training the segmentation network using the SAM-enhanced pseudo-labels;

[0012] Step 8: Test the trained SAM-based domain adaptation segmentation network on the test set and calculate the average crossover ratio.

[0013] As a further improvement of the present invention, in step 1, X t The training set X is divided into two parts at an 8:2 ratio. train and test set X test X t Contains N t Samples, training set X train Contains N train Samples, test set X test Contains N test One sample.

[0014] As a further improvement of the present invention, in step 2...

[0015] The main classifier and the auxiliary classifier have the same structure, each consisting of a dilated spatial pyramid pooling layer and an upsampling layer. The dilated spatial pyramid pooling layer is used to capture multi-scale image context information, with pooling sampling rates of 6, 12, 18 and 24, respectively. The upsampling layer is used to restore the pooled feature map to the size of the original image and generate the final semantic segmentation prediction map.

[0016] Both the main discriminant network and the auxiliary discriminant network are five-layer convolutional networks with a kernel size of 4×4, a stride of 2, and the number of channels of 64, 128, 256, 512 and 1, respectively. The output of the discriminant network represents the domain discrimination information. A result close to 1 indicates that the input prediction map is judged as the source domain, and a result close to 0 indicates that the input prediction map is judged as the target domain.

[0017] As a further improvement of the present invention, in step 4...

[0018] Source domain segmentation loss is used to ensure the performance of the segmentation network in the source domain. The loss functions of the main and auxiliary classifiers of the segmentation network are expressed as follows:

[0019]

[0020] In the formula, and Let these represent the losses of the main classifier and the auxiliary classifier in the source domain, respectively. This represents the i-th sample in the source domain; The true label represents the i-th sample in the source domain; N represents the number of training samples; C is the total number of categories; h and w are the image sizes; F T (·) represents the feature extractor for the teacher network. and These are the primary and secondary classifiers for the teacher network, respectively.

[0021] The final segmentation loss is expressed as:

[0022]

[0023] In the formula, and Let α represent the losses of the main classifier and the auxiliary classifier in the source domain, respectively. c With (1-α) c ) represent the coefficients of the main classifier and the auxiliary classifier, respectively.

[0024] As a further improvement of the present invention, in step 4...

[0025] During training, two discriminant networks, a main network and an auxiliary network, are used to discriminate the outputs of the main and auxiliary classifiers of the segmentation network, respectively. Specifically, the main prediction maps of the source domain and the target domain are input into the main discriminant network, and the auxiliary prediction maps of the source domain and the target domain are input into the auxiliary discriminant network to obtain their respective results.

[0026] The domain discriminant loss functions of the two discriminant networks are expressed as follows:

[0027]

[0028] In the formula, and These represent the inter-domain discrimination losses of the primary and secondary discrimination networks, respectively. This represents the i-th sample in the source domain; F represents the l-th sample in the target domain; T (·) represents the feature extractor for the teacher network; and D represents the primary and secondary classifiers of the teacher network, respectively; p (·) and D a (·) represent the primary and secondary discriminant networks, respectively;

[0029] Ultimately, the overall discriminant loss is expressed as:

[0030]

[0031] In the formula, and Let α represent the inter-domain discrimination loss of the master and slave discriminant networks, respectively. D With (1-α) D ) represent the weight parameters of the main discriminant network and the auxiliary discriminant network, respectively.

[0032] As a further improvement of the present invention, in step 5...

[0033] Use a sliding window to refine the mask and extract positive hints, including:

[0034] Information such as object shape and semantic category can be obtained from the predicted image. However, due to noise in the target domain infrared image, the predicted image output by the model also contains some discrete noise points. Therefore, discrete noise points are filtered out by counting the number of pixels with semantic information within the window. Specifically, a sliding window of size m×n is used to traverse the image. For each pixel within the window, a threshold function counts the number of pixels with a category (i.e., non-zero pixels). If this number exceeds the threshold, it is equivalent to the presence of enough "interesting" pixels within the window, and the pixel at the center of the window is marked as 255.

[0035] Predicted map (pseudo-label) of the target domain image Size is H×W, Let represent the pixel value at (x, y) in the predicted image. The sliding window size is h × w, where h = w = bs - w. For each sliding window, count the number of non-zero pixels within the window, centered at (x, y).

[0036]

[0037] In the formula, It is an indicator function, when y′ tWhen (x+i,y+j)≠0, Otherwise, it is 0; if the number of non-zero pixels exceeds The center pixel (x, y) within the window is marked as 255; this yields the thinned mask image y′. t After (x,y), the coordinates of non-zero pixels in the mask image are counted, and 5 points are selected at equal intervals as the coordinates of the Prompt points and stored in array P1; at this time, the average value of these 5 points is calculated to obtain the center point of the object to be segmented, denoted as C1(x1,y1).

[0038] Positive hints are extracted using point clustering on Canny edges, including:

[0039] Infrared images often have blurred edges, and a single point of interest pixel may not be enough to fully represent the information of an infrared ship. Adding edge information can help the model better capture the shape and contour of the target, thus improving segmentation accuracy.

[0040] First, the Canny operator is used to extract edge information, and then K-means clustering is performed on these points to extract positive cue points. Specifically: First, within the region centered at C1(x1,y1) and with radius R, the Canny edge detection algorithm is used to extract the edge information of the infrared ship, represented as:

[0041] edges(x,y)=Canny(I(x,y)) (8)

[0042] In the formula, (x-x1) 2 +(y-y1) 2 ≤R 2 ; I represents an infrared image;

[0043] Then, K-means clustering is used to cluster the points obtained from these edge information; finally, the centroids of different clusters are output and stored in set P2.

[0044] Extract negative hints using geometric offsets, including:

[0045] The above two steps can obtain foreground cue points; however, due to noise, occlusion, and the low contrast of infrared images, the foreground cue points may not be accurate enough. To provide SAM with more comprehensive prompt information, this invention selects negative cue points around the foreground cue points to effectively supplement their information. Specifically, taking the mean C2 of sets P1 and P2 as the center, one point is selected directly above and below the center point in the vertical direction at a distance of 90 pixels from the center point, and four points on the four diagonals are selected at a distance of 100 pixels from the center point as background negative cue points.

[0046] As a further improvement of the present invention, in step 6, the pseudo-label of the target domain having a semantic category is represented as follows:

[0047]

[0048] In the formula, P t (x,y) represents the prediction class index of a pixel in the target domain image in the first-stage prediction map; M t (x,y) represents the mask without semantic information obtained initially by SAM; These are pseudo-labels for target domains with semantic categories.

[0049] As a further improvement of the present invention, in step 7...

[0050] The losses of the main and auxiliary classifiers of the segmentation network during further training can be expressed as follows:

[0051]

[0052] in, Let F represent the predicted image obtained by the student network for the j-th target domain image. S (·) represents the feature extractor for the student network. and These represent the primary and secondary classifiers of the student network, respectively. N represents the pseudo-label of the j-th image in the target domain. t This indicates the number of images in the target domain.

[0053] The present invention also discloses an infrared ship semantic segmentation system based on SAM enhancement domain adaptation, which is used to implement the above-mentioned infrared ship semantic segmentation method based on SAM enhancement domain adaptation.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] This invention designs prompt information, uses the SAM model to obtain enhanced pseudo-labels, and finally trains on the target domain infrared image to obtain an adaptation model from the visible light domain to the infrared domain. This improves the network model's ability to classify and locate ships in infrared images and overcomes the problem of low average intersection-union ratio in infrared ship semantic segmentation. Attached Figure Description

[0056] Figure 1 This is a flowchart of the infrared ship semantic segmentation method based on SAM enhancement domain adaptation disclosed in this invention;

[0057] Figure 2 A structural diagram of adversarial learning in a backbone network;

[0058] Figure 3 SAM enhancement domain adaptation structure diagram;

[0059] Figure 4 for Figure 3 The visualization process of extracting Prompt information. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The present invention will now be described in further detail with reference to the accompanying drawings:

[0062] This invention provides a semantic segmentation method and system for infrared ships based on SAM augmentation domain adaptation. It utilizes a semantic segmentation model based on SAM augmentation domain adaptation to perform semantic segmentation on infrared ship images. The SAM model is a state-of-the-art, universal image segmentation model designed to segment any object in any type of image. The main idea is to obtain enhanced pseudo-labels by leveraging the powerful segmentation capabilities of SAM through self-supervision. Finally, the segmentation model is trained on the enhanced pseudo-labels, thereby improving the segmentation accuracy of the model on infrared images in the target domain.

[0063] like Figures 1-4 As shown, this invention provides an infrared ship semantic segmentation method based on SAM enhancement domain adaptation, comprising:

[0064] Step 1: Obtain the visible light ship image dataset X s and the corresponding label Y S Obtain the infrared ship image dataset X t , will X t Divided into training set X train and test set X test The visible light ship image dataset serves as the source domain, and the infrared ship image dataset serves as the target domain; X t The training set X is divided into two parts at an 8:2 ratio. train and test set X test X t Contains N t Samples, training set X train Contains N train Samples, test set X test Contains N testOne sample; first, a segmentation network based on SAM-enhanced domain adaptation is completed using the training set, and then tested using the test set;

[0065] Step 2: Construct an adversarial domain adaptive segmentation model based on DeepLab. The adversarial domain adaptive segmentation model consists of a segmentation network and a discriminant network; where, for example... Figure 2 As shown, the segmentation network includes a backbone network, a main classifier, and an auxiliary classifier. The backbone network consists of the first five convolutional layers of ResNet-101. The main classifier is connected after the fifth convolutional layer, and the auxiliary classifier is connected after the fourth convolutional layer. The discriminator network includes a main discriminator network and an auxiliary discriminator network. The main discriminator network is connected after the main classifier, and the auxiliary discriminator network is connected after the auxiliary classifier.

[0066] The main classifier and the auxiliary classifier have the same structure, each consisting of a dilated spatial pyramid pooling layer and an upsampling layer. The dilated spatial pyramid pooling layer is used to capture multi-scale image context information, with pooling sampling rates of 6, 12, 18 and 24, respectively. The upsampling layer is used to restore the pooled feature map to the size of the original image and generate the final semantic segmentation prediction map.

[0067] Both the main discriminant network and the auxiliary discriminant network are five-layer convolutional networks with a kernel size of 4×4, a stride of 2, and the number of channels of 64, 128, 256, 512 and 1, respectively. The output of the discriminant network represents the domain discrimination information. A result close to 1 indicates that the input prediction map is judged as the source domain, and a result close to 0 indicates that the input prediction map is judged as the target domain.

[0068] Step 3: Transfer the visible light ship image dataset X s and infrared ship image dataset X t The input is fed into the backbone network to obtain feature maps. These feature maps are then fed into the main classifier and the auxiliary classifier to obtain main and auxiliary semantic segmentation prediction maps. Both main and auxiliary semantic segmentation prediction maps are fed into the main discriminant network and the auxiliary discriminant network to obtain the judgment results of the segmentation prediction maps. That is, adversarial learning is performed through the main classifier and the main discriminant network, and the auxiliary classifier and the auxiliary discriminant network, respectively, to extract the knowledge shared by the source domain and the target domain and obtain pseudo-labels for the target domain.

[0069] Step 4: Calculate the source domain segmentation loss based on the main and auxiliary semantic segmentation prediction graphs, and calculate the discriminant loss based on the judgment results of the prediction graphs; whereby,

[0070] Source domain segmentation loss is used to ensure the performance of the segmentation network in the source domain. The loss functions of the main and auxiliary classifiers of the segmentation network are expressed as follows:

[0071]

[0072] In the formula, and Let these represent the losses of the main classifier and the auxiliary classifier in the source domain, respectively. This represents the i-th sample in the source domain; The true label represents the i-th sample in the source domain; N represents the number of training samples; C is the total number of categories; h and w are the image sizes; F T (·) represents the feature extractor for the teacher network. and These are the primary and secondary classifiers for the teacher network, respectively.

[0073] The final segmentation loss is expressed as:

[0074]

[0075] In the formula, and Let α represent the losses of the main classifier and the auxiliary classifier in the source domain, respectively. c With (1-α) c ) represent the coefficients of the main classifier and the auxiliary classifier, respectively.

[0076] During training, two discriminant networks, a main network and an auxiliary network, are used to discriminate the outputs of the main and auxiliary classifiers of the segmentation network, respectively. Specifically, the main prediction maps of the source domain and the target domain are input into the main discriminant network, and the auxiliary prediction maps of the source domain and the target domain are input into the auxiliary discriminant network to obtain their respective results.

[0077] The domain discriminant loss functions of the two discriminant networks are expressed as follows:

[0078]

[0079] In the formula, and These represent the inter-domain discrimination losses of the primary and secondary discrimination networks, respectively. This represents the i-th sample in the source domain; F represents the l-th sample in the target domain; T (·) represents the feature extractor for the teacher network; and D represents the primary and secondary classifiers of the teacher network, respectively; p (·) and D a (·) represent the primary and secondary discriminant networks, respectively;

[0080] Ultimately, the overall discriminant loss is expressed as:

[0081]

[0082] In the formula, and Let α represent the inter-domain discrimination loss of the master and slave discriminant networks, respectively. DWith (1-α) D ) represent the weight parameters of the main discriminant network and the auxiliary discriminant network, respectively.

[0083] Step 5: Extract the required Prompt information from the segmentation prediction map of the target domain image; the extraction of the required Prompt information includes: extracting positive cues using sliding window correction mask and point clustering on Canny edges, and extracting negative cues using geometric offsets; specifically:

[0084] Use a sliding window to refine the mask and extract positive hints, including:

[0085] Information such as object shape and semantic category can be obtained from the predicted image. However, due to noise in the target domain infrared image, the predicted image output by the model also contains some discrete noise points. Therefore, discrete noise points are filtered out by counting the number of pixels with semantic information within the window. Specifically, a sliding window of size m×n is used to traverse the image. For each pixel within the window, a threshold function counts the number of pixels with a category (i.e., non-zero pixels). If this number exceeds the threshold, it is equivalent to the presence of enough "interesting" pixels within the window, and the pixel at the center of the window is marked as 255.

[0086] Predicted map (pseudo-label) of the target domain image Size is H×W, Let represent the pixel value at (x, y) in the predicted image. The sliding window size is h × w, where h = w = bs - w. For each sliding window, count the number of non-zero pixels within the window, centered at (x, y).

[0087]

[0088] In the formula, It is an indicator function, when y′ t When (x+i,y+j)≠0, Otherwise, it is 0; if the number of non-zero pixels exceeds The center pixel (x, y) within the window is marked as 255; this yields the thinned mask image y′. t After (x,y), the coordinates of non-zero pixels in the mask image are counted, and 5 points are selected at equal intervals as the coordinates of the Prompt points and stored in array P1; at this time, the average value of these 5 points is calculated to obtain the center point of the object to be segmented, denoted as C1(x1,y1).

[0089] Positive hints are extracted using point clustering on Canny edges, including:

[0090] Infrared images often have blurred edges, and a single point of interest pixel may not be enough to fully represent the information of an infrared ship. Adding edge information can help the model better capture the shape and contour of the target, thus improving segmentation accuracy.

[0091] First, the Canny operator is used to extract edge information, and then K-means clustering is performed on these points to extract positive cue points. Specifically: First, within the region centered at C1(x1,y1) and with radius R, the Canny edge detection algorithm is used to extract the edge information of the infrared ship, represented as:

[0092] edges(x,y)=Canny(I(x,y)) (8)

[0093] In the formula, (x-x1) 2 +(y-y1) 2 ≤R 2 ; I represents an infrared image;

[0094] Then, K-means clustering is used to cluster the points obtained from these edge information; finally, the centroids of different clusters are output and stored in set P2.

[0095] Extract negative hints using geometric offsets, including:

[0096] The above two steps can obtain foreground cue points; however, due to noise, occlusion, and the low contrast of infrared images, the foreground cue points may not be accurate enough. To provide SAM with more comprehensive prompt information, this invention selects negative cue points around the foreground cue points to effectively supplement their information. Specifically, with C2 (the mean of all the positive cue points mentioned above, i.e., the mean of sets P1 and P2) as the center, one point is selected directly above and below it in the vertical direction at a distance of 90 pixels from the center point, and four points on the four diagonals are selected at a distance of 100 pixels from the center point as background negative cue points.

[0097] Step 6: Input the obtained positive and negative prompts and the infrared ship image into SAM to obtain the infrared ship mask image; perform semantic assignment on the mask image without semantic information output by SAM to obtain pseudo-labels of the target domain with semantic categories after SAM enhancement; wherein, the pseudo-labels of the target domain with semantic categories are represented as:

[0098]

[0099] In the formula, P t (x,y) represents the prediction class index of a pixel in the target domain image in the first-stage prediction map; M t(x,y) represents the mask without semantic information obtained initially by SAM; These are pseudo-labels for target domains with semantic categories.

[0100] Step 7: Continue training the segmentation network using the SAM-enhanced pseudo-labels, i.e., fine-tune the network on the enhanced pseudo-labels and target domain images to improve the segmentation accuracy of infrared images; the losses of the main and auxiliary classifiers of the segmentation network during further training can be expressed as follows:

[0101]

[0102] in, Let F represent the predicted image obtained by the student network for the j-th target domain image. S (·) represents the feature extractor for the student network. and These represent the primary and secondary classifiers of the student network, respectively. N represents the pseudo-label of the j-th image in the target domain. t This indicates the number of images in the target domain.

[0103] Step 8: Test the trained SAM-based domain adaptation segmentation network on the test set and calculate the average crossover ratio.

[0104] In the training process described above, stochastic gradient descent (SGD) was used to train the backbone network and the main and auxiliary classifiers; the Adam optimizer was used as the optimization method to train the main and auxiliary discriminator networks.

[0105] Example:

[0106] The infrared ship image dataset used in this invention contains four categories: three ship categories and one background category. Visible light images, serving as the source domain, comprise 2961 images and their true pixel-level labels, all used for training. Infrared images, serving as the target domain, comprise 614 images, with 429 images in the training set and 185 images in the test set. All images are 256×256 pixels in size and have 3 channels.

[0107] Based on steps 2 to 7 above, the trained segmentation network based on SAM enhancement domain adaptation is obtained; the segmentation network based on SAM enhancement domain adaptation is tested with the traditional segmentation model based on adversarial domain adaptation, and the test results are shown in Table 1.

[0108] Table 1

[0109]

[0110] Therefore, it can be seen that the method based on the SAM-enhanced domain adaptive model of the present invention can effectively improve the average intersection-union ratio of semantic segmentation of infrared ship images.

[0111] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for infrared ship semantic segmentation based on SAM enhanced domain adaptation, characterized in that, Comprise: Step 1, obtain a visible light ship image dataset X s and corresponding labels Y S , obtain an infrared ship image dataset X t , divide X t into a training set X train and a test set X test ; wherein the visible light ship image dataset is the source domain and the infrared ship image dataset is the target domain; Step 2, construct the adversarial domain adaptation segmentation model based on Deeplab, the adversarial domain adaptation segmentation model is composed of a segmentation network and a discriminator network; wherein the segmentation network comprises a backbone network, a main classifier and an auxiliary classifier, the backbone network is composed of the first 5 convolutional layers of ResNet-101, the main classifier is connected after the fifth convolutional layer, and the auxiliary classifier is connected after the fourth convolutional layer; the discriminator network comprises a main discriminator network and an auxiliary discriminator network, the main discriminator network is connected after the main classifier, and the auxiliary discriminator network is connected after the auxiliary classifier; Step 3, visible light ship image dataset X s and infrared ship image dataset X t is input into the backbone network to obtain a feature map, the feature map is respectively sent into a main classifier and an auxiliary classifier to obtain a main semantic segmentation prediction map and an auxiliary semantic segmentation prediction map, and the main semantic segmentation prediction map and the auxiliary semantic segmentation prediction map are both sent into a main discrimination network and an auxiliary discrimination network to obtain a judgment result of the segmentation prediction map. Step 4, calculate the source domain segmentation loss based on the main and auxiliary semantic segmentation prediction maps, and calculate the discrimination loss based on the judgment result of the prediction map; Step 5, extract the required Prompt information from the segmentation prediction map of the target domain image; wherein the required Prompt information includes: using a sliding window to correct the mask and using point clustering on the Canny edge to extract positive prompts, and using a geometric offset to extract negative prompts; Step 6, input the obtained positive and negative prompt information and infrared ship image into SAM to obtain the mask map of the infrared ship; the mask map without semantic information output by SAM is given a semantic value, so as to obtain the pseudo label of the target domain with semantic categories after SAM enhancement; Step 7, continue to train the segmentation network using the pseudo label enhanced by SAM; Step 8, test the trained segmentation network based on SAM enhanced domain adaptation on the test set, and calculate the average intersection over union.

2. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on a convolutional neural network (CNN) model. In step 1, X t The training set X is divided into two parts at an 8:2 ratio. train and test set X test X t Contains N t Samples, training set X train Contains N train Samples, test set X test Contains N test One sample.

3. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on a convolutional neural network (CNN) with a semantic attention module (SAM) and a multi-scale feature fusion module (MFFM). In the step 2, The main classifier and the auxiliary classifier have the same structure and are both composed of a dilated spatial pyramid pooling layer and an up-sampling layer; the dilated spatial pyramid pooling layer is used to capture multi-scale image context information, and the pooling sampling rate is 6, 12, 18 and 24 respectively; the up-sampling layer is used to restore the feature map after pooling to the size of the original image to generate the final semantic segmentation prediction map; The main discriminator network and the auxiliary discriminator network are both five-layer convolutional networks, the convolution kernel size is 4x4, the step is 2, and the channel number is 64, 128, 256, 512 and 1 respectively; The output result of the discriminator network represents the domain discrimination information, and the result close to 1 indicates that the input prediction map is discriminated as the source domain, and the result close to 0 indicates that the input prediction map is discriminated as the target domain.

4. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on a convolutional neural network (CNN) with a semantic attention module (SAM) and a multi-scale feature fusion module (MFFM). In the step 4, The source domain segmentation loss is used to ensure the performance of the segmentation network in the source domain, and the loss functions of the main and auxiliary classifiers of the segmentation network are represented as: wherein, and respectively represent the loss of the main classifier and the auxiliary classifier of the source domain, represents the i-th sample of the source domain; represents the real label corresponding to the i-th sample of the source domain; N represents the number of training samples; C is the total number of categories; h, w are the size of the image; F T (·) is the feature extractor of the teacher network, and respectively are the main and auxiliary classifiers of the teacher network; The final segmentation loss is represented as: In the formula, and Let α represent the losses of the main classifier and the auxiliary classifier in the source domain, respectively. c With (1-α) c ) represent the coefficients of the main classifier and the auxiliary classifier, respectively.

5. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on a convolutional neural network (CNN) with a semantic attention module (SAM) and a multi-scale feature fusion module (MFFM). In the step 4, In the training process, the main and auxiliary discriminator networks are used to discriminate the outputs of the main and auxiliary classifiers of the segmentation network respectively; specifically: input the main prediction map of the source domain and the target domain into the main discriminator network, and input the auxiliary prediction map of the source domain and the target domain into the auxiliary discriminator network to obtain the respective results; The domain discrimination loss functions of the two discriminator networks are represented as: In the formula, respectively represent the domain discriminative loss of the main and auxiliary discriminative networks; respectively represent the domain discriminative loss of the main and auxiliary discriminative networks; represents the i-th sample of the source domain; represents the l-th sample of the target domain; F T (·) is a feature extractor of the teacher network; and respectively represent the main and auxiliary classifiers of the teacher network; D p (·) and D a (·) respectively represent the main and auxiliary discriminative networks; Finally, the overall discrimination loss is represented as: In the formula, and Let α represent the inter-domain discrimination loss of the master and slave discriminant networks, respectively. D With (1-α) D ) represent the weight parameters of the main discriminant network and the auxiliary discriminant network, respectively.

6. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on a convolutional neural network (CNN) with a semantic attention module (SAM) and a multi-scale feature fusion module (MFFM). In the step 5, The positive prompts are extracted using a sliding window to correct the mask, which includes: A sliding window of size m x n is used to traverse the image, and for the pixels within the window, the threshold decision function counts the number of pixels of the class; if the number exceeds the threshold threshold, the pixel at the center of the window is marked as 255; Predicted map of target domain image of size H x W, denotes the pixel value of the predicted map at (x, y), the size of the sliding window is h x w, where h = w = bs w; for each sliding window, centered at (x, y), count the number of non-zero pixels within the window: wherein, is an indicator function, when y ′ t (x+i, y+j)≠0, Otherwise, 0; if the number of non-zero pixels exceeds Mark the center pixel (x, y) in the window as 255; get the thinned mask map y ′ t After (x, y), count the coordinates of non-zero pixels in the mask map, select 5 points as Prompt point coordinates at equal intervals, and save them in array P1; at this time, calculate the average of the 5 points to obtain the center point of the object to be segmented, denoted as C1(x1, y1); The positive cues are extracted using point clustering on Canny edges, including: First, the Canny operator is used to extract edge information, and then the K-means clustering method is used to extract positive cue points; specifically: first, using the Canny edge detection algorithm to extract the edge information of the infrared ship in the region centered at C1(x1, y1) with a radius of R, denoted as: edges(x, y) = Canny(I(x, y)) (8) where (x-x1) 2 +(y-y1) 2 ≤R 2 ; I represents an infrared image; Then, the K-means clustering is used to cluster the points obtained from the edge information; finally, the centroids of different clusters are output and saved in set P2; The negative cues are extracted using geometric offsets, including: By selecting negative cue points around the foreground cue points, the information of the foreground cue points is effectively supplemented; taking the mean C2 of sets P1 and P2 as the center, selecting one point each above and below the center point in the vertical direction at a distance of 90 pixels from the center point, and four points on the diagonal at a distance of 100 pixels from the center point, as background negative cue points.

7. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on a convolutional neural network (CNN) with a semantic attention module (SAM) and a multi-scale feature fusion module (MFFM). In step 6, the pseudo-labels of the target domain with semantic classes are represented as: In the formula, P t (x, y) represents the predicted class index of the pixel point in the target domain image in the first stage prediction map; M t (x, y) represents the mask without semantic information obtained through the SAM; is the pseudo label of the target domain with semantic categories.

8. The method of claim 1, wherein the SAM enhanced domain adaptive infrared ship semantic segmentation is based on, In step 7, The losses of the primary and auxiliary classifiers of the segmentation network being trained can be represented as: wherein, Fj represents the prediction map of the jth target domain image obtained by the student network, S (·) is a feature extractor of the student network, and and represent the main and auxiliary classifiers of the student network, respectively, Fj represents the pseudo label of the jth image of the target domain, N t N represents the number of target domain images.

9. An infrared ship semantic segmentation system based on SAM enhanced domain adaptation, for implementing the infrared ship semantic segmentation method based on SAM enhanced domain adaptation according to any one of claims 1-8.

Images