Adaptive infrared ship semantic segmentation method and system based on SAM enhanced domain

Abstract

The invention discloses an infrared ship semantic segmentation method and system based on SAM enhanced domain self-adaption, the first five layers of ResNet-101 are used as a backbone network to extract the features of an image, the segmentation loss of a source domain image is calculated, and at the same time, a main prediction map output by a main classifier and an auxiliary prediction map output by an auxiliary classifier are subjected to image segmentation. Respectively carrying out adversarial learning on the main classifier and the main discrimination network as well as the auxiliary classifier and the auxiliary discrimination network, and extracting common knowledge of the source domain and the target domain to obtain a pseudo label of the target domain; secondly, performing Prompt information extraction on the pseudo tag, and then sending the Prompt information and a target domain image into an SAM model to obtain an enhanced pseudo tag; and finally, network fine tuning is carried out on the enhanced pseudo tag and the target domain image, so that the segmentation accuracy of the infrared image is improved. According to the method based on the SAM enhanced domain adaptive model, the average intersection-union ratio of infrared ship image semantic segmentation can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision semantic segmentation, and in particular to a method and system for infrared ship semantic segmentation based on SAM enhanced domain adaptation. Background Art

[0002] Ships play a role in military defense and strategic deterrence. With the rapid development of modern science and technology, infrared imaging technology has become an important visual expansion tool. It provides us with a reliable means of observation by capturing the thermal radiation emitted by objects. However, due to the low contrast and high noise characteristics of infrared images, semantic segmentation of infrared ships becomes a challenging task. At the same time, visible light ship images obtained under good weather conditions have prominent targets and clear edges, making it easier to obtain pixel-level annotations. Therefore, using domain adaptation technology and training a segmentation network with visible light ships to achieve good segmentation accuracy on infrared ship images has become an idea to improve the accuracy of infrared ship segmentation. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides an infrared ship semantic segmentation method and system based on SAM enhanced domain adaptation.

[0004] The present invention discloses an infrared ship semantic segmentation method based on SAM enhanced domain adaptation, comprising:

[0005] Step 1: Obtain the visible light ship image dataset X s And the corresponding label Y S , obtain infrared ship image dataset X t , X t Divide into training set X train and the test set X test ; Among them, the visible light ship image dataset is used as the source domain, and the infrared ship image dataset is used as the target domain;

[0006] Step 2: Build an adversarial domain adaptation segmentation model based on Deeplab. The adversarial domain adaptation segmentation model consists of a segmentation network and a discriminant network. The segmentation network includes a backbone network, a main classifier, and an auxiliary classifier. The backbone network consists of the first five convolutional layers of ResNet-101. The main classifier is connected after the fifth convolutional layer, and the auxiliary classifier is connected after the fourth convolutional layer. The discriminant network includes a main discriminant network and an auxiliary discriminant network. The main discriminant network is connected after the main classifier, and the auxiliary discriminant network is connected after the auxiliary classifier.

[0007] Step 3: Convert the visible light ship image dataset X s and infrared ship image dataset X tInput into the backbone network to obtain feature maps, and send the feature maps to the main classifier and auxiliary classifier respectively to obtain the main and auxiliary semantic segmentation prediction maps; the main and auxiliary semantic segmentation prediction maps are both sent to the main discriminant network and auxiliary discriminant network to obtain the judgment results of the segmentation prediction map;

[0008] Step 4: Calculate the source domain segmentation loss based on the main and auxiliary semantic segmentation prediction images, and calculate the discrimination loss based on the judgment results of the prediction images;

[0009] Step 5, extracting the required prompt information from the segmentation prediction map of the target domain image; wherein extracting the required prompt information includes: extracting positive prompts using a sliding window correction mask and point clustering on a Canny edge, and extracting negative prompts using a geometric offset;

[0010] Step 6: Send the obtained positive and negative prompt information and the infrared ship image into SAM to obtain the mask map of the infrared ship; perform semantic assignment on the mask map without semantic information output by SAM, so as to obtain the pseudo label of the target domain with semantic category enhanced by SAM;

[0011] Step 7: Continue training the segmentation network using the pseudo-labels enhanced by SAM.

[0012] Step 8: Test the trained SAM-enhanced domain adaptation-based segmentation network on the test set and calculate the average intersection-over-union ratio.

[0013] As a further improvement of the present invention, in step 1, X t Divide into training set X in a ratio of 8:2 train and the test set X test , X t Contains N t samples, training set X train Contains N train samples, test set X test Contains N test samples.

[0014] As a further improvement of the present invention, in step 2,

[0015] The main classifier and the auxiliary classifier have the same structure, both consisting of a dilated spatial pyramid pooling layer and an upsampling layer; the dilated spatial pyramid pooling layer is used to capture multi-scale image context information, and the pooling sampling rates are 6, 12, 18, and 24 respectively; the upsampling layer is used to restore the pooled feature map to the size of the original image and generate the final semantic segmentation prediction map;

[0016] Both the main discriminant network and the auxiliary discriminant network are five-layer convolutional networks, with a convolution kernel size of 4×4, a step size of 2, and channel numbers of 64, 128, 256, 512, and 1 respectively; the discriminant network output results represent domain discrimination information, and a result close to 1 indicates that the input prediction map is judged as the source domain, and a result close to 0 indicates that the input prediction map is judged as the target domain.

[0017] As a further improvement of the present invention, in step 4,

[0018] The source domain segmentation loss is used to ensure the performance of the segmentation network on the source domain. The loss functions of the main and auxiliary classifiers of the segmentation network are expressed as:

[0019]

[0020] In the formula, and denote the loss of the main classifier and auxiliary classifier of the source domain, respectively. represents the i-th sample in the source domain; represents the true label corresponding to the i-th sample in the source domain; N represents the number of training samples; C is the total number of categories; h, w are the sizes of the image; and They are the main and auxiliary classifiers respectively;

[0021] The final segmentation loss is expressed as:

[0022]

[0023] In the formula, and Respectively represent the segmentation loss of the main classifier and the segmentation loss of the auxiliary classifier, α c and (1-α c ) represent the coefficients of the main classifier and the auxiliary classifier respectively.

[0024] As a further improvement of the present invention, in step 4,

[0025] During the training process, two discriminant networks, the main and auxiliary, are used to discriminate the outputs of the main and auxiliary classifiers of the segmentation network respectively; specifically, the main prediction graphs of the source domain and the target domain are input into the main discriminant network, and the auxiliary prediction graphs of the source domain and the target domain are input into the auxiliary discriminant network to obtain their respective results;

[0026] The domain discrimination loss functions of the two discriminant networks are expressed as:

[0027]

[0028] In the formula, and Respectively represent the inter-domain discrimination loss of the main and auxiliary discriminant networks; represents the i-th sample in the source domain; represents the lth sample in the target domain; F T (·) is the feature extractor of the teacher network; and Denote the main and auxiliary classifiers of the teacher network respectively; D p (·) and D a (·) represent the main and auxiliary discriminant networks respectively;

[0029] Finally, the overall discriminative loss is expressed as:

[0030]

[0031] In the formula, and Respectively represent the inter-domain discrimination loss generated by the main and auxiliary discriminant networks, α D and (1-α D ) represent the weight parameters of the main discriminant network and the auxiliary discriminant network respectively.

[0032] As a further improvement of the present invention, in step 5,

[0033] Extract positive cues using sliding window correction masks, including:

[0034] From the obtained prediction map, some information such as object shape and semantic category can be obtained; however, due to the noise in the target domain infrared image, there are also some discrete noise points in the prediction map output by the model, so the discrete noise points are filtered out by counting the number of pixels with semantic information in the window. Specifically, a sliding window of size m×n is used to traverse the image. For the pixels in the window, the threshold discriminant function will count the number of pixels with categories (i.e., non-zero pixels); if the number exceeds the threshold, it means that there are enough "interesting" pixels in the window, and the pixel at the center of the window is marked as 255;

[0035] Prediction map (pseudo label) of target domain image The size is H×W, Represents the pixel value of the predicted image at (x, y), the size of the sliding window is h×w, where h=w=bs_w; for each sliding window, with (x, y) as the center, count the number of non-zero pixels in the window:

[0036]

[0037] In the formula, is the indicator function, when y ′ t When (x+i,y+j)≠0, Otherwise 0; if the number of non-zero pixels exceeds Mark the central pixel (x, y) in the window as 255; get the refined mask image y ′ t After (z, y), the coordinates of non-zero pixels in the mask image are counted, and 5 points are selected at equal intervals as the coordinates of the prompt points and saved in the array P1; at this time, the average value of these 5 points is calculated to obtain the center point of the object to be segmented, which is recorded as C1(x1, y1);

[0038] Use point clustering on Canny edges to extract positive cues, including:

[0039] The edges of infrared images are relatively fuzzy, and a single interest point pixel may not be able to fully express the information of infrared ships. However, adding edge information can enable the model to better capture the shape and outline of the target and improve the segmentation accuracy.

[0040] First, the Canny operator is used to extract edge information, and then the K-means clustering method is used to extract positive prompt points. Specifically, first, in the area with C1(x1,y1) as the center and R as the radius, the Canny edge detection algorithm is used to extract the edge information of the infrared ship, which is expressed as:

[0041] edges(x,y)=Canny(I(x,y)) (8)

[0042] In the formula, (x-x1) 2 +(y-y1) 2 ≤R 2 ; I represents infrared image;

[0043] Then, use K-means clustering to cluster the points obtained from these edge information; finally, output the centroid points of different clusters and save them in set P2;

[0044] Extract negative hints using geometry offsets, including:

[0045] The above two steps can obtain foreground prompt points, however, due to noise, occlusion and low contrast of infrared images, the foreground prompt points may not be accurate enough. In order to provide SAM with more comprehensive prompt information, the present invention selects negative prompt points around the foreground prompt points to effectively supplement the information of the foreground prompt points; specifically, with C2 (the mean of all the positive prompt points mentioned above, that is, the mean of the sets P1 and P2) as the center, select one point each 90 pixels above and below the center point in the vertical direction, and the points 100 pixels away from the center point on the four diagonals as background negative prompt points.

[0046] As a further improvement of the present invention, in step 6, the pseudo label of the target domain with the semantic category is expressed as:

[0047]

[0048] Where P t (x, y) represents the predicted class index of the pixel in the target domain image in the first stage prediction map; M t (x, y) represents the mask without semantic information initially obtained by SAM; is the pseudo label of the target domain with semantic categories.

[0049] As a further improvement of the present invention, in step 7,

[0050] The losses of the main and auxiliary classifiers of the segmentation network that continue to be trained can be expressed as:

[0051]

[0052] in, represents the prediction image of the jth target domain image after the student network, F S (·) is the feature extractor of the student network, and Represent the main and auxiliary classifiers of the student network, represents the pseudo label of the jth image in the target domain, N t Represents the number of target domain images.

[0053] The present invention also discloses an infrared ship semantic segmentation system based on SAM enhanced domain adaptation, which is used to implement the above-mentioned infrared ship semantic segmentation method based on SAM enhanced domain adaptation.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention designs Prompt information and uses the SAM model to obtain enhanced pseudo-labels. Finally, it is trained on the target domain infrared image to obtain an adaptation model from the visible light domain to the infrared domain, thereby improving the network model's ability to classify and locate ships in infrared images and overcoming the problem of low average intersection-union ratio in infrared ship semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the infrared ship semantic segmentation method based on SAM enhanced domain adaptation disclosed by the present invention;

[0057] Figure 2 The structural diagram of adversarial learning for the backbone network;

[0058] Figure 3 Enhanced domain adaptation structure diagram for SAM;

[0059] Figure 4 for Figure 3 Visualization process of extracting prompt information. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0062] The present invention provides a semantic segmentation method and system for infrared ship based on SAM enhanced domain adaptation, which uses a semantic segmentation model based on SAM enhanced domain adaptation to perform semantic segmentation on infrared ship images, wherein the SAM model is a latest and universal image segmentation model, which aims to achieve segmentation of any object in any type of image; the main idea is: through self-supervision, using the powerful segmentation ability of SAM to obtain enhanced pseudo-labels, and finally training the segmentation model on the enhanced pseudo-labels, thereby improving the segmentation accuracy of the model on the target domain infrared image.

[0063] like Figures 1 to 4 As shown, the present invention provides an infrared ship semantic segmentation method based on SAM enhanced domain adaptation, comprising:

[0064] Step 1: Obtain the visible light ship image dataset X s And the corresponding label Y S , obtain infrared ship image dataset X t , X t Divide into training set X train and the test set X test ; Among them, the visible light ship image dataset is used as the source domain, and the infrared ship image dataset is used as the target domain; X t Divide into training set X in a ratio of 8:2 train and the test set X test , X t Contains N t samples, training set X train Contains N train samples, test set X test Contains N testsamples; first use the training set to complete the segmentation network based on SAM enhanced domain adaptation, and then test it based on the test set;

[0065] Step 2: Build an adversarial domain adaptation segmentation model based on Deeplab. The adversarial domain adaptation segmentation model consists of a segmentation network and a discriminant network. Figure 2 As shown in the figure, the segmentation network includes a backbone network, a main classifier and an auxiliary classifier. The backbone network is composed of the first five convolutional layers of ResNet-101. The main classifier is connected after the fifth convolutional layer, and the auxiliary classifier is connected after the fourth convolutional layer. The discriminant network includes a main discriminant network and an auxiliary discriminant network. The main discriminant network is connected after the main classifier, and the auxiliary discriminant network is connected after the auxiliary classifier.

[0066] The main classifier and the auxiliary classifier have the same structure, both consisting of a dilated spatial pyramid pooling layer and an upsampling layer; the dilated spatial pyramid pooling layer is used to capture multi-scale image context information, and the pooling sampling rates are 6, 12, 18, and 24 respectively; the upsampling layer is used to restore the pooled feature map to the size of the original image and generate the final semantic segmentation prediction map;

[0067] Both the main discriminant network and the auxiliary discriminant network are five-layer convolutional networks, with a convolution kernel size of 4×4, a step size of 2, and channel numbers of 64, 128, 256, 512, and 1 respectively; the discriminant network output results represent domain discrimination information, and a result close to 1 indicates that the input prediction map is judged as the source domain, and a result close to 0 indicates that the input prediction map is judged as the target domain.

[0068] Step 3: Convert the visible light ship image dataset X s and infrared ship image dataset X t Input into the backbone network to obtain a feature map, and send the feature map to the main classifier and the auxiliary classifier respectively to obtain the main and auxiliary semantic segmentation prediction maps; the main and auxiliary semantic segmentation prediction maps are both sent to the main discriminant network and the auxiliary discriminant network to obtain the judgment result of the segmentation prediction map; that is, adversarial learning is performed through the main classifier and the main discriminant network, the auxiliary classifier and the auxiliary discriminant network respectively, to extract the knowledge shared by the source domain and the target domain, and obtain the pseudo label of the target domain;

[0069] Step 4: Calculate the source domain segmentation loss based on the main and auxiliary semantic segmentation prediction images, and calculate the discrimination loss based on the judgment results of the prediction images;

[0070] The source domain segmentation loss is used to ensure the performance of the segmentation network on the source domain. The loss functions of the main and auxiliary classifiers of the segmentation network are expressed as:

[0071]

[0072] In the formula, and denote the loss of the main classifier and auxiliary classifier of the source domain, respectively. represents the i-th sample in the source domain; represents the true label corresponding to the i-th sample in the source domain; N represents the number of training samples; C is the total number of categories; h, w are the sizes of the image; and They are the main and auxiliary classifiers respectively;

[0073] The final segmentation loss is expressed as:

[0074]

[0075] In the formula, and Respectively represent the segmentation loss of the main classifier and the segmentation loss of the auxiliary classifier, α c and (1-α c ) represent the coefficients of the main classifier and the auxiliary classifier respectively.

[0076] During the training process, two discriminant networks, the main and auxiliary, are used to discriminate the outputs of the main and auxiliary classifiers of the segmentation network respectively; specifically, the main prediction graphs of the source domain and the target domain are input into the main discriminant network, and the auxiliary prediction graphs of the source domain and the target domain are input into the auxiliary discriminant network to obtain their respective results;

[0077] The domain discrimination loss functions of the two discriminant networks are expressed as:

[0078]

[0079] In the formula, and Respectively represent the inter-domain discrimination loss of the main and auxiliary discriminant networks; represents the i-th sample in the source domain; represents the lth sample in the target domain; F T (·) is the feature extractor of the teacher network; and Denote the main and auxiliary classifiers of the teacher network respectively; D p (·) and D a (·) represent the main and auxiliary discriminant networks respectively;

[0080] Finally, the overall discriminative loss is expressed as:

[0081]

[0082] In the formula, and Respectively represent the inter-domain discrimination loss generated by the main and auxiliary discriminant networks, α D and (1-α D) represent the weight parameters of the main discriminant network and the auxiliary discriminant network respectively.

[0083] Step 5: extract the required prompt information from the segmentation prediction map of the target domain image; wherein, extracting the required prompt information includes: extracting positive prompts using a sliding window correction mask and point clustering on the Canny edge, and extracting negative prompts using a geometric offset; specifically:

[0084] Extract positive cues using sliding window correction masks, including:

[0085] From the obtained prediction map, some information such as object shape and semantic category can be obtained; however, due to the noise in the target domain infrared image, there are also some discrete noise points in the prediction map output by the model, so the discrete noise points are filtered out by counting the number of pixels with semantic information in the window. Specifically, a sliding window of size m×n is used to traverse the image. For the pixels in the window, the threshold discriminant function will count the number of pixels with categories (i.e., non-zero pixels); if the number exceeds the threshold, it means that there are enough "interesting" pixels in the window, and the pixel at the center of the window is marked as 255;

[0086] Prediction map (pseudo label) of target domain image The size is H×W, Represents the pixel value of the predicted image at (x, y), the size of the sliding window is h×w, where h=w=bs_w; for each sliding window, with (x, y) as the center, count the number of non-zero pixels in the window:

[0087]

[0088] In the formula, is the indicator function, when y ′ t When (x+i,y+j)≠0, Otherwise 0; if the number of non-zero pixels exceeds Mark the central pixel (x, y) in the window as 255; get the refined mask image y ′ t After (z, y), the coordinates of non-zero pixels in the mask image are counted, and 5 points are selected at equal intervals as the coordinates of the prompt points and saved in the array P1; at this time, the average value of these 5 points is calculated to obtain the center point of the object to be segmented, which is recorded as C1(x1, y1);

[0089] Use point clustering on Canny edges to extract positive cues, including:

[0090] The edges of infrared images are relatively fuzzy, and a single interest point pixel may not be able to fully express the information of infrared ships. However, adding edge information can enable the model to better capture the shape and outline of the target and improve the segmentation accuracy.

[0091] First, the Canny operator is used to extract edge information, and then the K-means clustering method is used to extract positive prompt points. Specifically, first, in the area with C1(x1,y1) as the center and R as the radius, the Canny edge detection algorithm is used to extract the edge information of the infrared ship, which is expressed as:

[0092] edges(x,y)=Canny(I(x,y)) (8)

[0093] In the formula, (x-x1) 2 +(y-y1) 2 ≤R 2 ; I represents infrared image;

[0094] Then, use K-means clustering to cluster the points obtained from these edge information; finally, output the centroid points of different clusters and save them in set P2;

[0095] Extract negative hints using geometry offsets, including:

[0096] The above two steps can obtain foreground prompt points, however, due to noise, occlusion and low contrast of infrared images, the foreground prompt points may not be accurate enough. In order to provide SAM with more comprehensive prompt information, the present invention selects negative prompt points around the foreground prompt points to effectively supplement the information of the foreground prompt points; specifically, with C2 (the mean of all the positive prompt points mentioned above, that is, the mean of the sets P1 and P2) as the center, select one point each 90 pixels above and below the center point in the vertical direction, and the points 100 pixels away from the center point on the four diagonals as background negative prompt points.

[0097] Step 6: Send the obtained positive and negative prompt information and the infrared ship image into SAM to obtain the mask map of the infrared ship; perform semantic assignment on the mask map without semantic information output by SAM, so as to obtain the pseudo label of the target domain with semantic category enhanced by SAM; wherein the pseudo label of the target domain with semantic category is expressed as:

[0098]

[0099] Where P t (x, y) represents the predicted class index of the pixel in the target domain image in the first stage prediction map; M t(x, y) represents the mask without semantic information initially obtained by SAM; is the pseudo label of the target domain with semantic categories.

[0100] Step 7: Use the pseudo-labels enhanced by SAM to continue training the segmentation network, that is, fine-tune the network on the enhanced pseudo-labels and target domain images to improve the segmentation accuracy of infrared images; the losses of the main and auxiliary classifiers of the segmentation network that continues to be trained can be expressed as:

[0101]

[0102] in, represents the prediction image of the jth target domain image after the student network, F S (·) is the feature extractor of the student network, and Represent the main and auxiliary classifiers of the student network, represents the pseudo label of the jth image in the target domain, N t Represents the number of target domain images.

[0103] Step 8: Test the trained SAM-enhanced domain adaptation-based segmentation network on the test set and calculate the average intersection-over-union ratio.

[0104] In the above training process, the stochastic gradient descent method (SGD) is used to train the backbone network and the main and auxiliary classifiers; the Adam optimizer is used as the optimization method to train the main and auxiliary discriminant networks.

[0105] Example:

[0106] The infrared ship image dataset used in this paper contains 4 categories, including 3 ship categories and 1 background category. Visible light images are used as the source domain, including 2961 images and their true pixel-level labels, all of which are used for training. Infrared images are used as the target domain, including 614 images, of which the training set contains 429 images and the test set contains 185 images. The size of all images is 256×256, and the number of channels is 3.

[0107] Based on the above steps 2 to 7, the trained segmentation network based on SAM enhanced domain adaptation is obtained; the segmentation network based on SAM enhanced domain adaptation is tested with the traditional segmentation model based on adversarial domain adaptation, and the test results are shown in Table 1.

[0108] Table 1

[0109]

[0110] It can be seen from this that the method based on the SAM enhanced domain adaptive model of the present invention can effectively improve the average intersection-over-union ratio of semantic segmentation of infrared ship images.

[0111] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for infrared ship semantic segmentation based on SAM enhanced domain adaptation, characterized in that: include: Step 1: Obtain the visible light ship image dataset X s And the corresponding label Y S , obtain infrared ship image dataset X t , X t Divide into training set X train and the test set X test ; Among them, the visible light ship image dataset is used as the source domain, and the infrared ship image dataset is used as the target domain; Step 2: Build an adversarial domain adaptation segmentation model based on Deeplab. The adversarial domain adaptation segmentation model consists of a segmentation network and a discriminant network. The segmentation network includes a backbone network, a main classifier, and an auxiliary classifier. The backbone network consists of the first five convolutional layers of ResNet-101. The main classifier is connected after the fifth convolutional layer, and the auxiliary classifier is connected after the fourth convolutional layer. The discriminant network includes a main discriminant network and an auxiliary discriminant network. The main discriminant network is connected after the main classifier, and the auxiliary discriminant network is connected after the auxiliary classifier. Step 3: Convert the visible light ship image dataset X s and infrared ship image dataset X t Input into the backbone network to obtain feature maps, and send the feature maps to the main classifier and auxiliary classifier respectively to obtain the main and auxiliary semantic segmentation prediction maps; both the main and auxiliary semantic segmentation prediction maps are sent to the main discriminant network and the auxiliary discriminant network to obtain the judgment results of the segmentation prediction map; Step 4: Calculate the source domain segmentation loss based on the main and auxiliary semantic segmentation prediction images, and calculate the discrimination loss based on the judgment results of the prediction images; Step 5, extracting the required prompt information from the segmentation prediction map of the target domain image; wherein extracting the required prompt information includes: extracting positive prompts using a sliding window correction mask and point clustering on a Canny edge, and extracting negative prompts using a geometric offset; Step 6: Send the obtained positive and negative prompt information and the infrared ship image into SAM to obtain the mask map of the infrared ship; perform semantic assignment on the mask map without semantic information output by SAM, so as to obtain the pseudo label of the target domain with semantic category enhanced by SAM; Step 7: Continue training the segmentation network using the pseudo-labels enhanced by SAM. Step 8: Test the trained SAM-enhanced domain adaptation-based segmentation network on the test set and calculate the average intersection-over-union ratio.

2. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 1, X t Divide into training set X in a ratio of 8:2 train and the test set X test , X t Contains N t samples, training set X train Contains N train samples, test set X test Contains N test samples.

3. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 2, The main classifier and the auxiliary classifier have the same structure, both consisting of a dilated spatial pyramid pooling layer and an upsampling layer; the dilated spatial pyramid pooling layer is used to capture multi-scale image context information, and the pooling sampling rates are 6, 12, 18, and 24 respectively; the upsampling layer is used to restore the pooled feature map to the size of the original image and generate the final semantic segmentation prediction map; Both the main discriminant network and the auxiliary discriminant network are five-layer convolutional networks, with a convolution kernel size of 4×4, a step size of 2, and channel numbers of 64, 128, 256, 512, and 1 respectively; The output of the discriminant network represents the domain discrimination information. A result close to 1 indicates that the input prediction map is discriminated as the source domain, and a result close to 0 indicates that the input prediction map is discriminated as the target domain.

4. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 4, The source domain segmentation loss is used to ensure the performance of the segmentation network on the source domain. The loss functions of the main and auxiliary classifiers of the segmentation network are expressed as: In the formula, and denote the loss of the main classifier and auxiliary classifier of the source domain, respectively. represents the i-th sample in the source domain; represents the true label corresponding to the i-th sample in the source domain; N represents the number of training samples; C is the total number of categories; h, w are the sizes of the image; and They are the main and auxiliary classifiers respectively; The final segmentation loss is expressed as: In the formula, and Respectively represent the segmentation loss of the main classifier and the segmentation loss of the auxiliary classifier, α c and (1-α c ) represent the coefficients of the main classifier and the auxiliary classifier respectively.

5. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 4, During the training process, two discriminant networks, the main and auxiliary, are used to discriminate the outputs of the main and auxiliary classifiers of the segmentation network respectively; specifically, the main prediction graphs of the source domain and the target domain are input into the main discriminant network, and the auxiliary prediction graphs of the source domain and the target domain are input into the auxiliary discriminant network to obtain their respective results; The domain discrimination loss functions of the two discriminant networks are expressed as: In the formula, and Respectively represent the inter-domain discrimination loss of the main and auxiliary discriminant networks; represents the i-th sample in the source domain; represents the lth sample in the target domain; F T (·) is the feature extractor of the teacher network; and Denote the main and auxiliary classifiers of the teacher network respectively; D p (·) and D a (·) represent the main and auxiliary discriminant networks respectively; Finally, the overall discriminative loss is expressed as: In the formula, and Respectively represent the inter-domain discrimination loss generated by the main and auxiliary discriminant networks, α D and (1-α D ) represent the weight parameters of the main discriminant network and the auxiliary discriminant network respectively.

6. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 5, Extract positive cues using sliding window correction masks, including: Use a sliding window of size m×n to traverse the image. For the pixels in the window, the threshold discriminant function will count the number of pixels with categories; if the number exceeds the threshold, the pixel at the center of the window will be marked as 255; Prediction map of target domain image The size is H×W, Represents the pixel value of the predicted image at (x, y), the size of the sliding window is h×w, where h=w=bs_w; for each sliding window, with (x, y) as the center, count the number of non-zero pixels in the window: In the formula, is the indicator function, when y′ t When (x+i,y+j)≠0, Otherwise 0; if the number of non-zero pixels exceeds Mark the central pixel (x, y) in the window as 255; get the refined mask image y ′ t After (x, y), the coordinates of non-zero pixels in the mask image are counted, and 5 points are selected at equal intervals as the coordinates of the prompt points and saved in the array P1; at this time, the average value of these 5 points is calculated to obtain the center point of the object to be segmented, which is recorded as C1(x1, y1); Use point clustering on Canny edges to extract positive cues, including: First, the Canny operator is used to extract edge information, and then the K-means clustering method is used to extract positive prompt points. Specifically, first, in the area with C1(x1,y1) as the center and R as the radius, the Canny edge detection algorithm is used to extract the edge information of the infrared ship, which is expressed as: edges(x,y)=Canny(I(x,y)) (8) In the formula, (x-x1) 2 +(y-y1) 2 ≤R 2 ; I represents infrared image; Then, use K-means clustering to cluster the points obtained from these edge information; finally, output the centroid points of different clusters and save them in set P2; Extract negative hints using geometry offsets, including: By selecting Negative cue points around the foreground cue points, the information of the foreground cue points can be effectively supplemented; taking C2 (the mean of all the positive cue points mentioned above, that is, the mean of sets P1 and P2) as the center, select a point directly above and below the center point at 90 pixels in the vertical direction, and a point on the four diagonals at 100 pixels from the center point as the background Negative cue points.

7. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 6, the pseudo label of the target domain with semantic categories is expressed as: Where P t (x, y) represents the predicted class index of the pixel in the target domain image in the first stage prediction map; M t (x,y) represents the mask without semantic information initially obtained by SAM; is the pseudo label of the target domain with semantic categories.

8. The infrared ship semantic segmentation method based on SAM enhanced domain adaptation as claimed in claim 1, characterized in that: In step 7, The losses of the main and auxiliary classifiers of the segmentation network that continue to be trained can be expressed as: in, represents the prediction image of the jth target domain image after the student network, F S (·) is the feature extractor of the student network, and Represent the main and auxiliary classifiers of the student network, represents the pseudo label of the jth image in the target domain, N t Represents the number of target domain images.

9. An infrared ship semantic segmentation system based on SAM enhanced domain adaptation, used to implement the infrared ship semantic segmentation method based on SAM enhanced domain adaptation as described in any one of claims 1 to 8.

Images