Semantic segmentation method for infrared ship images based on pixel domain adaptation

Abstract

The present invention discloses a semantic segmentation method for infrared ship images based on pixel domain adaptation. First, a source domain visible light ship image and a target domain infrared ship image are input and passed to a pixel domain adaptation network to make the source domain and target domain ship images similar in appearance and style, generating a similarity loss. Second, the source domain and target domain images obtained through pixel domain adaptation are input to a segmentation network to predict the source domain and target domain to obtain segmentation outputs. The source predicted image obtained from the source domain output generates a supervised source domain segmentation loss. The two segmentation outputs are then used as inputs to a discriminator to generate an adversarial loss, which is then passed to the pixel domain adaptation network and the segmentation network. Finally, the network is trained alternately by minimizing the similarity loss and the segmentation loss and maximizing the adversarial loss. The present invention's semantic segmentation method for infrared ship images based on pixel domain adaptation effectively improves the segmentation performance of unsupervised domain adaptation infrared ship images.

Description

Technical Field

[0001] The present invention belongs to the field of unsupervised domain adaptation semantic segmentation, and in particular relates to an infrared ship image semantic segmentation method based on pixel domain adaptation. Background Art

[0002] Interpreting infrared ship images at sea is crucial for ensuring maritime safety. Semantic segmentation, which assigns a category label to each pixel in an image, is a key technology for image parsing. Current deep learning-based semantic segmentation algorithms require a large amount of precisely labeled data to train an efficient model. Visible light images have the characteristics of pre-labeled images, concise and clear objects, and rich features. However, acquired infrared ship images inherently suffer from image blur and unclear object edges, making the labeling process cumbersome, time-consuming, and labor-intensive. Therefore, applying segmentation models trained using existing labeled visible light ship data to infrared ship data is a very effective solution. However, fully supervised semantic segmentation methods in one domain cannot guarantee good generalization to test data from other domains due to the significant domain shift between the two domains. Therefore, unsupervised domain adaptation methods are used to correct this domain shift and improve the network's segmentation performance.

[0003] Traditional unsupervised domain adaptation methods typically use measures such as correlation distance metrics to align the data distributions of the source and target domains output by their deep networks. These distributions are usually non-parametric and artificial mathematical formulas, not specifically tailored to the dataset. This approach does not effectively address our problem. Therefore, an adversarial domain adaptation method has emerged, which trains the segmentation network and the discriminator network to align the distributions of the two domains, and has achieved remarkable results.

[0004] Furthermore, adversarial unsupervised domain adaptation can be further categorized into pixel-level and feature-level domain adaptation methods. Pixel-level domain adaptation methods leverage the inherently different characteristics of the two domain images, using an image conversion network to minimize the appearance differences between the two domain images. A segmentation network is then used to segment the images from the different domains. For example, the CYCada network first draws on CycleGAN to make the styles of the two domains similar. Then, adversarial domain adaptation between the two domains can effectively mitigate the impact of significant style differences between the two domains. However, segmentation performance can even degrade after conversion for infrared images with coarse information. Feature-level domain adaptation methods first input the source and target domain images into the segmentation network separately and then align them in the feature space. This method can partially address the domain shift issue. For example, AdaptSeg performs domain alignment in the feature output space, but does not consider the fact that significant differences in image information between the two domains can lead to poor segmentation performance. Currently, most unsupervised domain adaptation methods are developed for cases where both domains are visible color images. Few are designed for cases where visible light images are the source domain and infrared images are the target domain.

[0005] For infrared ship image datasets, simple image conversion networks cannot meet segmentation performance requirements. This paper utilizes a novel image conversion network to perform appearance similarity conversion between source and target domain images, which is then fed into the subsequent segmentation network and discriminator network models. This effectively improves the performance of the segmentation network model and increases the average intersection-over-union ratio for infrared ship image segmentation. The paper is highly operational and versatile, and is applicable to various domain-adaptive segmentation models based on infrared image datasets. Summary of the Invention

[0006] The technical problem addressed by this invention is to provide a semantic segmentation method for infrared ship images based on pixel domain adaptation, mitigating the domain shift between the source and target domains. This invention utilizes a novel pixel domain adaptation network to reduce the inter-domain shift between visible and infrared images, making the trained network model more robust and thereby improving the network's semantic segmentation performance for infrared ship images. To achieve this objective, the present invention employs the following technical solutions:

[0007] A semantic segmentation method for infrared ship images based on pixel domain adaptation includes the following steps:

[0008] Step 1: Obtain the source domain visible light ship dataset X S (Including N S_train samples) and corresponding labels Y S (Including N S_train samples), obtain the target domain infrared ship dataset X T , divide the target domain dataset into training set X according to the ratio of 6:4 T_train (Including NT_train samples) and the test set X T_test (Including N T_test samples and corresponding labels);

[0009] Step 2: Build a pixel-domain adaptation model based on a generative adversarial network. The generator consists of two sub-networks: an appearance similarity network and a segmentation network. The appearance similarity network is used to convert the source domain visible light image and the target domain infrared image into images with the same appearance as much as possible. The segmentation network receives the converted source and target domain images and generates corresponding segmented images. The discriminator uses a basic convolutional network to discriminate the generated segmented images and determine their domain category, thereby achieving infrared ship segmentation.

[0010] Step 2.1: Construct an appearance similarity network model, which is a convolutional network. The first three layers are convolutional layers, with convolution kernel sizes of {7×7, 3×3, 3×3} and channel numbers of {64, 128, 256}. The first convolution layer has a stride of 1, and the remaining two convolution layers have strides of 2. Each convolution layer is followed by a block normalization layer and a rectified linear unit activation function; followed by two residual blocks, each residual block contains two convolution layers, each convolution layer has a 3×3 convolution kernel and 256 channels, with a stride of 1, and each convolution layer is followed by a A block normalization layer is used; followed by two deconvolution layers, both with a 3×3 kernel size, {128, 64} channels, and a stride of 2. Each deconvolution layer is followed by a block normalization layer and a rectified linear unit activation function. Finally, a convolution layer with a 7×7 kernel size, 3 channels, and a stride of 1 is used. After convolution, the Tanh activation function is used to generate a color feature map, which is then added to the original image to generate the new source and target domain ship images.

[0011] Step 2.2: Input the newly generated source domain and target domain images into the segmentation network respectively to obtain the corresponding segmentation results. The segmentation result of the source domain visible light image is recorded as P S , the segmentation result of the target domain infrared image is recorded as P T ; The segmentation network is a Deeplab-v2 segmentation network with Resnet-101 as the backbone network;

[0012] Step 2.3: Construct the discriminator model. The network is a convolutional network. The first three layers are convolutional layers. The convolution kernel size is 4×4, the number of channels is {64, 128, 256}, the stride of the first two convolutional layers is 2, the stride of the third convolutional layer is 1, and each convolutional layer is followed by an unsaturated rectified linear unit activation function with a parameter of 0.2; the fourth layer is a dilated convolutional layer, which contains three parallel dilated convolutions with dilated convolution rates of {1, 3, 5}, the convolution kernel size is 4×4, the number of channels is 256, the stride is 1, and each dilated convolution is followed by an unsaturated rectified linear unit activation function with a parameter of 0.2; the fifth layer is a convolutional layer with a convolution kernel size of 4×4 and the number of channels is 1. This convolutional layer serves as a classifier and returns the discrimination result;

[0013] Step 3: Design loss functions for the new generator network and discriminator network respectively, specifically:

[0014] Step 3.1: The loss function of the appearance similarity network includes: overall diversity loss L varia , whose goal is to remove the rough texture information in the image; the conversion consistency loss L cyc , whose goal is to reduce the impact of image color differences between the two domains; color control loss L color , its purpose is to ensure the consistency of image information before and after conversion;

[0015] The overall diversity loss function is:

[0016]

[0017] Where, l∈{1≤l≤N S_train} is the lth sample, are the two domain images of the input, Two domain images generated for the appearance similarity network, is the data after the source domain image is converted, is the data after the target domain image is converted, N l For X l The total number of pixel values ​​in and are the gradient values ​​of color intensity between adjacent pixels along the x and y directions, respectively, and ||·||1 is the L1 norm of the sum of all pixel values;

[0018] The conversion consistency loss function is:

[0019]

[0020] Where, l∈{1≤l≤N S_train} is the lth sample, are the two domain images of the input, Two domain images generated for the appearance similarity network, is the data after the source domain image is converted, is the data after the conversion of the target domain image, ||·||1 is the L1 norm of the sum of all pixel values;

[0021] The color controllability loss function is:

[0022]

[0023] Where, is a function of the average pooling kernel size 32*32, M l for The number of pixel values ​​in E l The value of is dynamically changed and is set to the average color intensity value of the infrared image in each iteration;

[0024] The overall loss function of the appearance similarity model can be defined as:

[0025] L ASN =α1L varia +α2L cyc +α3L color (4)

[0026] Where, parameters α1, α2, and α3 are the weight parameters of the three loss functions;

[0027] Step 3.2: For the source domain output graph in the segmentation network, a portion of the corresponding labels are subjected to cross entropy loss to ensure the segmentation performance of the source domain;

[0028] The cross entropy loss function formula is:

[0029]

[0030] Where, l∈{1≤l≤N S_train} is the lth sample, h, w, c, C are the height, width, specific category and total category of the image respectively, The segmentation output map generated by the source domain data input network; the other part is the segmentation output of the target domain One piece serves as the input to the discriminator, ensuring that the images of the two domains are as close as possible to confuse the discriminator. This part requires adversarial loss to complete;

[0031] The adversarial loss function formula is:

[0032]

[0033] Where, l∈{1≤l≤N S_train} is the lth sample, The segmentation output map generated by the target domain data input network, D is the subsequent discriminator network;

[0034] Therefore, the loss function formula of the entire generator network is:

[0035]

[0036] Where λ1, λ2, and λ3 are the weight parameters of the three loss functions;

[0037] Step 3.3: The discriminator obtains the segmentation output maps of the two domains and requires the discriminator to correctly determine which domain the data comes from as much as possible;

[0038] The discriminator loss function is:

[0039]

[0040] Where z=0 means the sample comes from the target domain, z=1 means the sample comes from the source domain, P is the segmentation output of the two domains, and D is the discriminator network;

[0041] Step 4: Alternately iterate and train the generator and discriminator. This process uses the "maximum-minimum adversarial" idea to continuously train the entire network and save the network's weight parameters.

[0042] Step 5: Load the weight parameters trained in step 4, input the infrared ship test dataset into the new generator network to generate the segmentation map, and T_test Calculate the average intersection-union ratio above. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the basic method flow of the present invention;

[0044] Figure 2 Design of the overall network architecture for this article;

[0045] Figure 3 This is the pixel domain adaptation network module of this paper;

[0046] Figure 4 This is the discriminator network module of this paper;

[0047] Figure 5 is the IOU result of the test set; DETAILED DESCRIPTION

[0048] The present invention provides a semantic segmentation method for infrared ship images based on pixel domain adaptation. The present invention is explained and illustrated below with reference to the accompanying drawings.

[0049] The ship dataset used in this invention contains three categories. The visible light ship image dataset contains 3091 samples and corresponding labels, while the infrared ship dataset contains 790 samples, and only the test set has labels. The implementation process of this invention is as follows:

[0050] A semantic segmentation method for infrared ship images based on pixel domain adaptation includes the following steps:

[0051] Step 1: Obtain the source domain visible light ship dataset X S (Contains 3091 samples) and corresponding labels Y S (Contains 3091 samples), obtain the target domain infrared ship dataset X T , divide the target domain dataset into training set X according to the ratio of 6:4 T_train (containing 474 samples) and the test set X T_test (Contains 316 samples and corresponding labels);

[0052] Step 2: Build a pixel-domain adaptation model based on a generative adversarial network. The generator consists of two sub-networks: an appearance similarity network and a segmentation network. The appearance similarity network is used to convert the source domain visible light image and the target domain infrared image into images with the same appearance as much as possible. The segmentation network receives the converted source and target domain images and generates corresponding segmented images. The discriminator uses a basic convolutional network to discriminate the generated segmented images and determine their domain category, thereby achieving infrared ship segmentation.

[0053] Step 2.1: Construct an appearance similarity network model, which is a convolutional network. The first three layers are convolutional layers, with convolution kernel sizes of {7×7, 3×3, 3×3} and channel numbers of {64, 128, 256}. The first convolution layer has a stride of 1, and the remaining two convolution layers have strides of 2. Each convolution layer is followed by a block normalization layer and a rectified linear unit activation function; followed by two residual blocks, each residual block contains two convolution layers, each convolution layer has a 3×3 convolution kernel and 256 channels, with a stride of 1, and each convolution layer is followed by a A block normalization layer is used; followed by two deconvolution layers, both with a 3×3 kernel size, {128, 64} channels, and a stride of 2. Each deconvolution layer is followed by a block normalization layer and a rectified linear unit activation function. Finally, a convolution layer with a 7×7 kernel size, 3 channels, and a stride of 1 is used. After convolution, the Tanh activation function is used to generate a color feature map, which is then added to the original image to generate the new source and target domain ship images.

[0054] Step 2.2: Input the newly generated source domain and target domain images into the segmentation network respectively to obtain the corresponding segmentation results. The segmentation result of the source domain visible light image is recorded as PS , the segmentation result of the target domain infrared image is recorded as P T ; The segmentation network is a Deeplab-v2 segmentation network with Resnet-101 as the backbone network;

[0055] Step 2.3: Construct the discriminator model. The network is a convolutional network. The first three layers are convolutional layers. The convolution kernel size is 4×4, the number of channels is {64, 128, 256}, the stride of the first two convolutional layers is 2, the stride of the third convolutional layer is 1, and each convolutional layer is followed by an unsaturated rectified linear unit activation function with a parameter of 0.2; the fourth layer is a dilated convolutional layer, which contains three parallel dilated convolutions with dilated convolution rates of {1, 3, 5}, the convolution kernel size is 4×4, the number of channels is 256, the stride is 1, and each dilated convolution is followed by an unsaturated rectified linear unit activation function with a parameter of 0.2; the fifth layer is a convolutional layer with a convolution kernel size of 4×4 and the number of channels is 1. This convolutional layer serves as a classifier and returns the discrimination result;

[0056] Step 3: Design loss functions for the new generator network and discriminator network respectively, specifically:

[0057] Step 3.1: The loss function of the appearance similarity network includes: overall diversity loss L varia , whose goal is to remove the rough texture information in the image; the conversion consistency loss L cyc , whose goal is to reduce the impact of image color differences between the two domains; color control loss L color , its purpose is to ensure the consistency of image information before and after conversion;

[0058] The overall diversity loss function is:

[0059]

[0060] Where, l∈{1≤l≤N S_train} is the lth sample, are the two domain images of the input, Two domain images generated for the appearance similarity network, is the data after the source domain image is converted, is the data after the target domain image is converted, N l For X l The total number of pixel values ​​in and are the gradient values ​​of color intensity between adjacent pixels along the x and y directions, respectively, and ||·||1 is the L1 norm of the sum of all pixel values;

[0061] The conversion consistency loss function is:

[0062]

[0063] Where, l∈{1≤l≤N S_train} is the lth sample, are the two domain images of the input, Two domain images generated for the appearance similarity network, is the data after the source domain image is converted, is the data after the conversion of the target domain image, ||·||1 is the L1 norm of the sum of all pixel values;

[0064] The color controllability loss function is:

[0065]

[0066] Where, is a function of the average pooling kernel size 32*32, M l for The number of pixel values ​​in E l The value of is dynamically changed and is set to the average color intensity value of the infrared image in each iteration;

[0067] The overall loss function of the appearance similarity model can be defined as:

[0068] L ASN =α1L varia +α2L cyc +α3L color (4)

[0069] Where, parameters α1, α2, and α3 are the weight parameters of the three loss functions, and their values ​​are set to 10, 0.1, and 0.01 respectively;

[0070] Step 3.2: For the source domain output graph in the segmentation network, a portion of the corresponding labels are subjected to cross entropy loss to ensure the segmentation performance of the source domain;

[0071] The cross entropy loss function formula is:

[0072]

[0073] Where, l∈{1≤l≤N S_train} is the lth sample, h, w, c, C are the height, width, specific category and total category of the image respectively, The segmentation output map generated by the source domain data input network; the other part is the segmentation output of the target domain One piece serves as the input to the discriminator, ensuring that the images of the two domains are as close as possible to confuse the discriminator. This part requires adversarial loss to complete;

[0074] The adversarial loss function formula is:

[0075]

[0076] Where, l∈{1≤l≤N S_train} is the lth sample, The segmentation output map generated by the target domain data input network, D is the subsequent discriminator network;

[0077] Therefore, the loss function formula of the entire generator network is:

[0078]

[0079] Where λ1, λ2, and λ3 are the weight parameters of the three loss functions;

[0080] Step 3.3: The discriminator obtains the segmentation output maps of the two domains and requires the discriminator to correctly determine which domain the data comes from as much as possible;

[0081] The discriminator loss function is:

[0082]

[0083] Where z=0 means the sample comes from the target domain, z=1 means the sample comes from the source domain, P is the segmentation output of the two domains, and D is the discriminator network;

[0084] Step 4: Alternately iterate and train the generator and discriminator. This process uses the "maximum-minimum adversarial" idea to continuously train the entire network and save the network's weight parameters.

[0085] Step 5: Load the weight parameters trained in step 4, input the infrared ship test dataset into the new generator network to generate the segmentation map, and T_test The average intersection-over-union (IoU) is calculated on the dataset (containing 316 samples and corresponding labels).

[0086] The above examples are only used to illustrate the present invention, not to limit the technical solutions described in the present invention. Therefore, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A semantic segmentation method for infrared ship images based on pixel domain adaptation, characterized in that: The following steps are involved: Step 1: Obtain the source domain visible light ship dataset X S And the corresponding label Y S , obtain the target domain infrared ship dataset X T , divide the target domain dataset into training set X according to the ratio of 6:4 T_train and the test set X T_test ; The source domain visible light ship dataset X S Contains N S_train samples, the source domain visible light ship dataset corresponds to the label Y S Contains N S_train samples, training set X T_train Contains N T_train samples, test set X T_test Contains N T_test samples and corresponding labels; Step 2: Construct a pixel domain adaptation model based on a generative adversarial network. The generator contains two sub-networks: an appearance similarity network and a segmentation network. The appearance similarity network accumulates the generated feature maps with the original images to generate new source and target domain ship images. The appearance similarity network is used to convert the source domain visible light image and the target domain infrared image into images with the two domains as similar as possible in appearance. The segmentation network is a Deeplab-v2 segmentation network with Resnet-101 as the backbone network. The segmentation network receives the converted source and target domain images and generates corresponding segmented images. The discriminator contains three parallel dilated convolutions. The discriminator uses a basic convolutional network to discriminate the generated segmented images, determine their domain categories, and thus achieve infrared ship segmentation. Step 3: Design loss functions for the new generator network and discriminator network respectively; Step 4: Alternately iterate and train the generator and discriminator, using the "maximum-minimum adversarial" approach to continuously train the entire network and save the network's weight parameters. Step 5: Load the weight parameters trained in step 4, input the infrared ship test dataset into the new generator network to generate the segmentation map, and T_test Calculate the average intersection-union ratio above.

2. The method for semantic segmentation of infrared ship images based on pixel domain adaptation according to claim 1, characterized in that: Step 2 includes: Step 2.1: Construct an appearance similarity network model, which is a convolutional network; the first three layers are convolutional layers, the convolution kernel sizes are {7×7, 3×3, 3×3}, the number of channels are {64, 128, 256}, the first convolution layer has a step size of 1, and the remaining two convolution layers have a step size of 2. Each convolution layer is followed by a block normalization layer and a rectified linear unit activation function; followed by two residual blocks, each residual block contains 2 convolution layers, each convolution layer has a 3×3 convolution kernel and 256 channels The convolution layer is composed of two layers, each with a kernel size of 3×3, a number of channels {128, 64}, and a stride of 2. Each deconvolution layer is followed by a block normalization layer and a rectified linear unit activation function. The last layer is a convolution layer with a kernel size of 7×7, a number of channels 3, and a stride of 1. After the convolution, the Tanh activation function is used to generate a color feature map, which is then added to the original image to generate new source and target domain ship images. Step 2.2: Input the newly generated source domain and target domain images into the segmentation network respectively to obtain the corresponding segmentation results. The segmentation result of the source domain visible light image is recorded as P S , the segmentation result of the target domain infrared image is recorded as P T ; The segmentation network is a Deeplab-v2 segmentation network with Resnet-101 as the backbone network; Step 2.3: Construct the discriminator model. The network is a convolutional network. The first three layers are convolutional layers. The convolution kernel size is 4×4, the number of channels is {64, 128, 256}, the stride of the first two convolutional layers is 2, the stride of the third convolutional layer is 1, and each convolutional layer is followed by an unsaturated rectified linear unit activation function with a parameter of 0.2; the fourth layer is a dilated convolutional layer, which contains three parallel dilated convolutions with dilated convolution rates of {1, 3, 5}, the convolution kernel size is 4×4, the number of channels is 256, the stride is 1, and each dilated convolution is followed by an unsaturated rectified linear unit activation function with a parameter of 0.2; the fifth layer is a convolutional layer with a convolution kernel size of 4×4 and the number of channels is 1. This convolutional layer returns the discrimination result as a classifier.

3. The method for semantic segmentation of infrared ship images based on pixel domain adaptation according to claim 1, characterized in that: Step 3 includes, Step 3.1: The loss function of the appearance similarity network includes: overall diversity loss L varia , whose goal is to remove the rough texture information in the image; the conversion consistency loss L cyc , whose goal is to reduce the impact of image color differences between the two domains; color control loss L color , its purpose is to ensure the consistency of image information before and after conversion; The overall diversity loss function is: Where, l∈{1≤l≤N S_train } is the lth sample, are the two domain images of the input, Two domain images generated for the appearance similarity network, is the data after the source domain image is converted, is the data after the target domain image is converted, N l For X l The total number of pixel values ​​in and are the gradient values ​​of color intensity between adjacent pixels along the x and y directions, respectively, and ||·||1 is the L1 norm of the sum of all pixel values; The conversion consistency loss function is: Where, l∈{1≤l≤N S_train } is the lth sample, are the two domain images of the input, Two domain images generated for the appearance similarity network, is the data after the source domain image is converted, is the data after the conversion of the target domain image, ||·||1 is the L1 norm of the sum of all pixel values; The color controllability loss function is: Where, is a function of the average pooling kernel size 32*32, M l for The number of pixel values ​​in E l The value of is dynamically changed and is set to the average color intensity value of the infrared image in each iteration; The overall loss function of the appearance similarity model is defined as: L ASN =α1L varia +α2L cyc +α3L color (4) Where, parameters α1, α2, and α3 are the weight parameters of the three loss functions; Step 3.2: For the source domain output graph in the segmentation network, a portion of the corresponding labels are subjected to cross entropy loss to ensure the segmentation performance of the source domain; The cross entropy loss function formula is: Where, l∈{1≤l≤N S_train } is the lth sample, h, w, c, and C are the height, width, specific category, and total category of the image, respectively. The segmentation output map generated by the source domain data input network; the other part is the segmentation output of the target domain One piece serves as the input to the discriminator, ensuring that the images of the two domains are as close as possible to confuse the discriminator. This part requires adversarial loss to complete; The adversarial loss function formula is: Where, l∈{1≤l≤N S_train } is the lth sample, The segmentation output map generated by the target domain data input network, D is the subsequent discriminator network; The loss function formula of the entire generator network is: Where λ1, λ2, and λ3 are the weight parameters of the three loss functions; Step 3.3: The discriminator obtains the segmentation output maps of the two domains and requires the discriminator to correctly determine which domain the data comes from as much as possible; The discriminator loss function is: Where z = 0 means the sample comes from the target domain, z = 1 means the sample comes from the source domain, P is the segmentation output of the two domains, and D is the discriminator network.

Images