An end-to-end domain-adaptive semantic segmentation method for infrared images

Abstract

The present invention provides an end-to-end domain-adaptive semantic segmentation method for infrared images. This method removes and merges duplicate semantic categories from a semantic segmentation dataset, obtains an infrared image dataset, and achieves semantic label consistency between the visible light dataset and the infrared dataset. A pseudo-visible light image is obtained, which is fused and input into a semantic segmentation network. The trained semantic segmentation model is used to segment the segmented infrared image to obtain the final result. This method significantly alleviates the domain adaptation challenges faced by infrared image semantic segmentation datasets. Furthermore, compared to separate generation and segmentation tasks, combining these two tasks yields better generated images and segmentation results, making it suitable for any semantic segmentation model. Furthermore, because this domain adaptation method allows for very large datasets, it far exceeds the generalization capabilities of typical small-scale infrared semantic datasets. This method can be used to train large models without overfitting.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, in particular to an infrared image semantic segmentation method. Background Art

[0002] Infrared imaging systems have advantages such as long imaging range and strong environmental adaptability, and are widely used in many fields such as military surveillance, medical diagnosis, environmental monitoring, and autonomous driving. However, how to effectively interpret these images poses significant challenges, one of which is the semantic segmentation problem.

[0003] Semantic segmentation is the process of dividing an image into multiple regions, each corresponding to a specific category, such as a person, vehicle, or building. In the context of infrared imaging, the task is to segment an image into objects or regions of interest that are distinguishable from the background by their thermal signature. This is significantly more complex than segmenting visible light images due to the low contrast, high noise, and lack of color information of infrared images.

[0004] The field of image segmentation has undergone a major revolution with the advent of convolutional neural networks. Existing networks have proven effective at learning to recognize complex patterns in images, making them particularly well-suited for semantic segmentation tasks. However, applying these techniques to infrared imagery presents its own set of challenges. A major issue is the lack of labeled training data, which is necessary for training deep learning models. Labeling infrared images is a time-consuming and expensive process, and in some cases, it may not even be possible due to privacy concerns or inaccessible areas.

[0005] To address this problem, researchers have turned to semi-supervised learning methods. These methods use a small amount of labeled data and a large amount of unlabeled data. They leverage the unlabeled data to learn useful representations of the data and then apply this knowledge to improve performance on the labeled data.

[0006] Domain adaptation techniques allow a model trained on one type of data (e.g., visible light images) to be applied to another type (e.g., infrared images). These techniques are particularly valuable for overcoming the scarcity of labeled infrared images.

[0007] The paper "Learning to Adapt Structured Output Space for Semantic Segmentation" (Tsai YH, Hung WC, Schulter S, et al. Learning to adapt structured output space for semantic segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7472-7481.) uses a shared convolutional layer to extract features from two image domains (both visible light domains) and adds a domain adaptation layer, including a segmentation network and a discriminator network. This means that they are based on GAN methods, but the GAN network training is unstable and there is no unique optimization for converting infrared images to visible light images.

[0008] The paper "Domain-Agnostic Prior for Transfer Semantic Segmentation" (Huo X, Xie L, Hu H, et al. Domain-agnostic prior for transfer semantic segmentation [C] / / Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition. 2022: 7075-7085.) introduces domain-independent prior coding, replacing the corresponding category in the image mask with the prior coding. However, this is only suitable for the case where the distribution of the source domain and the target domain is not much different. Because the domain cannot be fully represented by the prior coding, the effect is poor in the current context. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this paper provides an end-to-end semantic segmentation method for infrared image domain adaptation. To fully leverage the advantages of generative models, an image conversion flow is designed to migrate infrared images to the visible light domain. Furthermore, to ensure training stability, a progressive image fusion strategy is designed to achieve effective segmentation of the converted images.

[0010] The technical solution adopted by the present invention to solve its technical problem is:

[0011] An end-to-end domain-adapted semantic segmentation method for infrared images is implemented through the following steps:

[0012] S1. Obtain a semantic segmentation dataset of the original visible light image, remove and merge duplicate semantic categories in the semantic segmentation dataset, and ultimately obtain a preprocessed semantic segmentation dataset containing N non-background categories and 1 background category;

[0013] S2. Obtain an infrared image dataset, concatenate N categories from the semantic segmentation dataset for visible light images to obtain a text description, input this text description into an existing open-source open vocabulary semantic segmentation model for segmentation, and obtain pseudo labels for the infrared images. Unsegmented portions are labeled as background, ensuring consistency in semantic labels between the visible light and infrared datasets.

[0014] S3. Randomly select images I from the infrared image dataset A , randomly select image I from the semantic segmentation dataset of preprocessed visible light images B , for image I A and image I B Adopt linear interpolation strategy to get image I t , and the image I t The image is input into the image conversion flow module, and then the transfer speed f(I t ,t), t is a positive real number less than 1, where f represents the image conversion flow module, by solving dI t / dt=f(I t ,t) obtain the transfer process, let t = 1, obtain I A The corresponding pseudo visible light image I' A ;

[0015] S4. Calculate the fusion weight w = 1-e^(-L flow )∈[0,1], where L flow Represents the loss function of the image conversion flow module, which is used to make the distribution of infrared images and visible light images consistent. T categories in the preprocessed visible light image semantic segmentation dataset are randomly selected and the image I B The mask area is divided into two parts according to whether it belongs to T categories to form a new mask M. The pixels belonging to T categories are marked as 1, and those not belonging are marked as 0. For the area where M=1, the fused image is wI B +(1-w)I' A , for the area M=0, the fused image is (1-w)I B +wI' A ;

[0016] S5. Input the fused image into the semantic segmentation network for training. Stop training when the number of training iterations is ≥ 100,000.

[0017] S6. Use the semantic segmentation model trained in step S5 to segment the infrared image with segmentation to obtain the final result; during inference, it is only necessary to use the image conversion flow to convert the infrared image into a visible light image, and then directly input it into the semantic segmentation network.

[0018] In step S3, the image conversion flow module has a U-Net structure, including an encoder, a bottleneck layer, a decoder, and a time step embedding layer; the image conversion flow module has C layers, C ≥ 2, and each of the first layer to the C-1 layer includes D encoders and D decoders, D ≥ 2, the C-th layer is a bottleneck layer, and the first encoder of each layer from the first layer to the C-1 layer receives the temporal information encoded by the time step embedding layer and the feature map output by the first encoder of the previous layer as input, and the feature map of the first layer is considered to be I t , the 2nd to Dth encoders of each layer take the output of the previous encoder of the same layer as input; the first decoder of each layer from the 1st to the C-1th layer receives the temporal information encoded by the time step embedding layer, the feature map output by the first decoder of the next layer, and the feature map output by the Dth encoder of the same layer as input, and the kth encoder of each layer takes the output of the k-1th decoder of the same layer and the output of the D+1-kth encoder of the same layer as input, 2≤k≤D;

[0019] In step S5, the semantic segmentation network is arbitrary, and YoloV8 is used.

[0020] In step S5, the loss function of the semantic segmentation network is L seg , the total loss function is recorded as:

[0021] L total =L seg +αL flow +βL tv +γL edge

[0022] Among them, α, β, and γ are hyperparameters to balance the various loss functions. α, β, and γ are all non-negative. tv Represents the total variation loss, which is used to control noise; L edge =(∑ i,j |P A -P A’ |) / ∑ i,j P I Used to maintain the edge, where P A represents the result of applying MCI to the infrared image in a 3×3 patch and normalizing it to [0,1]. A’It represents the calculation result of the generated pseudo visible light image. The edge detection method MCI is proposed by the paper "Multifeature-based surround inhibition improves contour detection in natural images" (Yang KF, Li CY, Li Y J. Multifeature-based surround inhibition improves contour detection in natural images [J]. IEEE Transactions on Image Processing, 2014, 23 (12): 5020-5032.).

[0023] The L flow =∑ t [E(||(I B -I A )-(I t ,t)|| 2 )],L tv =∑ i,j [(x i,j-1 -x i,j ) 2 +(x i+1,j -x i,j )] 1 / 2 , where E represents the mathematical expectation and x represents the pseudo visible light image I' A .

[0024] The beneficial effect of the present invention is that by providing an end-to-end infrared image domain adaptation semantic segmentation method, the image conversion flow module is used to convert unlabeled infrared images into visible light images and provide fusion weights, which greatly alleviates the domain adaptation problem of infrared image semantic segmentation datasets. In addition, compared with the separate generation task and segmentation task, the combination of these two tasks will obtain better generated images and segmentation results. For example, when generating images, the semantic information will be well maintained without artifacts. At the same time, the semantic segmentation method proposed in the present invention is suitable for any semantic segmentation model, and because it is a domain adaptation method, the dataset size can be very large, far exceeding the generalization ability of general small-scale infrared semantic datasets. The present invention can be used to train large models without causing overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the overall structural diagram of the present invention.

[0026] Figure 2 This is a schematic diagram of the image conversion flow module proposed in the present invention.

[0027] Figure 3 Schematic diagram of the encoder module used in the present invention.

[0028] Figure 4 Schematic diagram of the bottleneck layer used in the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and examples.

[0030] An end-to-end domain-adapted semantic segmentation method for infrared images, comprising the following steps:

[0031] S1. Obtain the visible light dataset and its mask for preprocessing and obtain the category:

[0032] Obtain a visible light semantic segmentation dataset (including but not limited to public datasets). First, resize all images to 512×512. Repeated semantic categories in these datasets need to be removed and merged. Finally, a preprocessed semantic segmentation dataset S containing N non-background and 1 background categories is obtained. B , the non-background categories are recorded as 1, 2, 3, ..., N, and the background category is recorded as 0.

[0033] S2. Obtain an infrared image dataset and use a large semantic segmentation model to obtain pseudo labels for infrared images:

[0034] Similarly, we first resize the infrared image to 512×512, then concatenate the N non-background categories of the preprocessed visible light semantic segmentation dataset to obtain a text description, such as "cat, dog, people, car". This text description is input into the open vocabulary semantic segmentation model CAT-Seg for segmentation to obtain pseudo labels for the infrared image. The unsegmented part is marked as background to ensure semantic consistency between the visible light dataset and the infrared dataset.

[0035] S3. Randomly select images I from the infrared image dataset A And select image I from the preprocessed visible light image dataset B , structure I t =(1-t)I A +tI B , according to the transfer speed f(I t ,t), where f represents the image conversion flow module and by solving dI t / dt=f(I t ,t) obtain the transfer process, let t=1 to obtain I A The corresponding pseudo visible light image I' A :

[0036] like Figure 2 As shown in FIG, the image conversion flow module in this embodiment has a U-Net structure with a total of 4 layers. Dense jump connections are required between the same layers. Each layer is composed of 3 encoders and 3 decoders. Figure 3 As shown, first the time step is encoded by the time step embedding layer module to obtain a vector of 1×1×time_c dimensions. In this embodiment, a series of sine and cosine function sequences (sinx, cosx, sin2x, cos2x…) are used to positionally encode the time step constant. Then it is projected into a time series feature vector of 1×1×out_c through the time projection layer. The input feature map dimension is h×w×in_c. After the convolution layer, the channel dimension is increased to out_c. The time series feature vector is added to the feature map and a convolution operation is performed to obtain a new feature map. The new feature map is input into the channel space attention CBAM module (composed of the channel attention and spatial attention modules) to obtain the final output feature map. As shown Figure 4 As shown in Figure 1, the bottleneck layer mainly consists of a downsampling module and an upsampling module, where the upsampling module is similar to the downsampling module. The decoder is similar to the encoder, but it also receives the output of the corresponding encoder of the same layer (concatenated with the original input feature map).

[0037] S4. Calculate the fusion weight w, randomly select T categories from the preprocessed visible light dataset, and transform I B The mask area is divided into two parts according to whether it belongs to the T categories, forming a new mask M. B with I' A To perform the fusion:

[0038] The fusion weight is 1-e^(-L flow ) is obtained, where L flow The loss function of the module is used to close the distribution of infrared images and visible light images. For the region M = 1, the fused image is wI B +(1-w)I' A , for the region M=0, the fused image is (1-w)I B +wI' A , performing the same fusion strategy on the masked portion. To ensure convergence, T decreases as training progresses, so that the fused image gradually approaches the pure transformed image. In this embodiment, the initial value of T is set to 10, and the final value is set to 1.

[0039] S5. Input the fused image into the semantic segmentation network and calculate the relevant loss function for training:

[0040] The semantic segmentation network can be arbitrary. The pre-trained YoloV8 performs slightly better on infrared images, so YoloV8 will be selected in this invention. The loss function of the semantic segmentation network is uniformly denoted as L seg , the total function is recorded as:

[0041] L total =L seg +αL flow +βL tv +γL edge

[0042] Among them, α, β, and γ are hyperparameters to balance the various loss functions. In this embodiment, α=1, β=5, γ=2, and L tv Represents the total variation loss, used to control noise, L edge Used to maintain the edge, it is obtained by the edge detection method MCI after some calculations.

[0043] Specifically, in this embodiment,

[0044] L flow =∑ t [E(||(I B -I A )-(I t ,t)|| 2 )],L tv =∑ i,j [(x i,j-1 -x i,j ) 2 +(x i+1,j -x i,j )] 1 / 2 , L edge =(∑ i,j |P A -P A '|) / ∑ i,j P I , where E represents the mathematical expectation and x represents I' A ,P A represents the result of applying MCI to the infrared image in a 3×3 patch and normalizing it to [0,1]. A’ It represents the calculation result of the generated visible light image.

[0045] In addition, when calculating L seg When the loss function L of the original semantic segmentation network is required s The correction is made based on , that is, only the area of ​​M = 1 is calculated. Taking into account the imbalance of semantic segmentation categories, it is also necessary to multiply the weighting factor λ, which represents the proportion of the selected T categories in the total categories.

[0046] S6. Use the trained semantic segmentation model to segment the infrared image with segmentation to obtain the final result:

[0047] During inference, the infrared images in the test dataset are first normalized as input, and then reasoned based on the trained model (the fusion strategy is removed here) to obtain the semantic segmentation results.

[0048] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An end-to-end domain-adaptive semantic segmentation method for infrared images, characterized by The steps include: S1. Obtain a semantic segmentation dataset of the original visible light image, remove and merge duplicate semantic categories in the semantic segmentation dataset, and ultimately obtain a preprocessed semantic segmentation dataset containing N non-background categories and 1 background category; S2. Obtain an infrared image dataset, concatenate N categories from the semantic segmentation dataset for visible light images to obtain a text description, input this text description into an existing open-source open vocabulary semantic segmentation model for segmentation, and obtain pseudo labels for the infrared images. Unsegmented portions are labeled as background, ensuring consistency in semantic labels between the visible light and infrared datasets. S3. Randomly select images I from the infrared image dataset A , randomly select image I from the semantic segmentation dataset of preprocessed visible light images B , for image I A and image I B Adopt linear interpolation strategy to get image I t , and the image I t The image is input into the image conversion flow module, and then the transfer speed f(I t ,t), t is a positive real number less than 1, where f represents the image conversion flow module, by solving dI t / dt=f(I t ,t) obtain the transfer process, let t = 1, obtain I A The corresponding pseudo visible light image I ’ A ; S4. Calculate the fusion weight w = 1-e^(-L flow )∈[0,1], where L flow Represents the loss function of the image conversion flow module, which is used to make the distribution of infrared images and visible light images consistent. T categories in the preprocessed visible light image semantic segmentation dataset are randomly selected and the image I B The mask area is divided into two parts according to whether it belongs to T categories to form a new mask M. The pixels belonging to T categories are marked as 1, and those not belonging are marked as 0. For the area where M=1, the fused image is wI B +(1-w)I ’ A , for the area M=0, the fused image is (1-w)I B +wI ’ A ; S5. Input the fused image into the semantic segmentation network for training. Stop training when the number of training iterations is ≥ 100,000. S6. Use the semantic segmentation model trained in step S5 to segment the infrared image with segmentation to obtain the final result; during inference, it is only necessary to use the image conversion flow to convert the infrared image into a visible light image, and then directly input it into the semantic segmentation network.

2. The end-to-end infrared image domain adaptive semantic segmentation method according to claim 1, characterized in that: In step S3, the image conversion flow module has a U-Net structure, including an encoder, a bottleneck layer, a decoder, and a time step embedding layer; the image conversion flow module has C layers, C ≥ 2, and each of the first layer to the C-1 layer includes D encoders and D decoders, D ≥ 2, the C-th layer is a bottleneck layer, and the first encoder of each layer from the first layer to the C-1 layer receives the temporal information encoded by the time step embedding layer and the feature map output by the first encoder of the previous layer as input, and the feature map of the first layer is considered to be I t , the 2nd to Dth encoders of each layer take the output of the previous encoder of the same layer as input; the first decoder of each layer from the 1st to the C-1th layer receives the temporal information encoded by the time step embedding layer, the feature map output by the 1st decoder of the next layer, and the feature map output by the Dth encoder of the same layer as input, and the kth encoder of each layer takes the output of the k-1th decoder of the same layer and the output of the D+1-kth encoder of the same layer as input, 2≤k≤D.

3. The end-to-end infrared image domain adaptive semantic segmentation method according to claim 1, characterized in that: In step S5, the semantic segmentation network is arbitrary, and YoloV8 is used.

4. The end-to-end infrared image domain adaptive semantic segmentation method according to claim 1, characterized in that: In step S5, the loss function of the semantic segmentation network is L seg , the total loss function is recorded as: L total =L seg +αL flow +βL tv +γL edge Among them, α, β, and γ are hyperparameters to balance the various loss functions. α, β, and γ are all non-negative. tv Represents the total variation loss, which is used to control noise; L edge =(∑ i,j |P A -P A’ |) / ∑ i,j P I Used to maintain the edge, where P A represents the result of applying MCI to the infrared image in a 3×3 patch and normalizing it to [0,1]. A’ It represents the calculation result of the generated pseudo visible light image.

5. The end-to-end infrared image domain adaptive semantic segmentation method according to claim 1, characterized in that: The L flow =∑ t [E(||(I B -I A )-(I t ,t)|| 2 )],L tv =∑ i,j [(x i,j-1 -x i,j ) 2 +(x i+1,j -x i,j )] 1 / 2 , where E represents the mathematical expectation and x represents the pseudo visible light image I' A .

Images