A cross-spectral image semantic segmentation method based on texture-independent features

Abstract

The application provides a cross-spectrum image semantic segmentation method based on texture-independent features, extracts texture-independent features, performs semantic segmentation based on the texture-independent features, generates a large amount of training data of the same structure but different textures through stylized images, explicitly punishes the network to learn texture-related information through full-symmetry cumulative texture-independent loss, requires the network to extract features containing enough non-texture information to recover semantics through a reconstruction loss based on structure information, and makes the model parameters better biased to the downstream task through a task-oriented fine-tuning method. The cross-spectrum semantic segmentation effect of the application can be run on all other spectrums, and the order of the sequence can be arbitrarily replaced without changing the loss function value, so that the similarity of the encoding of all styles to the real latent space distribution can be better considered, and better training performance can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a cross-spectrum image semantic segmentation method. BACKGROUND

[0002] Pixels in an image carry diverse semantic information, and the goal of image semantic segmentation is to classify these pixels with different semantics to achieve in-depth understanding of the image at the pixel level. In the past few years, semantic segmentation methods based on convolutional neural networks have made significant progress. However, due to the local nature of convolution operations, these methods have certain limitations in capturing global and long-distance semantic information interactions.

[0003] In recent years, the Vision Transformer model has been widely used in the field of computer vision. Due to the advantages of handling long-range dependencies and excellent performance, the powerful image feature extraction capability of the Transformer model makes it a key research direction in the field of image semantic segmentation.

[0004] In the cross-spectrum semantic segmentation task, the annotation of the training data is inconsistent, only part of the spectrum has annotation, and other spectra have no annotation. In addition, the data in the actual running environment are more from the spectrum without annotation. Different spectra may have different texture features, for example, compared with visible light images, infrared images have almost no texture. This difference leads to overfitting of the texture of the training spectrum of the network trained on a single spectrum, making it difficult to generalize to other spectra, which reduces the performance on other spectra. SUMMARY

[0005] In order to overcome the deficiencies of the prior art, the present application provides a cross-spectrum image semantic segmentation method based on texture-independent features. The purpose of the present application is to provide a cross-spectrum image semantic segmentation method. The key of the present application is to overcome the inconsistency between multiple spectra, so as to achieve the effect of training on one spectrum and running on all spectra. The key design of the present application is two steps: 1. Extract texture-independent features, 2. Perform semantic segmentation based on texture-independent features. It includes: generating a large amount of training data with the same structure but different textures through stylized images, explicitly punishing the network to learn texture-related information through the full-symmetry cumulative texture-independent loss, requiring the network to extract features containing enough non-texture information to recover semantics through the structure information-based reconstruction loss, and using the task-oriented fine-tuning method to make the model parameters better biased to the downstream task.

[0006] While different spectra have distinct textural features, they share similar geometric characteristics. These features share certain commonalities across spectra, allowing for similarities in shape and structure even when the textural features differ. The goal of this invention is to achieve improved cross-spectral performance by leveraging these similar non-textural features across spectra.

[0007] To address inconsistent data annotation, this paper uses a weakly supervised learning approach. Because only some spectra in the training data are labeled, while others are unlabeled, weakly supervised learning utilizes limited labeled data and a large amount of unlabeled data for training, thereby alleviating the problem of insufficient labeled data and playing an effective role in cross-spectral tasks.

[0008] The specific steps of the technical solution adopted by the present invention to solve the technical problem are as follows:

[0009] S1. Extracting texture-independent features

[0010] This step includes two requirements: first, the extracted texture-independent features do not contain texture information; second, the extracted texture-independent features should contain as much information as possible in the image except for texture, that is, non-texture information can be reconstructed from the features; the training method for texture-independent feature extraction is carried out as follows:

[0011] S1.1 For the i-th training data x in the training set (i) , firstly, the style transfer method is used to transfer x (i) Transform into a variety of different image styles

[0012] S1.2 For each stylized image Use any feature extraction network f of ResNet (residual neural network), ViT (Vision Transformer) and PVT (pyramid vision transformer) to extract features and obtain Latent space encoding

[0013] S1.3 Use any reconstruction module g of the deconvolutional neural network, self-attention decoder, or Transformer decoder to reconstruct the original image from the latent space encoding

[0014] S1.4 Calculate the fully symmetric cumulative texture-independent loss and the reconstruction loss L based on structural information r=a·ssim(x',x)+b·edge(x',x), where loss(·,·) is any binary loss function that measures similarity, such as mean square error, KL divergence, cross entropy, or cosine loss function. express The average value of ; ssim is the structural similarity loss, edge is the edge loss; a, b and c are all coefficient hyperparameters; inspired by the sparse autoencoder and the contraction autoencoder, the sparse penalty loss L is added s and sensitivity penalty loss L c , and obtain the final loss L = L t +L r +L s +L c After calculating the final loss value, the gradient of the loss value is forward propagated to the network parameters, and the gradient is used as the change in the network parameters. The network is trained through this process;

[0015] S2. Cross-spectral semantic segmentation based on texture-independent features

[0016] This step completes the cross-spectral semantic segmentation task, that is, only needs to be trained on one type of spectral data to achieve good performance on all spectra. The specific method is as follows:

[0017] For any spectral image x, first extract its texture-independent feature z according to step S1, then use any neural network algorithm including deconvolutional neural network, self-attention decoder, and Transformer decoder method to upsample the latent space code z, and finally pass it through the softmax layer to predict the category y of each pixel. The entire module is denoted as s. From the input and output point of view, module s inputs a latent space code and outputs a mask map of the semantic segmentation result, which is expressed by the formula. The above is the operation process of the cross-spectral semantic segmentation network. The network adopts the mean square error or cross entropy loss function for training;

[0018] S3. Task-oriented fine-tuning

[0019] After the training of the texture-independent feature extraction module and the cross-spectral semantic segmentation module is completed, the texture-independent feature extraction module is fine-tuned using all the training data. The fine-tuning steps are as follows:

[0020] If the training data is labeled data, when training the texture-independent feature extraction network, according to the experimental effect when training the network, the total number of training data is multiplied by the proportion α to obtain selected data, and α of the application is 0.3. The selected data is sent to the downstream cross-spectrum semantic segmentation module, and the network is trained by using the loss gradient generated by the semantic segmentation module to make the texture-independent feature extraction network better adapt to the downstream algorithm; the rest of the data not selected is sent to the reconstruction module according to step S1 for training.

[0021] S4. Deploy the trained model on the required platform; during runtime, send the image to be segmented x into the texture-independent feature extraction module f to obtain the texture-independent feature z = f(x), and then send the texture-independent feature into the cross-spectrum semantic segmentation module s to obtain the final segmentation result y = s(f(x)).

[0022] The style transfer refers to arbitrary modification of the texture and color of an image, but ensuring that all style-transferred images have similar geometric structures, for example, an apple in a realistic style, an apple in a simple sketch style, and an apple in a black-and-white style are all apples, and they have similar appearances; specifically, various deep learning-based style transfer algorithms such as SEAN (Style-Enhanced Adversarial Network) and FUNIT (Few-Shot Unsupervised Image-to-Image Translation) are used for style transfer, and various graphics algorithms such as tone mapping, cartoon rendering, and various non-photorealistic rendering techniques of different styles are used for style transfer, and finally multiple different style variants of the same image are obtained;

[0023] The steps S1.1 to S1.4 are methods for training the texture-independent feature extraction module, and during runtime, one of the following two methods is selected according to different requirements:

[0024] 1) In order to achieve faster running efficiency, the original picture x to be extracted for texture-independent features is directly sent to the feature extraction network f, and the hidden space encoding z = f(x) of x is calculated;

[0025] 2) In order to achieve more robust performance, a plurality of style-transferred images x j are generated from the original picture x j , and then sent to the feature extraction network f to obtain a group of hidden space encodings z j = f(x j ), and finally the group of hidden space encodings is averaged to obtain the final output hidden space encoding z = avg({z j}

[0026] During training, various common dataset enhancement methods and network training methods are used to achieve better model performance. For example:

[0027] Before sending the training data into the network training, the images are cropped, rotated, translated, distorted and denoised to enhance the data set, thereby better improving the robustness of the model.

[0028] Before each network starts training, the Kaiming initialization and Xavier initialization methods are used to set the initial values ​​of the network parameters.

[0029] During training, dropout layers and batch normalization layers are added according to the network structure, and various adaptive learning rate algorithms are used to accelerate convergence and reduce the possibility of model overfitting.

[0030] Regularization terms of various regularization methods such as L0 regularization term, L1 regularization term, and L2 regularization term are added to the loss function to improve the generalization ability of the model.

[0031] When the image is input into the network, the original image needs to be reshaped at the input and output of the model so that the image size matches the input and output size of the model.

[0032] During feature extraction, if the texture-independent feature extraction module has a multi-layer structure, the outputs of the intermediate layers are concatenated and input into the reconstruction module and the cross-spectral semantic segmentation module to fuse information at different scales.

[0033] The beneficial effects of the present invention are:

[0034] The method for extracting texture-independent features in an image can extract shape information in the image that is more critical for semantic segmentation, thereby achieving a cross-spectral semantic segmentation effect that only requires training on one spectrum to run on all other spectra.

[0035] The advantages of the fully symmetric cumulative texture-independent loss function are: 1. Full symmetry, meaning The loss function value will not change if the order of the images is arbitrarily permuted, thus better considering the similarity of all styles of encoding to the true latent space distribution; this property ensures that the texture-independent features learned by the model remain unchanged when the image order changes, which is conducive to improving the robustness and generalization ability of the model. 2. The algorithm complexity is O(n), rather than O(n) used by the mean absolute error. 2 ) complexity, thus achieving better training performance.

[0036] The advantages of the task-oriented fine-tuning approach are:

[0037] (1) When fine-tuning the encoder, bias the encoder toward the downstream segmentation task to improve segmentation accuracy;

[0038] (2) It is not advisable to fine-tune the encoder using only labeled data, because during the fine-tuning process, the encoder will gradually tend to the spectral features of the labeled training set, resulting in a weakening of its cross-spectral capability;

[0039] (3) This method performs better than methods without fine-tuning, because the method of fixing the encoder weights cannot guarantee that the encoder output results match the processing of the downstream network. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the structure of the texture-independent feature extraction module of the present invention.

[0041] Figure 2 Schematic diagram of the operation process of the operation phase of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and examples.

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.

[0044] Example

[0045] 1. Data preprocessing stage:

[0046] We obtain a set of semantic segmentation training images of uniform size w×h, including annotated visible spectrum images and unannotated infrared images. We first enhance the dataset by rotating, cropping, slightly distorting, adding noise, translating, and flipping. Then, we transform all images {x (i)} i Perform style transfer. This invention adopts SEAN (Style-Enhanced Adversarial Network) and FUNIT (Few-Shot Unsupervised Image-to-Image Translation) methods, as well as Otsu binarization, cartoon rendering technology, grayscale image extraction, color inversion, stroke, pencil drawing style texture, tone mapping, blurring and other methods to perform style transfer respectively, and obtain a large number of stylized images of the original data.

[0047] 2. Feature extraction network training stage

[0048] Follow these three steps

[0049] S1. Extract features using texture-independent feature extraction module. Specifically, for each stylized image First, bilinear interpolation is used to reshape the image into a 512×512 image. Then, PVT (pyramid vision transformer) with pre-trained weights is used for feature extraction to obtain its latent space encoding. In this example, the specific structure of the feature extraction network is divided into four stages. The first stage divides the image into Each block is 4×4×3 in size. These blocks are then linearly mapped to obtain embeddings, which are then fed into a 6-layer Transformer encoder along with the positional embeddings. The output is reshaped to 128×128, resulting in the output z1 of the first stage. The Transformer encoder is structured as follows:

[0050] At each layer, the input x is first subjected to three linear transformations, resulting in three values ​​Q, K, and V, respectively. The weight matrices of these three linear transformations are learnable parameters. These linear transformations project the input x into different subspaces for subsequent attention calculations. Next, the attention score matrix is ​​calculated through dot products. The score matrix is ​​used to perform a weighted summation of the value sequence to obtain the output of the self-attention. The output of the self-attention stage is then residually connected with the input sequence x, that is, the output of the self-attention is added to the input to obtain a rich representation. Layer Normalization is then used to normalize the output after the residual connection. The output after the self-attention is further extracted through a feedforward neural network consisting of two fully connected layers and an activation function layer.

[0051] The subsequent two to four stages take the output of the previous stage as input. The structure of each stage is similar to the first stage, except that the image blocks and the shape of the final output result are different. Specifically, the block size of the second stage is The output shape is 64×64; the block size of the third stage is The output shape is 32×32; the fourth step block size is The output shape is 16×16; the feature maps of the outputs of stages 2 to 4 are denoted as z2, z3, and z4 respectively. Finally, the total output of the four stages {z1, z2, z3, z4} is obtained as the final output.

[0052] The image reshape operation is used to reshape all feature maps into square sizes to facilitate subsequent operations.

[0053] S2. Use the reconstruction module to reconstruct the original image from the latent space The reconstruction module in this example is a deconvolutional neural network. The specific structure is similar to the decoder structure of UNet. It is also composed of 4 blocks. The output of each layer is recorded as {z'1,z'2,z'3,z'4}, and each block contains 3 layers:

[0054] (1) Upsampling layer. Use 2×2 transposed convolution kernel with ReLU activation function and stride 2. The i-th block input z 4-i The upsampling result is recorded as u i .

[0055] (2) Skip connection layer: Add the feature map of the corresponding encoder layer and the feature map of the current decoder layer to obtain the fused feature map v i =u i +z 4-i .

[0056] (3) Convolutional layer: Use a 3×3 convolution kernel with ReLU activation function, a step size of 1, and a padding of 1. Output feature map z' i .

[0057] Finally, the output layer is added, which is a 1×1 convolutional layer, which is used to map the last layer feature map z'4 of the decoder to a predicted image x' of the same size as the input image.

[0058] S3. Calculate the fully symmetric cumulative texture-independent loss and the reconstruction loss L based on structural information r =a·ssim(x',x)+b·edge(x',x), where loss is mean square error, ssim is structural similarity loss, edge is edge loss, a=2, b=c=1. Add sparse penalty loss Get the final loss L = L t +L r +L s , and use this loss to train the network. During training, the Xavier initialization method and the Adam adaptive learning rate algorithm are used. The initial learning rate is set to 0.001, the learning rate decay parameter is set to 0.1, the momentum gradient descent factor is set to 0.9, and a quadratic regularization loss is added to the neural network parameters.

[0059] 3. Segmentation network training stage

[0060] The features extracted by the encoder are used to train the cross-spectral semantic segmentation module. In this example, the network structure of the cross-spectral semantic segmentation module is almost identical to that of the reconstruction module, and the training method is also the same. The only difference is that a softmax activation function layer is connected after the final output layer, and the loss function uses the cross-entropy loss with respect to the ground truth.

[0061] 4. Fine-tuning stage

[0062] Fine-tune the model using all the data. In this example, the training data is visible light data. When fine-tuning, select data according to a certain ratio (here set to 0.3):

[0063] The selected data will be sent to the downstream cross-spectral semantic segmentation module, and the loss gradient generated by the semantic segmentation module will be used to train the network to better adapt the texture-independent feature extraction network to the downstream algorithm.

[0064] The unselected data undergoes the same encoder training steps as in step 2. Specifically, for each visible light data stylized image x, a uniformly distributed random variable r~U(0,1) is sampled.

[0065] If r≤0.3, x is first sent to the texture-independent feature extraction module f for feature extraction to obtain z=f(x), and then z is sent to the cross-spectral semantic segmentation module s to obtain the semantic segmentation prediction result y=s(z). The entire model is then trained using the cross-entropy loss on the true result in the third step.

[0066] If r>0.3, then x is first sent to the texture-independent feature extraction module f for feature extraction to obtain z=f(x), and then z=f(x) is sent to the reconstruction module g to obtain the reconstructed image x'=g(x). And according to the loss function L=L in the second step t +L r +L s Train the entire model.

[0067] 5. Operation Phase

[0068] The trained model is deployed on the desired platform. At runtime, the image to be segmented x is fed into the texture-independent feature extraction module f to obtain the texture-independent features z = f(x). The texture-independent features are then fed into the cross-spectral semantic segmentation module s to obtain the final segmentation result y = s(f(x)).

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-spectral image semantic segmentation method based on texture-independent features, characterized by The steps include: S1. Extracting texture-independent features The training method for texture-independent feature extraction is carried out in the following steps: S1.1 For the i-th training data x in the training set (i) , firstly, the style transfer method is used to transfer x (i) Transform into a variety of different image styles S1.2 For each stylized image Use any feature extraction network f of ResNet, ViT and PVT to extract features and obtain Latent space encoding S1.3 Use any reconstruction module g of the deconvolutional neural network, self-attention decoder, or Transformer decoder to reconstruct the original image from the latent space encoding S1.4 Calculate the fully symmetric cumulative texture-independent loss and the reconstruction loss L based on structural information r =a·ssim(x',x)+b·edge(x',x), where loss(·,·) is any binary loss function that measures similarity, such as mean square error, KL divergence, cross entropy, or cosine loss function. express The average value of ssim is the structural similarity loss, edge is the edge loss; a, b and c are all coefficient hyperparameters; at the same time Inspired by sparse autoencoders and contraction autoencoders, sparse penalty loss L is added s and sensitivity penalty loss L c , and obtain the final loss L = L t +L r +L s +L c After calculating the final loss value, the gradient of the loss value is forward propagated to the network parameters, and the gradient is used as the change in the network parameters. The network is trained through this process; S2. Cross-spectral semantic segmentation based on texture-independent features The specific method is: For any spectral image x, first extract its texture-independent feature z according to step S1, then use any neural network algorithm including deconvolutional neural network, self-attention decoder, and Transformer decoder method to upsample the latent space code z, and finally pass it through the softmax layer to predict the category y of each pixel. The entire module is denoted as s. From the input and output point of view, module s inputs a latent space code and outputs a mask map of the semantic segmentation result. This is the operation process of the cross-spectral semantic segmentation network. The network adopts the mean square error or cross entropy loss function for training; S3. Task-oriented fine-tuning After the training of the texture-independent feature extraction module and the cross-spectral semantic segmentation module is completed, the texture-independent feature extraction module is fine-tuned using all the training data. The fine-tuning steps are as follows: If the training data is labeled data, then when training the texture-independent feature extraction network, based on the experimental results during network training, the total number of training data is multiplied by a ratio α to obtain selected data, with α set to 0.

3. The selected data is sent to the downstream cross-spectral semantic segmentation module, and the network is trained using the loss gradient generated by the semantic segmentation module to better adapt the texture-independent feature extraction network to the downstream algorithm; the remaining unselected data is sent to the reconstruction module for training according to step S1; S4. Deploy the trained model on the required platform. During runtime, the image to be segmented x is fed into the texture-independent feature extraction module f to obtain the texture-independent feature z = f(x). The texture-independent feature is then fed into the cross-spectral semantic segmentation module s to obtain the final segmentation result y = s(f(x)).

2. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: Style transfer refers to the arbitrary modification of an image's texture and color, while ensuring that all transferred images have similar geometric structures. Specifically, style transfer is performed using various deep learning-based style transfer algorithms from existing SEAN and FUNIT, and then using various graphics algorithms, including tone mapping, cartoon rendering, and various non-realistic rendering techniques, to ultimately obtain multiple different style variants of the same image.

3. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: Steps S1.1 to S1.4 are methods for training the texture-independent feature extraction module. During runtime, one of the following two methods can be selected based on different requirements: 1) For faster running efficiency, the original image x with texture-independent features to be extracted is directly fed into the feature extraction network f, and the latent space encoding z = f(x) of x is calculated; 2) For more robust performance, we first generate several style-transferred images x from the original image x j , and then sent to the feature extraction network f to obtain a set of latent space encoding z j =f(x j ), and finally average this set of latent space codes to get the final output latent space code z=avg({z j } j ), where avg is the function used to calculate the average.

4. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: Before sending the training data into the network training, the images are cropped, rotated, translated, distorted and denoised to enhance the data set, thereby better improving the robustness of the model.

5. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: Before each network starts training, the Kaiming initialization and Xavier initialization methods are used to set the initial values ​​of the network parameters.

6. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: During training, dropout layers and batch normalization layers are added according to the network structure, and various adaptive learning rate algorithms are used to accelerate convergence and reduce the possibility of model overfitting.

7. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: Regularization terms of various regularization methods such as L0 regularization term, L1 regularization term and L2 regularization term are added to the loss function to improve the generalization ability of the model.

8. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: When the image is input into the network, the original image needs to be reshaped at the input and output of the model so that the image size matches the input and output size of the model.

9. The method for cross-spectral image semantic segmentation based on texture-independent features according to claim 1, characterized in that: During feature extraction, if the texture-independent feature extraction module has a multi-layer structure, the outputs of the intermediate layers are concatenated and input into the reconstruction module and the cross-spectral semantic segmentation module to fuse information at different scales.

Images