A multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model

Abstract

The application discloses a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model, and relates to the field of remote sensing image semantic segmentation. The method first extracts isomorphic features of each mode, uses a parameter non-shared full connection layer to extract attention weights of the isomorphic features of each mode and performs weighting, adds the weighted mode features to obtain multi-modal features after adaptive fusion, then adds random noise on a real probability distribution matrix by using a state transition matrix to obtain a probability matrix in which probabilities of all categories are equal, derives inverse diffusion real distribution of the discrete diffusion model through a Bayesian formula, and finally takes the fused multi-modal features as a condition, takes the diffusion real probability matrix as input, and predicts the real probability matrix without diffusion. The method can effectively deal with the heterogeneity and information conflict problems among different modes, significantly improves the understanding ability of the model to complex environment, and enhances the generalization ability and robustness of the model in various scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image semantic segmentation, and particularly relates to a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model. BACKGROUND

[0002] Multi-modal data involves image data from various modalities, such as visible light images and infrared images. Visible light images have rich color and detailed texture representation, but their imaging quality is severely degraded in foggy, overcast, and other environments due to their dependence on lighting conditions. Infrared images can penetrate smoke and obstacles to provide information that is difficult to detect in visible light by capturing the thermal radiation information of objects. By effectively fusing these multi-modal data, the limitations of a single modality in remote sensing scenarios can be overcome, and the remote sensing scene analysis capability can be improved.

[0003] Semantic segmentation is one of the key tasks in the field of computer vision, aiming to make dense predictions for each pixel in the input image to achieve accurate analysis of the scene. This task not only involves assigning specific class labels to each pixel, but also requires the model to understand and capture the shape boundaries and context information of objects in the image, and is an important means for applications such as autonomous driving, medical image analysis, and remote sensing scene understanding.

[0004] Multi-modal remote sensing image semantic segmentation technology aims to fuse multi-modal data collected by multiple sensors to achieve accurate pixel-by-pixel remote sensing scene analysis. This technology can be applied to land planning, agricultural detection, ecological protection, national security, and a series of other scenarios, providing technical support for intelligent management of various industries. However, current multi-modal remote sensing image semantic segmentation technology faces the following problems:

[0005] 1) The imaging characteristic differences between different modal images cause information domain bias, leading to confusion during the training process and affecting the performance of semantic segmentation. At the same time, the rigid fusion method limits the effective transmission of cross-modal information and causes the low-quality imaging modality to dominate during the fusion process, reducing the overall semantic segmentation performance.

[0006] 2) Existing semantic segmentation methods use discriminative classifiers to establish the mapping relationship between images and corresponding class labels, and do not deeply understand the distribution of deep semantic information of multi-modal images, resulting in poor generalization ability. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model, which dynamically fuses deep features of different modalities and uses a generative approach for pixel-by-pixel dense prediction, improving the generalization ability and performance of the network.

[0008] To achieve the above object, the application provides a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model, and the steps are as follows:

[0009] S1, collecting visible light and infrared paired images, the paired images between different modalities can be basically registered;

[0010] S2, adaptively fusing multi-modal image features;

[0011] S3, performing semantic segmentation operation by using a discrete diffusion model;

[0012] S4, training the model by using a batch stochastic gradient descent method.

[0013] Preferably, in S1, the RANSAC-Flow depth registration algorithm is used to perform high-precision registration on the image pairs of different modalities, and the Labelme labeling software is used to perform semantic segmentation mask labeling on each image pair.

[0014] Preferably, in S2, the multi-modal image feature adaptive fusion specifically includes the following steps:

[0015] S21, using a feature encoder with non-shared parameters to map the images of visible light and infrared modalities into an isomorphic space respectively, and extracting isomorphic features of each modality;

[0016] S22, using a parameter non-shared full connection layer to extract attention weights of the isomorphic features of each modality;

[0017] S23, weighting the modality features, and adding the weighted modality features to obtain the adaptively fused multi-modal features.

[0018] Preferably, in S21, the feature encoder includes a visible light modality image feature extraction network B rgb and an infrared modality image feature extraction network B ir ; the encoded image features of the visible light modality image feature extraction network B rgb are B rgb (I rgb ), and the encoded image features of the infrared modality image feature extraction network B ir are B ir (I ir ).

[0019] Preferably, in S22, a dynamic attention mechanism is used to weight the image features of each modality, and a full connection layer is used to extract attention weights of the visible light modality features and the infrared modality features, and the formula is as follows:

[0020] α rgb =σ(W rgb ·Brgb (I rgb )+b rgb );

[0021] α ir =σ(W ir ·B ir (I ir )+b ir );

[0022] Where, α rgb α represents the attention weights for visible light modal features. ir W represents the attention weights for infrared modal features. rgb and W ir Let b be the weight matrix. rgb and b ir σ is the bias term, and σ is the sigmoid activation function.

[0023] Preferably, in S23, the modal features are weighted by their corresponding attention weights and summed to obtain the multimodal features, as shown in the following formula:

[0024] N = α rgb ·B rgb (I rgb )+α ir ·B ir (I ir );

[0025] Where N represents multimodal features.

[0026] Preferably, the adaLN normalization method is used, taking the fused multimodal features N and time step codes t as inputs, and dynamically learning the bias and expansion parameters based on the input data, as shown in the following formula:

[0027] scale = W scale ·(N+t)+b scale );

[0028] shift = W shift ·(N+t)+b shift );

[0029] Where sale is the deviation parameter, W scale The weight parameters of the biased fully connected layer, b scale Here are the bias parameters for the biased fully connected layer, shift is the expansion parameter, and W... shift To expand the weight parameters of the fully connected layer, b shift To extend the bias parameters of the fully connected layer;

[0030] Attach scale and shift to the real mask F one-hot (x t) is characterized by the following formula:

[0031] N' = scale o Conv(F one-hot (x t ))+ shift;

[0032] where N' is the normalized modal feature.

[0033] Preferably, in S3, the specific steps of the semantic segmentation operation are as follows:

[0034] S31, add random noise to the real probability distribution matrix using the state transition matrix, gradually make the probability of each class of the probability matrix tend to be uniform, and obtain a probability matrix in which all class probabilities are equal;

[0035] S32, derive the inverse diffusion real distribution of the discrete diffusion model through the Bayes formula, so that the initialized probability matrix in which each element is equal is restored to the real probability matrix through gradual denoising;

[0036] S33, take the fused multi-modal visual feature as a condition, and take the real probability matrix after diffusion as input, to predict the real probability matrix without diffusion.

[0037] Preferably, in S31, the real probability distribution matrix is obtained by one-hot encoding operation on the real class mask M, and the formula is as follows:

[0038] P = F one-hot (M);

[0039] where P is the real probability distribution matrix, F one-hot (.) is the one-hot encoding operation;

[0040] Add T time step noise to the real class mask M to obtain T-1 hidden variables x1,...,x T , wherein the intermediate hidden variable x t is obtained by sampling from the distribution obtained by adding noise to x t-1 , and the formula is as follows:

[0041] x t ~ Q(x t | x t-1 ) = Cat(x t ; P t = F one-hot (x t-1 )·T t );

[0042] where x t is the intermediate hidden variable, x t-1 is the previous step hidden variable of x t , and Fone-hot (x t-1 ) is x t-1 one-hot label after one-hot encoding, T t is a state transition matrix, p t is a probability matrix, Q(.|.) is a diffusion process, and Cat(.) is a sampling distribution.

[0043] The state transition matrix is a symmetric matrix, the dimension of which is related to the number of real categories that need to be distinguished, the sum of the elements of any row or column is 1, and the diagonal elements are The non-diagonal elements are where β t is a preset hyperparameter greater than 0 and less than 1, and C is the number of categories.

[0044] The Gumbel-Softmax reparameterization trick is used for random sampling, and the category index formula of random sampling is as follows:

[0045] z = one-hot(argmax i {g i + log p i});

[0046] where p i is the probability of the i-th category, argmax i is the maximum value parameter operation, and g i is a random factor, and its calculation formula is as follows:

[0047] g i = -log(-log(u i ));

[0048] where u i is a randomly sampled value from a uniform distribution U(0, 1);

[0049] One-step noise addition technique is used to diffuse from the initial sampling x0 to time step t in one step, and the formula is as follows:

[0050]

[0051] where F one-hot (x0) is the initialized real mask after adding noise, is the factorial of T0,..., T t .

[0052] Preferably, in S32, the Bayesian derivation formula is as follows:

[0053]

[0054] where, The hidden space variable before state transition.

[0055] Therefore, the application provides a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model, which has the following beneficial effects:

[0056] (1) The application fully exploits the complementarity of multi-sensor data to improve the semantic segmentation accuracy in complex scenes;

[0057] (2) The application realizes adaptive fusion of multi-modal remote sensing image features by constructing a discrete diffusion model, effectively addressing the inherent heterogeneity and information conflict between different modalities;

[0058] (3) The application generates a class probability matrix by step-by-step generative prediction semantic segmentation, which not only significantly improves the model's understanding of complex environments, but also enhances its generalization ability and robustness in various scenarios.

[0059] The technical solutions of the application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model proposed by the application. DETAILED DESCRIPTION

[0061] In order to make the technical solutions, advantages and purposes of the application clearer, the technical solutions of the embodiments of the application will be described clearly and completely below. The described embodiments are part of the embodiments of the application, not all of the embodiments. Based on the described embodiments of the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the protection scope of the present application.

[0062] Unless otherwise defined, the technical terms or scientific terms used in the application should be understood as their usual meanings understood by those skilled in the art in the field of the application.

[0063] As shown in Figure 1 The application provides a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model, and the specific steps are as follows:

[0064] 1. Data preparation stage

[0065] A large number of visible light and infrared paired images are collected by unmanned aerial vehicle aerial photography, and the paired images between different modalities can be basically registered. RANSAC-Flow depth registration algorithm is used to perform high-precision registration on the image pairs of different modalities. Labelme labeling software is used to perform semantic segmentation mask labeling on each image pair, mainly including two categories of people and vehicles, and the remaining pixels are classified as background.

[0066] 2. Model design

[0067] 2.1. The overall model contains two feature extraction networks B for extracting features of visible light and infrared modal images respectively rgb and B ir , and an inverse diffusion distribution prediction network U composed of three fully connected layers. The data set is denoted as D∈{I rgb ,I ir ,M} where I rgb ∈R H×W×3 and I ir ∈R H×W×1 represent input images of visible light and infrared modal respectively, represent the real mask of semantic segmentation.

[0068] 2.2. The diffusion object of the discrete diffusion process is the real probability matrix P∈R H×W×C of semantic segmentation, which is obtained by one-hot encoding operation on the real class mask M, and the formula is as follows:

[0069] P=F one-hot (M);

[0070] where C represents the number of categories.

[0071] The number of categories of the data set used in the application is 3, and the initial sample of the diffusion process is x0, which is initialized as the real class mask M; T time step noise is added to x0 to obtain T-1 hidden variables x1,...,x T , wherein the intermediate hidden variable x t is sampled from the distribution obtained by adding noise to x t-1 , and the formula is as follows:

[0072] x t ~Q(x t |x t-1 )=Cat(x t ;P t =F one-hot (x t-1 )·T t );

[0073] where · is a matrix multiplication operation, T t is a state transition matrix, P t is a probability matrix, Q(.|.) is a diffusion process, and Cat(.) is a sampling distribution.

[0074] In order to meet the condition that the noise disturbance increases gradually with the increase of time step, T tis a symmetric matrix, and its dimension is related to the number of real categories that need to be distinguished, which is CxC here, and the sum of any row or column elements is 1, and T t The diagonal element of T is The non-diagonal element of T t is t where beta t is a preset hyperparameter greater than 0 and less than 1.

[0075] The present application adopts the reparameterization technique of Gumbel-Softmax to simulate the process of random sampling, and sets the probability matrix P t The probability vector of a certain pixel of P 1, is [p 1, p2,...,p C ], and the class index randomly sampled from the vector is:

[0076] z=one-hot(argmax i {g i +logp i});

[0077] where p i is the probability of the i-th class, and g i is a random factor, and its calculation formula is as follows:

[0078] g i =-log(-log(u i ));

[0079] where u i is a value randomly sampled from a uniform distribution U(0,1).

[0080] In order to ensure that the diffusion process can be carried out efficiently, a preset one-step noise adding technique is adopted to diffuse from the initial sampling x0 to any time step t in one step, and the formula is as follows:

[0081]

[0082] where, is the factorial of T0,...,T t .

[0083] For the theoretical inverse diffusion process corresponding to the discrete diffusion process, the Bayesian formula is used for derivation, and the formula is as follows:

[0084]

[0085] where F one-hot (x t ) is the noisy real mask, F one-hot (x0) is the noisy real mask initialized, is T0,...,T tfactorial of n, is the latent variable before state transition.

[0086] 2.3, the inverse diffusion distribution prediction network U receives the multi-modal feature extraction network B rgb and B ir encoded image features B rgb (I rgb ) and B ir (I ir ), time step t, and the noisy real mask F one-hot (x t ) as input. Among them, the multi-modal image features are fused as the condition to guide the prediction of the probability distribution, the time step t guides the network U to predict the probability distribution parameters of the current time step, and the noisy real mask F one-hot (x t ) can provide real semantic constraints and geometric constraints. In order to adaptively highlight the effective information of a single modality and suppress the redundant information of other modalities without affecting the complementarity of each modality image feature, a dynamic attention mechanism is used to weight each modality image feature. Specifically, two fully connected layers are used to extract the attention weights of the visible light modality feature and the infrared modality feature respectively, and the formula is as follows:

[0087] α rgb =σ(W rgb ·B rgb (I rgb )+b rgb );

[0088] α ir =σ(W ir ·B ir (I ir )+b ir );

[0089] Where W rgb and W ir are trainable weight matrices, b rgb and b ir are bias terms, and σ is a sigmoid activation function.

[0090] All modal features are weighted by the corresponding attention weights and added to the fused multi-modal features, and the formula is as follows:

[0091] N=α rgb ·B rgb (I rgb )+α ir ·B ir (I ir );

[0092] To reduce the internal covariate shift and accelerate the convergence of the model and improve the performance, the adaLN normalization layer is adopted, which receives the fused multi-modal feature N and time step encoding t as input and dynamically learns the bias and scaling parameters according to the input data:

[0093] scale=W scale ·(N+t)+b scale );

[0094] shift=W shift ·(N+t)+b shift );

[0095] The learned scale and shift are attached to the features of the real mask F one-hot (x t ) as scaling parameters, which helps to dynamically adjust the normalization degree and increase the stability and adaptability of the model:

[0096] N'=scale⊙Conv(F one-hot (x t ))+shift;

[0097] To ensure the lightweight of the model, the normalized features N' are passed through the lightweight fully connected layer to interact the features, and the residual connection is used to add the features after interaction and the features before interaction, so as to avoid the catastrophic forgetting of the model.

[0098] 2.4, the loss function used to build the training model is constructed, and the relative entropy D KL (Q(x t-1 |x t )∥P θ (x t-1 |x t )) between the real inverse diffusion distribution Q(x t-1 |x t ) derived in step 2.2 and the inverse diffusion distribution P θ (x t-1 |x t ) learned by the model is minimized. Since the time step t in P θ (x t-1 |x t ) is known, only the initial sample x0 needs to be predicted, and the cross-entropy loss function is used to predict the initial sample x0, and the formula is as follows:

[0099]

[0100] Where F one-hot (x0) is the predicted mask, y is the real mask, and C is the number of predicted categories.

[0101] 3, model training phase

[0102] 3.1 Divide the multimodal remote sensing semantic segmentation dataset D obtained in step 1 into a training set T and a validation set V.

[0103] 3.2. Transfer the multimodal data I from the training set T. rgb and I ir The input is fed into the model designed in step 2, and the model is trained using the batch stochastic gradient descent method. The maximum time step is preset to 1000. The time step parameter t is randomly sampled in the discrete interval [1,...,1000], and the result is added to the true probability matrix F. one-hot Adding noise at time step t to (M) yields the distribution. Randomly sample x from it using the Gumbel-Softmax reparameterization technique. t Multimodal data I rgb and I ir After passing through the multimodal feature extraction network B rgb and B ir Encoding, with x t Together with the time step t, the predicted x0 is obtained as input to the inverse diffusion distribution prediction network U. This predicted x0 is then used to construct the cross-entropy loss, which serves as the derivative signal. During the training phase, a validation set V is used to verify the model's training performance. That is, training stops when the model achieves good semantic segmentation results on the validation set V, and this accuracy cannot be further improved during the training process.

[0104] 3.3 Finally, the model is obtained after training is completed.

[0105] 4. Model Testing Phase

[0106] The input data is the registered multimodal data I. rgb and I ir After passing through the multimodal feature extraction network B rgb and B ir Encoding yields multimodal image features, from which all elements are... x is obtained by random sampling from the probability matrix using the Gumbel-Softmax reparameterization technique. t To ensure inference efficiency, the time step t is initialized to 100. Multimodal features, x t The predicted x0 is obtained by using t as input to the inverse diffusion distribution prediction network U, and then processed by Softmax to obtain the final class probability matrix. The predicted x0 in this step can be used as the distribution parameter sampling x in the next step. t-1 Repeat this process 100 times to obtain the final result x0.

[0107] Therefore, the multi-modal remote sensing image semantic segmentation method based on the discrete diffusion model can effectively alleviate the performance loss caused by noise, modal inconsistency and distribution deviation while maintaining high-resolution segmentation results, and exhibits excellent cross-modal fusion capability and fine pixel-level classification performance.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can still be modified or replaced by equivalents, and these modifications or equivalent replacements should not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A multimodal remote sensing image semantic segmentation method based on a discrete diffusion model, characterized in that, The steps are as follows: S1. Collect visible light and infrared paired images; the paired images between different modes can be basically registered. S2. Adaptive fusion of multimodal graphic features; S3. Perform semantic segmentation using a discrete diffusion model; S4. Use batch stochastic gradient descent to train the model; In S2, the adaptive fusion of multimodal graphical features specifically includes the following steps: S21. Using a feature encoder with non-shared parameters, the images of the visible light and infrared modes are mapped to the isomorphic space respectively, and the isomorphic features of each mode are extracted. S22. Use a fully connected layer with non-shared parameters to extract the attention weights for each modal isomorphic feature; S23. Assign weights to the modal features and add the weighted modal features together to obtain the adaptively fused multimodal features; In S21, the feature encoder includes a visible light modality image feature extraction network. Infrared modal image feature extraction network The visible light modal image feature extraction network The encoded image features are ( The infrared modal image feature extraction network The encoded image features are ; In S22, a dynamic attention mechanism is used to weight the image features of each modality, and a fully connected layer is used to extract the attention weights for visible light modality features and infrared modality features, as shown in the following formula: ； ； in, Attention weights for visible light modal features. Attention weights for infrared modal features. and This is the weight matrix. and For bias terms, It is the sigmoid activation function; In S23, the modal features are weighted by their corresponding attention weights and summed to obtain the multimodal features, as shown in the following formula: ； in, It is a multimodal feature.

2. The multimodal remote sensing image semantic segmentation method based on a discrete diffusion model according to claim 1, characterized in that, In S1, the RANSAC-Flow deep registration algorithm is used to perform high-precision registration of image pairs with different modalities, and the Labelme annotation software is used to perform semantic segmentation mask annotation for each image pair.

3. The multimodal remote sensing image semantic segmentation method based on a discrete diffusion model according to claim 1, characterized in that, The adaLN normalization method is used to normalize the fused multimodal features. N Using the time step code t as input, the bias and expansion parameters are dynamically learned based on the input data, as shown in the following formula: ； ； in, For deviation parameters, The weight parameters of the biased fully connected layer The bias parameters are for the biased fully connected layer. To expand parameters, To expand the weight parameters of the fully connected layer, To extend the bias parameters of the fully connected layer; Will and Attached to the real mask Based on the characteristics, the formula is as follows: ； in, These are the normalized modal features.

4. The multimodal remote sensing image semantic segmentation method based on a discrete diffusion model according to claim 1, characterized in that, In S3, the specific steps of semantic segmentation are as follows: S31. By using the state transition matrix to add random noise to the true probability distribution matrix, a probability matrix in which the probabilities of all categories are equal is obtained. S32. Derive the true inverse diffusion distribution of the discrete diffusion model using Bayes' theorem; S33. Using the fused multimodal visual features as conditions and the diffused true probability matrix as input, predict the undiffused true probability matrix.

5. The multimodal remote sensing image semantic segmentation method based on a discrete diffusion model according to claim 4, characterized in that, In S31, the true probability distribution matrix is ​​composed of a true class mask. M The formula obtained after one-hot encoding is as follows: ； in, This is the true probability distribution matrix. This is a one-hot encoding operation; In the real category mask By adding T time-step noise to the existing data, we obtain T-1 latent variables. Among them, intermediate hidden variables It is by The formula for sampling from the noise-added distribution is as follows: ； in, For intermediate implicit variables, for The previous hidden variable, for The uniquely coded label Here is the state transition matrix. It is a probability matrix. For diffusion process, For sampling distribution; The state transition matrix Its dimension is related to the number of true categories that need to be distinguished; the sum of any row or column element is 1; and the diagonal elements are... Off-diagonal elements are ,in For hyperparameters, C Number of categories; We employ the reparameterization technique of Gumbel-Softmax to perform random sampling. The formula for the category index of the random sampling is as follows: ； in, For the first The probability of each category To retrieve the maximum value parameter, The random factor is calculated using the following formula: ； in, To conform to a uniform distribution The value of the random sample; Employing a one-step noise-adding technique, from the initial sampling... Diffusion to time step t The formula is as follows: ； in, The actual mask initialized after adding noise. for factorial.

6. The multimodal remote sensing image semantic segmentation method based on a discrete diffusion model according to claim 4, characterized in that, In S32, the Bayesian derivation formula is as follows: ； in, These are the hidden space variables before the state transition.

Images