Multimodal remote sensing image semantic segmentation method based on discrete diffusion model

Abstract

The invention discloses a multi-modal remote sensing image semantic segmentation method based on a discrete diffusion model, and relates to the field of remote sensing image semantic segmentation. The method comprises the following steps: firstly, extracting the isomorphic feature of each modal, extracting the attention weight of the isomorphic feature of each modal by using a full connection layer with non-shared parameters, performing weighting, and adding the weighted modal features to obtain a multi-modal feature after adaptive fusion; then adding random noise on the real probability distribution matrix by using a state transition matrix to obtain a probability matrix in which all types of probabilities are equal, and deriving inverse diffusion real distribution of the discrete diffusion model through a Bayesian formula; and finally, predicting an undiffused real probability matrix by taking the fused multi-modal features as conditions and the diffused real probability matrix as input. According to the method, the problems of heterogeneity and information conflict among different modes can be effectively solved, the understanding ability of the model to a complex environment is remarkably improved, and the generalization ability and robustness of the model in various scenes are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a multimodal remote sensing image semantic segmentation method based on a discrete diffusion model. Background Art

[0002] Multimodal data involves image data from various modalities, such as visible light images and infrared images. Visible light images have rich colors and detailed textures, but their dependence on lighting conditions causes their imaging quality to be severely reduced in environments such as haze and cloudy days. Infrared images can capture the thermal radiation information of objects, penetrate smoke and obstacles, and provide information that is difficult to detect under visible light. By effectively fusing these multimodal data, the limitations of a single modality in remote sensing scenarios can be overcome and the remote sensing scene resolution capability can be improved.

[0003] Semantic segmentation is one of the key tasks in the field of computer vision, which aims to make dense pixel-by-pixel predictions on the input image to achieve accurate analysis of the scene. This task not only involves assigning specific category labels to each pixel, but also requires the model to understand and capture the shape boundaries and contextual information of objects in the image. It is an important means for applications such as autonomous driving, medical image analysis, and remote sensing scene understanding.

[0004] Multimodal remote sensing image semantic segmentation technology aims to integrate multimodal data collected by multiple sensors to perform accurate pixel-by-pixel remote sensing scene analysis. This technology can be applied to a series of scenarios such as land planning, agricultural detection, ecological protection, and national security, providing technical support for intelligent management of various industries. However, the current multimodal remote sensing image semantic segmentation technology faces the following problems:

[0005] 1) The difference in imaging characteristics between images of different modalities causes information domain deviation, resulting in confusion during training, which in turn affects the performance of semantic segmentation. At the same time, the rigid fusion method limits the effective transmission of cross-modal information and causes the modality with low-quality imaging to dominate during the fusion process, reducing the overall semantic segmentation performance.

[0006] 2) Existing semantic segmentation methods use the method of constructing discriminative classifiers to establish the mapping relationship between images and corresponding category labels, but do not have an in-depth understanding of the deep semantic information distribution of multimodal images, resulting in poor generalization ability. Summary of the invention

[0007] The purpose of this invention is to propose a multimodal remote sensing image semantic segmentation method based on a discrete diffusion model. This method dynamically fuses the deep features of different modalities and uses a generative approach to perform pixel-by-pixel dense prediction, thereby improving the semantic segmentation generalization ability and performance of the network.

[0008] To achieve the above object, the present invention proposes a multimodal remote sensing image semantic segmentation method based on a discrete diffusion model, the steps of which are as follows:

[0009] S1. Collect visible light and infrared paired images. Paired images between different modalities can be basically registered.

[0010] S2, adaptive fusion of multimodal graphic features;

[0011] S3, using discrete diffusion model to perform semantic segmentation operation;

[0012] S4. Use batch stochastic gradient descent method to train the model.

[0013] Preferably, in S1, the RANSAC-Flow deep registration algorithm is used to perform high-precision registration on image pairs of different modalities, and the Labelme annotation software is used to perform semantic segmentation mask annotation on each image pair.

[0014] Preferably, in S2, the multimodal graphic feature adaptive fusion specifically includes the following steps:

[0015] S21, using a feature encoder with non-shared parameters to map the visible light and infrared modality images into homogeneous spaces respectively, and extract homogeneous features of each modality;

[0016] S22, use a fully connected layer with non-shared parameters to extract the attention weights of each modality isomorphic feature;

[0017] S23, weighting the modal features, and adding the weighted modal features to obtain multi-modal features after adaptive fusion.

[0018] Preferably, in S21, the feature encoder includes a visible light modality image feature extraction network B rgb and infrared modality image feature extraction network B ir ; The visible light modality image feature extraction network B rgb The encoded image feature is B rgb (I rgb ), the infrared modality image feature extraction network B ir The encoded image feature is B ir (I ir ).

[0019] Preferably, in S22, a dynamic attention mechanism is used to weight the image features of each modality, and a fully connected layer is used to extract the attention weights of the visible light modality features and the infrared modality features, and the formula is as follows:

[0020] α rgb =σ(W rgb ·Brgb (I rgb )+b rgb );

[0021] α ir =σ(W ir ·B ir (I ir )+b ir );

[0022] Among them, α rgb is the attention weight of the visible light modality feature, α ir is the attention weight of infrared modality features, W rgb and W ir is the weight matrix, b rgb and b ir is the bias term, and σ is the sigmoid activation function.

[0023] Preferably, in S23, the modal features are weighted by the corresponding attention weights, and the multimodal features are added together to obtain the following formula:

[0024] N=α rgb ·B rgb (I rgb )+α ir ·B ir (I ir );

[0025] Among them, N is the multimodal feature.

[0026] Preferably, the adaLN normalization method is used, the fused multimodal features N and the time step encoding t are used as input, and the bias and expansion parameters are dynamically learned according to the input data. The formula is as follows:

[0027] scale=W scale ·(N+t)+b scale );

[0028] shift=W shift ·(N+t)+b shift );

[0029] Among them, sale is the deviation parameter, W scale The weight parameter of the biased fully connected layer, b scale is the bias parameter of the biased fully connected layer, shift is the expansion parameter, and W shift is the weight parameter of the expanded fully connected layer, b shift is the bias parameter for the extended fully connected layer;

[0030] Append scale and shift to the true mask F one-hot (x t), the formula is as follows:

[0031] N'=scale⊙Conv(F one-hot (x t ))+shift;

[0032] Among them, N' is the normalized modal feature.

[0033] Preferably, in S3, the specific steps of the semantic segmentation operation are as follows:

[0034] S31, using the state transfer matrix to add random noise to the true probability distribution matrix, gradually making the probability of each category of the probability matrix tend to be uniform, and obtaining a probability matrix in which the probabilities of all categories are equal;

[0035] S32, deriving the inverse diffusion true distribution of the discrete diffusion model through the Bayesian formula, so that the initialization probability matrix with each element equal is restored to the true probability matrix through step-by-step denoising;

[0036] S33. Take the fused multimodal visual features as conditions and the diffused true probability matrix as input to predict the undiffused true probability matrix.

[0037] Preferably, in S31, the true probability distribution matrix is ​​obtained by one-hot encoding the true category mask M, and the formula is as follows:

[0038] P=F one-hot (M);

[0039] Among them, P is the true probability distribution matrix, F one-hot (.) is the one-hot encoding operation;

[0040] Add T time step noise to the real category mask M to get T-1 hidden variables x1,...,x T , where the intermediate latent variable x t By x t-1 The formula is as follows:

[0041] x t ~Q(x t |x t-1 )=Cat(x t ;P t =F one-hot (x t-1 )·T t );

[0042] Among them, x t is the intermediate hidden variable, x t-1 For x t The previous step hidden variable, Fone-hot (x t-1 ) is x t-1 The one-hot encoded one-hot label, T t is the state transfer matrix, p t is the probability matrix, Q(.|.) is the diffusion process, and Cat(.) is the sampling distribution;

[0043] The state transfer matrix is ​​a symmetric matrix whose dimension is related to the number of real categories that need to be distinguished. The sum of any row or column elements is 1, and the diagonal elements are The off-diagonal elements are where β t is a preset hyperparameter greater than 0 and less than 1, and C is the number of categories;

[0044] The Gumbel-Softmax reparameterization technique is used for random sampling. The category index formula for random sampling is as follows:

[0045] z=one-hot(argmax i {g i +log p i});

[0046] Among them, p i is the probability of the i-th category, argmax i For the maximum value parameter operation, g i is a random factor, and its calculation formula is as follows:

[0047] g i =-log(-log(u i ));

[0048] Among them, u i is a randomly sampled value from uniform distribution U(0,1);

[0049] Using the one-step noise addition technique, diffuse from the initial sample x0 to the time step t in one step. The formula is as follows:

[0050]

[0051] Among them, F one-hot (x0) is the real mask initialized after adding noise, For T0,...,T t The factorial of .

[0052] Preferably, in S32, the Bayesian derivation formula is as follows:

[0053]

[0054] in, is the latent space variable before state transfer.

[0055] Therefore, the present invention proposes a multimodal remote sensing image semantic segmentation method based on a discrete diffusion model, and its beneficial effects are as follows:

[0056] (1) The present invention fully exploits the complementarity of multi-sensor data and improves the accuracy of semantic segmentation in complex scenes;

[0057] (2) The present invention realizes the adaptive fusion of multimodal remote sensing image features by constructing a discrete diffusion model, effectively coping with the inherent heterogeneity and information conflict problems between different modalities;

[0058] (3) The present invention not only significantly improves the model's ability to understand complex environments, but also enhances its generalization ability and robustness in various scenarios by gradually generating and predicting the category probability matrix of semantic segmentation.

[0059] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of a multimodal remote sensing image semantic segmentation method based on a discrete diffusion model proposed in the present invention. DETAILED DESCRIPTION

[0061] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention are clearly and completely described below. The described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.

[0062] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0063] like Figure 1 As shown, the present invention proposes a multimodal remote sensing image semantic segmentation method based on a discrete diffusion model, and the specific steps are as follows:

[0064] 1. Data preparation stage

[0065] A large number of visible light and infrared paired images were collected by drone aerial photography, and paired images between different modalities can be basically registered. The RANSAC-Flow deep registration algorithm is used to perform high-precision registration of image pairs of different modalities. The Labelme annotation software is used to perform semantic segmentation mask annotation on each image pair, which mainly includes two categories: people and cars, and the remaining pixels are classified as background.

[0066] 2. Model design

[0067] 2.1. The overall model consists of two feature extraction networks B for extracting visible light and infrared modality image features respectively. rgb and B ir , and the inverse diffusion distribution prediction network U consisting of three fully connected layers. The data set is denoted as D∈{I rgb ,I ir ,M}, where I rgb ∈R H×W×3 and I ir ∈R H×W×1 represent the input images in visible and infrared modalities, respectively, Represents the ground-truth mask for semantic segmentation.

[0068] 2.2. The diffusion object of the discrete diffusion process is the true probability matrix P∈R of semantic segmentation H×W×C , obtained by one-hot encoding the true category mask M, the formula is as follows:

[0069] P=F one-hot (M);

[0070] Here, C represents the number of categories.

[0071] The number of categories of the data set used in the present invention is 3. Assume that the initial sample of the diffusion process is x0, and x0 is initialized to the true category mask M. Add T time step noise on the basis of x0 to obtain T-1 hidden variables x1,...,x T , where the intermediate latent variable x t By x t-1 The formula is as follows:

[0072] x t ~Q(x t |x t-1 )=Cat(x t ;P t =F one-hot (x t-1 )·T t );

[0073] Where, · is the matrix multiplication operation, T t is the state transfer matrix, P t is the probability matrix, Q(.|.) is the diffusion process, and Cat(.) is the sampling distribution.

[0074] In order to satisfy the condition that the noise disturbance gradually increases with the increase of time steps, T tis a symmetric matrix, and its dimension is related to the number of real categories that need to be distinguished, here it is C×C, and the sum of any row or column elements is 1, and T t The diagonal elements of The off-diagonal elements are Among them, β t is a preset hyperparameter greater than 0 and less than 1.

[0075] The present invention adopts the Gumbel-Softmax reparameterization technique to simulate the random sampling process. Let the probability matrix P t The probability vector of a pixel is [p 1, p2,...,p C ], then the category index randomly sampled from this vector is:

[0076] z=one-hot(argmax i {g i +logp i});

[0077] Among them, p i is the probability of the i-th category, g i is a random factor, and its calculation formula is as follows:

[0078] g i =-log(-log(u i ));

[0079] Among them, u i is a value randomly sampled from a uniform distribution U(0,1).

[0080] In order to ensure that the diffusion process can be carried out efficiently, a preset one-step noise addition technique is used to diffuse from the initial sample x0 to any time step t in one step. The formula is as follows:

[0081]

[0082] in, For T0,...,T t The factorial of .

[0083] For the theoretical inverse diffusion process corresponding to the discrete diffusion process, the Bayesian formula is used to derive the formula as follows:

[0084]

[0085] Among them, F one-hot (x t ) is the true mask after adding noise, F one-hot (x0) is the real mask initialized after adding noise, For T0,...,T tThe factorial of is the latent space variable before state transfer.

[0086] 2.3. The inverse diffusion distribution prediction network U receives the multimodal feature extraction network B rgb and B ir Encoded image feature B rgb (I rgb ) and B ir (I ir ), time step t, and the true mask F after adding noise one-hot (x t As input. Among them, the multimodal image features are fused as the conditional guidance probability distribution prediction, the time step t guides the network U to predict the probability distribution parameters of the current time step, and the real mask F after noise addition one-hot (x t ) can provide real semantic constraints and geometric constraints. In order to adaptively highlight the effective information of a single modality and suppress the redundant information of other modalities without affecting the complementarity of the image features of each modality, a dynamic attention mechanism is used to weight the image features of each modality. Specifically, two fully connected layers are used to extract the attention weights of visible light modality features and infrared modality features respectively. The formula is as follows:

[0087] α rgb =σ(W rgb ·B rgb (I rgb )+b rgb );

[0088] α ir =σ(W ir ·B ir (I ir )+b ir );

[0089] Where W rgb and W ir is the trainable weight matrix, b rgb and b ir is the bias term, and σ is the sigmoid activation function.

[0090] The features of all modalities are weighted by the corresponding attention weights and added together to form the fused multimodal features. The formula is as follows:

[0091] N=α rgb ·B rgb (I rgb )+α ir ·B ir (I ir );

[0092] In order to reduce the internal covariate shift and thus speed up the convergence of the model and improve the performance, an adaLN normalization layer is used, which receives the fused multimodal features N and the time step encoding t as input, and dynamically learns the bias and expansion parameters according to the input data:

[0093] scale=W scale ·(N+t)+b scale );

[0094] shift=W shift ·(N+t)+b shift );

[0095] The learned scale and shift are appended to the true mask F as extension parameters one-hot (x t ) features, which helps to dynamically adjust the degree of normalization and increase model stability and adaptability:

[0096] N'=scale⊙Conv(F one-hot (x t ))+shift;

[0097] To ensure the lightweight of the model, the normalized feature N' is passed through the lightweight fully connected layer interaction feature, and the residual connection is used to add the post-interaction feature to the pre-interaction feature to avoid catastrophic forgetting of the model.

[0098] 2.4. Construct the loss function used in the training model to minimize the true inverse diffusion distribution Q(x t-1 |x t ) and the inverse diffusion distribution P learned by the model θ (x t-1 |x t )'s relative entropy D KL (Q(x t-1 |x t )∥P θ (x t-1 |x t )). Due to P θ (x t-1 |x t ) where the time step t is known, so only the initial sample x0 needs to be predicted. The present invention uses the cross entropy loss function for prediction, and its formula is as follows:

[0099]

[0100] Among them, F one-hot (x0) is the predicted mask, y is the true mask, and C is the number of predicted categories.

[0101] 3. Model training phase

[0102] 3.1. Divide the multimodal remote sensing semantic segmentation dataset D obtained in step 1 into a training set T and a validation set V.

[0103] 3.2. The multimodal data I in the training set T rgb and I ir Input into the model designed in step 2, and use batch stochastic gradient descent method to train the model. The maximum time step is preset to 1000, and the time step parameter t is randomly sampled in the discrete interval [1,...,1000] and in the real probability matrix F one-hot Adding noise at time step t to (M) yields the distribution Use the Gumbel-Softmax reparameterization technique to randomly sample x from t . Multimodal Data I rgb and I ir After multimodal feature extraction network B rgb and B ir Encoding, with x t Together with the time step t, it is used as the input of the inverse diffusion distribution prediction network U to obtain the predicted x0, and the cross entropy loss is constructed with the real x0 as the derivative signal. During the training phase, the validation set V is used to verify the model training effect. That is, when the model obtains a good semantic segmentation result on the validation set V and the accuracy cannot be improved with the training process, the training is stopped;

[0104] 3.3. Finally, the training is completed and the model is obtained.

[0105] 4. Model testing phase

[0106] The input data is the registered multimodal data I rgb and I ir , after passing through the multimodal feature extraction network B rgb and B ir Encoding obtains multimodal image features, from which all elements are The probability matrix of x is randomly sampled using the Gumbel-Softmax reparameterization technique t ,To ensure the inference efficiency, the time step t is initialized to 100. Multimodal features, x t The predicted x0 is obtained by using t as the input of the inverse diffusion distribution prediction network U, and the final category probability matrix is ​​obtained by Softmax processing. The predicted x0 in this step can be used as the distribution parameter sampling x in the next step. t-1 , and repeat this 100 times to get the final result x0.

[0107] Therefore, the multimodal remote sensing image semantic segmentation method based on the discrete diffusion model proposed in the present invention can effectively alleviate the performance loss caused by noise, modal inconsistency and distribution deviation while maintaining high-resolution segmentation results, and demonstrates excellent cross-modal fusion capabilities and fine pixel-level classification performance.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal remote sensing image semantic segmentation method based on discrete diffusion model, characterized in that: Here are the steps: S1. Collect visible light and infrared paired images. Paired images between different modalities can be basically registered. S2, adaptive fusion of multimodal graphic features; S3, using discrete diffusion model to perform semantic segmentation operation; S4. Use batch stochastic gradient descent method to train the model.

2. According to claim 1, a multimodal remote sensing image semantic segmentation method based on discrete diffusion model is characterized in that: In S1, the RANSAC-Flow deep registration algorithm is used to perform high-precision registration of image pairs of different modalities, and the Labelme annotation software is used to perform semantic segmentation mask annotation on each image pair.

3. The multimodal remote sensing image semantic segmentation method based on discrete diffusion model according to claim 1, characterized in that: In S2, the multimodal graphic feature adaptive fusion specifically includes the following steps: S21, using a feature encoder with non-shared parameters to map the visible light and infrared modality images into homogeneous spaces respectively, and extract homogeneous features of each modality; S22, use a fully connected layer with non-shared parameters to extract the attention weights of each modality isomorphic feature; S23, weighting the modal features, and adding the weighted modal features to obtain multi-modal features after adaptive fusion.

4. The multimodal remote sensing image semantic segmentation method based on discrete diffusion model according to claim 3, characterized in that: In S21, the feature encoder includes a visible light modality image feature extraction network B rgb and infrared modality image feature extraction network B ir ; The visible light modality image feature extraction network B rgb The encoded image feature is B rgb (I rgb ), the infrared modality image feature extraction network B ir The encoded image feature is B ir (I ir ).

5. The multimodal remote sensing image semantic segmentation method based on discrete diffusion model according to claim 3, characterized in that: In S22, the dynamic attention mechanism is used to weight the image features of each modality, and the fully connected layer is used to extract the attention weights of the visible light modality features and the infrared modality features. The formula is as follows: a rgb =σ(W rgb ·B rgb (I rgb )+b rgb ); a ir =σ(W ir ·B ir (I ir )+b ir ); Among them, α rgb is the attention weight of the visible light modality feature, α ir is the attention weight of infrared modality features, W rgb and W ir is the weight matrix, b rgb and b ir is the bias term, and σ is the sigmoid activation function.

6. The multimodal remote sensing image semantic segmentation method based on discrete diffusion model according to claim 3, characterized in that: In S23, the modal features are weighted by the corresponding attention weights and added together to obtain the multimodal features. The formula is as follows: N=a rgb ·B rgb (I rgb )+a ir ·B ir (I ir ); Among them, N is the multimodal feature.

7. The method for semantic segmentation of multimodal remote sensing images based on discrete diffusion model according to claim 6, characterized in that: Adopting the adaLN normalization method, the fused multimodal feature N and time step encoding t are taken as input. According to the input data, the bias and expansion parameters are dynamically learned. The formula is as follows: scale=W scale ·(N+t)+b scale ); shift=W shift ·(N+t)+b shift ); Among them, scale is the deviation parameter, W scale The weight parameter of the biased fully connected layer, b scale is the bias parameter of the biased fully connected layer, shift is the expansion parameter, and W shift is the weight parameter of the expanded fully connected layer, b shift is the bias parameter for the extended fully connected layer; Append scale and shift to the true mask F one-hot (x t ), the formula is as follows: N'=scale⊙Conv(F one-hot (x t ))+shift; Among them, N' is the normalized modal feature.

8. The method for semantic segmentation of multimodal remote sensing images based on discrete diffusion model according to claim 1, characterized in that: In S3, the specific steps of semantic segmentation operation are as follows: S31, using the state transfer matrix to add random noise to the true probability distribution matrix, to obtain a probability matrix in which all categories have equal probabilities; S32. The true distribution of inverse diffusion of discrete diffusion model is derived through Bayesian formula; S33. Take the fused multimodal visual features as conditions and the diffused true probability matrix as input to predict the undiffused true probability matrix.

9. The method for semantic segmentation of multimodal remote sensing images based on discrete diffusion model according to claim 8, characterized in that: In S31, the true probability distribution matrix is ​​obtained by one-hot encoding the true category mask M, and the formula is as follows: P=F one-hot (M); Among them, P is the true probability distribution matrix, F one-hot (.) is the one-hot encoding operation; Add T time step noise to the real category mask M to get T-1 hidden variables x1,...,x T , where the intermediate latent variable x t By x t-1 The formula is as follows: x t ~Q(x t |x t-1 )=Cat(x t ;P t =F one-hot (x t-1 )·T t ); Among them, x t is the intermediate hidden variable, x t-1 For x t The previous step hidden variable, F one-hot (x t-1 ) is x t-1 The one-hot encoded one-hot label, T t is the state transfer matrix, P t is the probability matrix, Q(.|.) is the diffusion process, and Cat(.) is the sampling distribution; The state transfer matrix T t Its dimension is related to the number of real categories that need to be distinguished. The sum of any row or column elements is 1, and the diagonal elements are The off-diagonal elements are where β t is a hyperparameter, C is the number of categories; The Gumbel-Softmax reparameterization technique is used for random sampling. The category index formula for random sampling is as follows: z=one-hot(argmax i {g i +log p i }); Among them, p i is the probability of the i-th category, argmax i For the maximum value parameter operation, g i is a random factor, and its calculation formula is as follows: g i =-log(-log(u i )); Among them, u i is a randomly sampled value from uniform distribution U(0,1); Using the one-step noise addition technique, diffuse from the initial sample x0 to the time step t in one step. The formula is as follows: Among them, F one-hot (x0) is the real mask initialized after adding noise, For T0,...,T t The factorial of .

10. The method for semantic segmentation of multimodal remote sensing images based on discrete diffusion model according to claim 8, characterized in that: In S32, the Bayesian derivation formula is as follows: in, is the latent space variable before state transfer.

Images