Method for classifying missing modal remote sensing images based on cross-modal learnable query and diffusion reconstruction

Abstract

The present application relates to a kind of modal missing remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction, from remote sensing data take HSI image, LiDAR image and label image, HSI and LiDAR sample block are mapped into token sequence, the cross-modal similarity of two modalities is calculated, and according to threshold value, it is divided into low similarity area and high similarity area.Low similarity area image reconstruction and high similarity area reconstruction are carried out.In the second stage modal missing inference, LiDAR feature is completed using HSI feature and learnable query matrix, and HSI feature is completed using LiDAR feature and learnable query matrix, finally, available modal feature, completed modal feature and shared encoding feature are fused, and the ground object category is output through Softmax classifier.The present application can effectively recover missing modal feature under the condition of modal missing, improve the robustness and accuracy of HSI and LiDAR remote sensing image classification, and can be used for remote sensing ground object classification, land cover mapping and multi-source remote sensing intelligent interpretation task.

Description

Technical Field

[0001] This invention belongs to the fields of remote sensing image processing, computer vision, and multimodal deep learning, and relates to a method for classifying modal-deficient remote sensing images based on cross-modal learnable queries and diffusion reconstruction. Specifically, it relates to a method for classifying modal-deficient remote sensing images using hyperspectral (HSI) and LiDAR images. This method learns a shared Transformer encoder, a cross-modal similarity partitioning mechanism, a cross-modal learnable query matrix, and a diffusion image reconstruction module through self-supervised training. Even when only HSI or only a single LiDAR modality is available during the testing phase, it can still complete the missing modal features and complete the land cover classification. Background Technology

[0002] With the development of aerospace remote sensing, airborne lidar, and multi-source sensor technologies, the same surface area can be obtained from various remote sensing data by different sensors. Hyperspectral images (HSI) can record spectral reflectance information across continuous bands, possessing rich spectral identification capabilities; LiDAR images can reflect ground feature height, surface structure, and three-dimensional geometric information, exhibiting strong spatial structure representation capabilities. For complex scenes such as cities, roads, vegetation, and buildings, relying solely on a single modality often makes it difficult to simultaneously obtain spectral and height structure information. Therefore, joint classification of HSI and LiDAR has become an important direction in the intelligent interpretation of remote sensing images.

[0003] Existing HSI and LiDAR fusion classification methods typically include traditional feature fusion methods and deep learning fusion methods. Traditional methods generally extract spectral features, texture features, height features, or principal component features first, and then use classifiers such as support vector machines or random forests for classification. Deep learning methods, on the other hand, automatically extract deep features through convolutional neural networks, graph convolutional networks, attention mechanisms, or Transformer structures, and achieve multimodal information complementarity through feature concatenation, weighted fusion, gated fusion, or cross-attention.

[0004] However, existing multimodal remote sensing classification methods typically assume that complete HSI and LiDAR data can be obtained during both the training and testing phases. In practical remote sensing applications, different sensors may have varying acquisition times, coverage areas, spatial resolutions, and data quality. Remote sensing data is also susceptible to factors such as sensor malfunctions, cloud and fog obstructions, flight mission limitations, registration errors, data corruption, or transmission anomalies, which can lead to a particular modality being unavailable during the testing phase. For joint HSI and LiDAR classification tasks, situations may arise where only HSI is available while LiDAR is missing, or vice versa.

[0005] Traditional multimodal models typically exhibit significant performance degradation when modalities are missing. If the missing modalities are discarded directly, the model degenerates into a single-modal classification model, unable to utilize the complementary information originally contained in the missing modalities. If zero-padding, mean-padding, or random-padding is used to replace the missing modalities, the padding results lack true physical meaning and are prone to causing the network to learn incorrect distributions. If additional training is performed to generate a network to recover the missing modalities, it usually requires complex structures and strong supervision signals, making training difficult.

[0006] On the other hand, not all regions between HSI and LiDAR exhibit the same cross-modal relationships. Some tokens show high similarity in both modalities, indicating stable shared semantics; others show low similarity, suggesting significant modal differences and complementary information. Existing methods mostly employ a uniform fusion strategy for all regions without differentiating based on similarity levels, which can easily lead to insufficient learning of shared information and underutilization of complementary information.

[0007] Self-supervised learning, masked autoencoders, and diffusion models offer novel approaches to addressing the modality missing problem. If similarity masks can be used to distinguish high-similarity and low-similarity regions during the bimodal full-training phase, and feature-level and image-level reconstructions can be performed separately, the model can learn cross-modal consistency, complementarity, and the ability to complete missing modalities. Therefore, a modality missing remote sensing image classification method for HSI and LiDAR data is needed to improve classification robustness under single-modal input conditions. Summary of the Invention

[0008] Technical problems to be solved To overcome the shortcomings of existing technologies, and addressing the problems of decreased classification accuracy, difficulty in recovering complementary information of missing modes, and insufficient utilization of cross-modal local differences in existing HSI and LiDAR multimodal remote sensing classification methods, this invention proposes a modality-missing remote sensing image classification method based on cross-modal learnable queries and diffusion reconstruction. This method comprises two processes: a first-stage self-supervised training and a second-stage modality-missing inference classification.

[0009] The present invention achieves the above objectives by: reading registered HSI images, LiDAR images, and ground truth images with only partial category labels from remote sensing data; normalizing, boundary padding, size alignment, and sample block cropping for both modalities; mapping the sample blocks to token sequences and inputting them into a shared Transformer encoder; calculating the cross-modal similarity between corresponding tokens in HSI and LiDAR, and dividing them into low-similarity and high-similarity regions based on a threshold; performing image-level reconstruction of low-similarity regions by fusing complementary information through cross-attention and combining it with a diffusion model; performing feature-level reconstruction of high-similarity regions through a cross-modal learnable query matrix; and finally, in the modality missing inference stage, using the trained learnable query matrix to complete the missing modal features, and fusing available features, completed features, and shared features to complete the classification.

[0010] Technical solution A modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction, characterized by including a first-stage self-supervised training network and a second-stage modality-deficient inference network; The first-stage self-supervised training network includes an HSI dimension mapping module, a LiDAR dimension mapping module, a PatchEmbed layer, a shared Transformer encoder, a cross-modal similarity analysis module, a low-similarity diffusion image reconstruction branch, and a high-similarity cross-modal feature reconstruction branch. The second-stage modality-missing inference network: When only HSI modality input is available and LiDAR modality is missing, the HSI sample block is input into the trained encoder to obtain modality-usable features. and utilize learnable query vectors The LiDAR feature, which completes the modality, is obtained through cross-attention. The two feature components are then fused together using the cross-attention mechanism to obtain shared features. Finally, the three feature components are fused and fed into a classifier to obtain the classification result. When only the LiDAR modality is input and the HSI modality is missing, the LiDAR sample block is input into the trained encoder to obtain the classification result. and utilize learnable query vectors The HSI features are completed through cross-attention, and the shared features are obtained by combining the two feature sets through the cross-attention mechanism. Finally, the three feature sets are fused and fed into the classifier to obtain the classification result. The classification method steps are as follows: Step 1: Read in the registered HSI image, LiDAR image, and ground truth map with only partial area classification from the remote sensing data; preprocess the two modalities of HSI image and LiDAR image to obtain HSI sample blocks and LiDAR sample blocks; Step 2: After mapping the two sample blocks to token sequences, encode them using a shared Transformer encoder to obtain the encoded HSI sequence features and LiDAR sequence features; Step 3: Calculate the cross-modal similarity between the tokens corresponding to HSI sequence features and LiDAR sequence features, and divide them into low similarity regions and high similarity regions according to the similarity threshold; Step 4: Perform image-level reconstruction and feature-level reconstruction: Step 4.1 Low-similarity regions are reconstructed at the image level by fusing complementary information through cross-attention and combining it with a diffusion model; Step 4.2 High similarity regions are reconstructed at the feature level using a cross-modal learnable query matrix; Set a self-supervised training loss function for training the first-stage network. Step 5: Train the network in two stages. In the first stage, self-supervised training is performed using a self-supervised training loss function, and the trained learnable query vectors are retained. The second stage involves training based on the encoder parameters obtained in the first stage. The input single-modal image is processed by the encoder to obtain usable modal features. The missing modal features are then obtained by using the trained learnable vectors to complete the missing modal features. Finally, the features are fused together to obtain shared features. Step 6: Concatenate and fuse the available modal features, the completed modal features, and the shared features to obtain the fused features. And input the data into the Softmax classifier to obtain the probabilities of each class:

[0011] The land cover category with the highest probability is used as the classification result of the remote sensing image; Let W be the probability of each category, and W be the weight matrix of the classification layer. This is the bias vector for the classification layer.

[0012] The preprocessing in step 1 specifically includes normalization, boundary padding, size alignment, and sample block cropping: cropping an N×N neighborhood sample block centered on the labeled pixel, and normalization using the following formula:

[0013] in, Represents any pixel value in the image. and These represent the minimum and maximum values ​​in the current modal image, respectively.

[0014] Step 2 specifically involves mapping the HSI sample blocks and LiDAR sample blocks into token sequences via a dimension mapping module and an embedding layer, respectively. Two-dimensional sine and cosine position codes and time step codes t are added to the token sequence and then input into the Transformer encoder to obtain the encoded HSI sequence features and LiDAR sequence features; where: D is the embedding dimension; The dimensions of the image.

[0015] Step 3 specifically involves: calculating the cosine similarity between corresponding token positions for the encoded HSI sequence features and LiDAR sequence features to obtain a similarity vector. : , ; in: Let i be the similarity score corresponding to the i-th token. This represents the feature of the i-th token in the HSI mode. This represents the feature of the i-th token in the LiDAR mode; It represents a very small positive number to avoid the denominator being zero; T represents the vector transpose operation. The token is divided into low-similarity and high-similarity regions based on a similarity threshold τ. The low-similarity mask and the high-similarity mask are as follows:

[0016] in: For low similarity mask, High similarity mask, where 1[·] represents an indicator function; The low-similarity regions are used for cross-modal complementary information fusion and image-level reconstruction; The high similarity regions are used for cross-modal learnable queries and feature-level reconstruction.

[0017] In step 4.1, during the low-similarity diffusion image reconstruction branch, the low-similarity tokens are masked using a MAE method. The current modality's low-similarity token is used as the query Q, and the other modality's high-similarity token is used as the key K and value Vt. Complementary information is fused through cross-attention, and the fused low-similarity complementary features are input into the diffusion reconstruction module to reconstruct the HSI and LiDAR images. The cross-attention formula is:

[0018] The diffusion model time step of the diffusion reconstruction module is set to T=1000. During training, the time step t∈{0,1,…,999} is randomly sampled, and noise is added to the original image to obtain the result. The network predicts the denoising result based on t and cross-modal fusion features; The noise is added using forward noise generation, and the process is as follows:

[0019] Equivalent land, ; Reconstruct the image by predicting using the decoder; in: The conditional probability distribution representing the forward noise addition process in the diffusion model. Indicates Gaussian distribution, This represents the noisy image corresponding to the t-th diffusion time step. Represents the original input image, The signal retention coefficient at step t represents the signal retention coefficient at step t. Indicates the cumulative signal retention rate; The decoder embedding dimension =128, decoder depth =3, number of decoder attention heads =8.

[0020] Step 4.2 High Similarity Cross-Modal Feature Reconstruction Branch: In the high-similarity cross-modal feature reconstruction branch, a learnable query matrix from HSI to LiDAR is set. and the learnable query matrix from LiDAR to HSI , LiDAR features were recovered using HSI-related features as a condition. HSI features are recovered using LiDAR-related features as conditions, and high-similarity cross-modal features are reconstructed through a cross-attention mechanism; When reconstructing the high-similarity cross-modal feature branch, the feature dimension of the learnable query matrix is ​​128.

[0021] In the first stage of self-supervised training network, both the HSI dimension mapping module and the LiDAR dimension mapping module consist of two layers of 1×1 convolution, BatchNorm and ReLU, with 32 hidden channels.

[0022] The self-supervised training loss function includes low-similarity region image reconstruction loss, visible region image preservation loss, high-similarity region feature reconstruction loss, and classification assistance loss, wherein:

[0023] The feature reconstruction loss is: ; The total loss function is: , in , This represents the image reconstruction loss of the masked region. This indicates the loss of image preservation in the visible region. This represents the feature reconstruction loss.

[0024] The similarity threshold τ = 0.5.

[0025] When reconstructing the high-similarity cross-modal feature branch, the feature dimension of the learnable query matrix is ​​1024.

[0026] Beneficial effects This invention proposes a modality-missing remote sensing image classification method based on cross-modal learnable queries and diffusion reconstruction, primarily addressing the issue of decreased classification accuracy in joint classification tasks of hyperspectral (HSI) and LiDAR images during the testing phase when a single modality is missing. The scheme involves: reading registered HSI and LiDAR images and their corresponding ground truth labels from remote sensing data; normalizing, filling boundaries, aligning sizes, and cropping sample blocks; constructing a first-stage self-supervised training network; mapping HSI and LiDAR sample blocks to token sequences and inputting them into a shared Transformer encoder; calculating the cross-modal similarity between the two modal tokens; and dividing the tokens into low-similarity and high-similarity regions based on a similarity threshold. For low-similarity regions, a cross-attention mechanism is used to fuse complementary information from another modality, combined with a diffusion model for image-level reconstruction; for high-similarity regions, a cross-modal learnable query matrix is ​​used for feature-level reconstruction. The network is jointly optimized using image reconstruction loss, feature reconstruction loss, and classification assistance loss. In the second-stage modality missing inference, when a LiDAR modality is missing, HSI features and a learnable query matrix are used to complete the LiDAR features; when an HSI modality is missing, LiDAR features and a learnable query matrix are used to complete the HSI features. Finally, the available modality features, completed modality features, and shared encoded features are fused, and the land cover category is output through a Softmax classifier. This invention can effectively recover missing modality features under modality missing conditions, improve the robustness and accuracy of HSI and LiDAR remote sensing image classification, and can be used for remote sensing land cover classification, land cover mapping, and multi-source remote sensing intelligent interpretation tasks.

[0027] Compared with the prior art, the present invention has the following advantages: 1. This invention constructs different masks and reconstruction strategies based on the similarity between HSI and LiDAR tokens, which can learn cross-modal shared information and complementary information respectively, thereby improving the ability to model cross-modal relationships.

[0028] 2. This invention utilizes high similarity regions to train a cross-modal learnable query matrix, enabling it to complete missing modal features based on a single available modality during the inference stage, thereby improving classification robustness under modality missing conditions.

[0029] 3. This invention employs dual self-supervised constraints of image-level reconstruction and feature-level reconstruction, reducing reliance on a large number of manual labels and enabling it to serve HSI and LiDAR joint classification tasks. Attached Figure Description

[0030] Figure 1 Flowchart of the first stage of bimodal self-supervised training in this invention Figure 2 Flowchart of inference classification under the second-stage modality missing condition of this invention Figure 3 Rendering of classification results from an embodiment of the present invention a) Rendered images of bimodal input classification results for HSI and LiDAR images from the Trento dataset; b) Single-modal input classification results for HSI images from the Trento dataset. Detailed Implementation The present invention will now be further described in conjunction with the embodiments and accompanying drawings: The approach of this invention is as follows: Registered HSI images, LiDAR images, and ground truth images with only partial category labels are read from remote sensing data; normalization, boundary padding, size alignment, and sample block cropping are performed on both modalities; the sample blocks are mapped to token sequences and input into a shared Transformer encoder; cross-modal similarity between corresponding tokens of HSI and LiDAR is calculated, and regions are divided into low-similarity and high-similarity regions based on a threshold; low-similarity regions undergo image-level reconstruction by fusing complementary information through cross-attention and combining it with a diffusion model; high-similarity regions undergo feature-level reconstruction using a cross-modal learnable query matrix; finally, in the modality missing inference stage, the trained learnable query matrix is ​​used to complete the missing modal features, and the available features, completed features, and shared features are fused to complete the classification.

[0031] The specific steps of this invention include the following: I. Reading in HSI and LiDAR remote sensing data The registered hyperspectral images (HSI and LiDAR) and corresponding ground truth labels with only partial category annotations are read from remote sensing data; training samples, validation samples, and test samples are created according to the pixel locations with category annotations in the label images. (1) Image preprocessing to create training samples The HSI and LiDAR images undergo min-max normalization, pixel values ​​are mapped to [0,1], boundary padding is applied, size alignment is performed, and sample blocks are cropped. An N×N neighborhood sample block is cropped centered on the labeled pixel; in one embodiment, an 11×11 neighborhood block is cropped centered on the labeled pixel as the input sample. When the center pixel is close to the image boundary, the image boundary is padded to ensure all sample blocks are of the same size. =1. The normalization formula is:

[0032] in, Represents any pixel value in the image. and These represent the minimum and maximum values ​​in the current modal image, respectively.

[0033] II. Constructing the first-stage self-supervised training network. The self-supervised training network includes an HSI dimension mapping module, a LiDAR dimension mapping module, a PatchEmbed embedding layer, a shared Transformer encoder, a cross-modal similarity analysis module, a low-similarity diffusion image reconstruction branch, and a high-similarity cross-modal feature reconstruction branch. Both the HSI and LiDAR dimension mapping modules consist of two 1×1 convolutional layers, BatchNorm, and ReLU, with 32 hidden channels.

[0034] In this example: Patch_size is 256; Transformer encoder The value is 128, and the depth is 24. It is 8. The value is 2.

[0035] III. Token Sequence Encoding. HSI and LiDAR sample blocks are mapped into token sequences via the dimension mapping module and the PatchEmbed layer, respectively. and ,in N is the image size, and D is the embedding dimension. In one embodiment, =121, D=128. The token sequence is fed into a Transformer encoder after adding two-dimensional sine and cosine position codes and time step codes t; the Transformer encoder depth is 4, and the number of attention heads is... =8, single attention head dimension d=16, MLP scaling ratio The value is 2, and the EPS of LayerNorm is 1×10. -6 .

[0036] IV. Cross-modal similarity calculation. The cosine similarity between corresponding tokens is calculated for the encoded HSI sequence features and LiDAR sequence features, resulting in a similarity vector S, where the similarity of the i-th token is: , ; in: Let i be the similarity score corresponding to the i-th token. This represents the feature of the i-th token in the HSI mode. This represents the feature of the i-th token in the LiDAR mode; This represents a very small positive number to avoid the denominator being zero; T represents the vector transpose operation. V. Construct a similarity-based masking module. Similarity-based masking partitioning: Normalize the HSI token features and LiDAR token features, and calculate the cosine similarity of tokens at corresponding positions. The similarity threshold τ is set to 0.5, and the masking ratio is set to 0.6. Tokens with a similarity of ≤τ are marked as low-similarity tokens. The token of >τ is marked as a highly similar token; The token is divided into low-similarity and high-similarity regions based on a similarity threshold τ, where τ = 0.5 in one embodiment. The low-similarity mask and the high-similarity mask are as follows:

[0037] Where 1[·] represents the indicator function; low similarity regions are used for cross-modal complementary information fusion and image-level reconstruction, while high similarity regions are used for cross-modal learnable queries and feature-level reconstruction; VI. Low-Similarity Diffusion Image Reconstruction: After masking low-similarity regions, the low-similarity tokens of the current modality and another modality are input into the cross-attention module to obtain features with fused complementary information. The diffusion model time step is set to 1000, and time step t is randomly sampled during training. Noise is added to the original image to obtain the final image. The network predicts the denoising results based on t and cross-modal fusion features, and finally reconstructs the HSI image and LiDAR image.

[0038] In the low-similarity diffusion image reconstruction branch, low-similarity tokens are masked using a MAE method, with a masking ratio of... =0.6, the number of tokens to retain The mask token count is 73. Using the current modality's low-similarity token as the query Q, and another modality's high-similarity token as the key K and value V, complementary information is fused through cross-attention. The cross-attention formula is:

[0039] Where d=16 represents the feature dimension of a single attention head. The fused low-similarity complementary features are input into the diffusion reconstruction module for image-level reconstruction, with a diffusion step count T=1000 and time steps t∈{0,1,...,999}. The forward noise addition process is as follows:

[0040] Equivalent land, The image is reconstructed by predicting the decoder, where the decoder embedding dimension is... =128, decoder depth =3, number of decoder attention heads =8; 7. High-similarity cross-modal feature reconstruction. Set up a learnable query matrix from HSI to LiDAR. and the learnable query matrix from LiDAR to HSI The feature dimension of the learnable query matrix is ​​1024; another modality feature is reconstructed through a cross-attention mechanism; Set up a learnable query matrix in two directions. and At that time, its embedding dimension is 1024. LiDAR features were recovered using HSI-related features as a condition. HSI features are recovered conditioned on LiDAR-related features. The cross-attention module uses eight attention heads to output the reconstructed cross-modal features, and calculates the mean squared error loss with the real modality-encoded features.

[0041] 8. Constructing the self-supervised training loss function: The loss for reconstructing the masked region image, the loss for preserving the visible region image, and the loss for reconstructing the learnable query matrix features all use mean squared error, while the classification auxiliary loss uses cross-entropy loss. The training optimizer uses Adam. 1×10 -4 .

[0042] The loss function includes low-similarity region image reconstruction loss, visible region image preservation loss, high-similarity region feature reconstruction loss, and classification assistance loss, wherein:

[0043] The feature reconstruction loss is: ; The total loss function is: , in , This represents the image reconstruction loss of the masked region. This indicates the loss of image preservation in the visible region. The feature reconstruction loss is represented by ; in one embodiment =1.0. Training uses the Adam optimizer with weight decay. =1×10 -4 9. Construct the second-stage modality-missing inference network. When only HSI modality input is available and LiDAR modality is missing, input the HSI sample block into the trained encoder to obtain... and utilize Completed LiDAR features are obtained through cross-attention; when only LiDAR modal input is available and HSI modal input is missing, LiDAR sample blocks are input into the trained encoder to obtain... and utilize Complete HSI features are obtained through cross-attention.

[0044] The second-stage classification network loads the encoder and learnable query vectors obtained from the first-stage training. The classification head input is a concatenation of three features: available modal features, complete modal features, and shared features. The concatenated dimension is 3×1024, and the number of output nodes is the number of land cover categories, C.

[0045] 10. Classification under Modality Missing Conditions. When only HSI modality input is available and LiDAR modality is missing, the HSI sample block is input into the trained encoder to obtain... (Modal available features), and utilize learnable query vectors Completed LiDAR features (complete modal features) are obtained through cross-attention. Shared features are then obtained by combining the two feature sets through a cross-attention mechanism. Finally, the three feature sets are fused and fed into a classifier to obtain the classification result. When only LiDAR modal input is available and HSI modal input is missing, LiDAR sample blocks are input into the trained encoder to obtain the classification result. and utilize learnable query vectors The complete HSI features are obtained through cross-attention, and the shared features are obtained by combining the two features through the cross-attention mechanism. Finally, the available features, complete features and shared features are input into the classification head to obtain the class probability of each test pixel.

[0046] Output the classification results. The available modality features, completed modality features, and shared coding features are fused to obtain the fused features. And input the data into the Softmax classifier to obtain the probabilities of each class:

[0047] The land cover category with the highest probability is output as the remote sensing image classification result.

[0048] Experimental Validation. This invention can be validated on the Trento dataset. The experimental data includes two modalities: HSI and LiDAR, and tests are conducted for three scenarios: complete bimodality, HSI available but LiDAR missing, and LiDAR available but HSI missing. Experimental results show that, compared to direct single-modal classification and zero-filling of missing modalities, this invention can maintain more stable classification performance under modality missing conditions.

[0049] Simulation conditions: The hardware platform is: RTX-3090-24GB-GPU, 128GB RAM.

[0050] The software platform is Python, using the PyTorch deep learning framework.

[0051] Simulation content and results: The simulation experiments in this invention use hyperspectral (HSI) and LiDAR images from the Trento dataset. This dataset consists of HSI and LiDAR images registered for the same region, along with corresponding ground truth label images. HSI images provide spectral information about ground features, while LiDAR images provide height and spatial structure information. The labeled classification scenes include six categories: apple trees, buildings, ground, forests, vineyards, and roads.

[0052] The experiment was conducted under three input conditions: the first was a complete input of both HSI and LiDAR modalities; the second was an input of only HSI modality with missing LiDAR modality; and the third was an input of only LiDAR modality with missing HSI modality. After training, under the condition of a complete input of both HSI and LiDAR modalities, the overall classification accuracy of the method of the present invention was 99.58%, the average classification accuracy was 99.34%, and the Kappa coefficient was 0.9944; under the condition of an input of only HSI modality with missing LiDAR modality, the overall classification accuracy of the method of the present invention was 98.23%, the average classification accuracy was 97.17%, and the Kappa coefficient was 0.9763; and under the condition of an input of only LiDAR modality with missing HSI modality, the overall classification accuracy of the method of the present invention was 93.04%, the average classification accuracy was 92.36%, and the Kappa coefficient was 0.9078.

Claims

1. A modality-deficient remote sensing image classification method based on cross-modal learnable queries and diffusion reconstruction, characterized in that... This includes a first-stage self-supervised training network and a second-stage modality-deficient inference network; The first-stage self-supervised training network includes an HSI dimension mapping module, a LiDAR dimension mapping module, a PatchEmbed layer, a shared Transformer encoder, a cross-modal similarity analysis module, a low-similarity diffusion image reconstruction branch, a high-similarity cross-modal feature reconstruction branch, and a classification auxiliary head. The second-stage modality-missing inference network: When only HSI modality input is available and LiDAR modality is missing, the HSI sample block is input into the trained encoder to obtain modality-usable features. and utilize learnable query vectors The LiDAR feature, which completes the modality, is obtained through cross-attention. The two feature components are then fused together using the cross-attention mechanism to obtain shared features. Finally, the three feature components are fused and fed into a classifier to obtain the classification result. When only the LiDAR modality is input and the HSI modality is missing, the LiDAR sample block is input into the trained encoder to obtain the classification result. and utilize learnable query vectors The HSI features are completed through cross-attention, and the shared features are obtained by combining the two feature sets through the cross-attention mechanism. Finally, the three feature sets are fused and fed into the classifier to obtain the classification result. The classification method steps are as follows: Step 1: Read in the registered HSI image, LiDAR image, and ground truth map with only partial area classification from the remote sensing data; preprocess the two modalities of HSI image and LiDAR image to obtain HSI sample blocks and LiDAR sample blocks; Step 2: After mapping the two sample blocks to token sequences, encode them using a shared Transformer encoder to obtain the encoded HSI sequence features and LiDAR sequence features; Step 3: Calculate the cross-modal similarity between the tokens corresponding to HSI sequence features and LiDAR sequence features, and divide them into low similarity regions and high similarity regions according to the similarity threshold; Step 4: Perform image-level reconstruction and feature-level reconstruction: Step 4.1 Low-similarity regions are reconstructed at the image level by fusing complementary information through cross-attention and combining it with a diffusion model; Step 4.2 High similarity regions are reconstructed at the feature level using a cross-modal learnable query matrix; Set a self-supervised training loss function for training the first-stage network. Step 5: Train the network in two stages. In the first stage, self-supervised training is performed using a self-supervised training loss function, and the trained learnable query vectors are retained. The second stage involves training based on the encoder parameters obtained after the first stage of training. The input single-modal image is processed by the encoder to obtain usable modal features. The missing modal features are then obtained by completing the missing modal features using the trained learnable vectors and fused to obtain shared features. Step 6: Concatenate and fuse the available modal features, the completed modal features, and the shared features to obtain the fused features. And input the data into the Softmax classifier to obtain the probabilities of each class: The land cover category with the highest probability is used as the classification result of the remote sensing image; Let W be the probability of each category, and W be the weight matrix of the classification layer. This is the bias vector for the classification layer.

2. The modality-deficient remote sensing image classification method based on cross-modal learnable queries and diffusion reconstruction according to claim 1, characterized in that: The preprocessing in step 1 specifically includes normalization, boundary padding, size alignment, and sample block cropping: cropping an N×N neighborhood sample block centered on the labeled pixel, and normalization using the following formula: in, Represents any pixel value in the image. and These represent the minimum and maximum values ​​in the current modal image, respectively.

3. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: Step 2 specifically involves mapping the HSI sample blocks and LiDAR sample blocks into token sequences via a dimension mapping module and an embedding layer, respectively. Two-dimensional sine and cosine position codes and time step codes t are added to the token sequence and then input into the Transformer encoder to obtain the encoded HSI sequence features and LiDAR sequence features; where: D is the embedding dimension; The dimensions of the image.

4. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: Step 3 specifically involves: calculating the cosine similarity between corresponding token positions for the encoded HSI sequence features and LiDAR sequence features to obtain a similarity vector. : ， ； in: Let i be the similarity score corresponding to the i-th token. This represents the feature of the i-th token in the HSI mode. This represents the feature of the i-th token in the LiDAR mode; It represents a very small positive number to avoid the denominator being zero; T represents the vector transpose operation. The token is divided into low-similarity and high-similarity regions based on a similarity threshold τ. The low-similarity mask and the high-similarity mask are as follows: in: For low similarity mask, High similarity mask, where 1[·] represents an indicator function; The low-similarity regions are used for cross-modal complementary information fusion and image-level reconstruction; The high similarity regions are used for cross-modal learnable queries and feature-level reconstruction.

5. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: In step 4.1, during the low-similarity diffusion image reconstruction branch, the low-similarity tokens are masked using a MAE method. The current modality's low-similarity token is used as the query Q, and the other modality's high-similarity token is used as the key K and value Vt. Complementary information is fused through cross-attention, and the fused low-similarity complementary features are input into the diffusion reconstruction module to reconstruct the HSI and LiDAR images. The cross-attention formula is: The diffusion model time step of the diffusion reconstruction module is set to T=1000. During training, the time step t∈{0,1,…,999} is randomly sampled, and noise is added to the original image to obtain x_t. The network predicts the denoising result based on t and cross-modal fusion features. The noise is added using forward noise generation, and the process is as follows: Equivalent land, The image is reconstructed by predicting using a decoder. in: The conditional probability distribution representing the forward noise addition process in the diffusion model. Indicates Gaussian distribution, This represents the noisy image corresponding to the t-th diffusion time step. Represents the original input image, The signal retention coefficient at step t represents the signal retention coefficient at step t. Indicates the cumulative signal retention rate; The decoder embedding dimension =123, decoder depth =3, Number of decoder attention heads =8.

6. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: Step 4.2, the high similarity cross-modal feature reconstruction branch: In the high-similarity cross-modal feature reconstruction branch, a learnable query matrix from HSI to LiDAR is set. and the learnable query matrix from LiDAR to HSI , LiDAR features were recovered using HSI-related features as a condition. HSI features are recovered using LiDAR-related features as conditions, and high-similarity cross-modal features are reconstructed through a cross-attention mechanism; When reconstructing the high-similarity cross-modal feature branch, the feature dimension of the learnable query matrix is ​​1024.

7. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: In the first stage of self-supervised training network, both the HSI dimension mapping module and the LiDAR dimension mapping module consist of two layers of 1×1 convolution, BatchNorm and ReLU, with 32 hidden channels.

8. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: The self-supervised training loss function includes low-similarity region image reconstruction loss, visible region image preservation loss, high-similarity region feature reconstruction loss, and classification assistance loss, wherein: The feature reconstruction loss is: ; The total loss function is: , in , This represents the image reconstruction loss of the masked region. This indicates the loss of image preservation in the visible region. This represents the feature reconstruction loss.

9. The modality-deficient remote sensing image classification method based on cross-modal learnable query and diffusion reconstruction according to claim 1, characterized in that: The similarity threshold τ = 0.

5.

10. The modality-deficient remote sensing image classification method based on cross-modal learnable queries and diffusion reconstruction according to claim 6, characterized in that: When reconstructing the high-similarity cross-modal feature branch, the feature dimension of the learnable query matrix is ​​128.

Images