A Multimodal Remote Sensing Image Classification Method with Modality Missing

Abstract

The application is suitable for the technical field of remote sensing image processing, and provides a multi-modal remote sensing image classification method under modal loss, comprising: acquiring a multi-modal remote sensing image dataset; training a teacher network by using the multi-modal remote sensing image dataset; constructing a student network with the same network structure as the teacher network, and setting a modal loss module between a feature encoder module and a Fourier cross-fusion module of the student network, the modal loss module being used for randomly losing one of hyperspectral image features and laser radar image features output by the feature encoder module of the student network; distilling multi-modal information learned by the teacher network to the student network, and completing student network training under modal loss; and classifying hyperspectral images and laser radar images of a research area by using the trained student network, to obtain a classification result of the research area. The application can improve the classification precision of remote sensing images.

Description

Technical Field

[0001] This application belongs to the field of remote sensing image processing technology, and in particular relates to a multimodal remote sensing image classification method under modality loss. Background Technology

[0002] Land cover classification based on remote sensing imagery is one of the core technologies of Earth observation and has been widely applied in key areas such as urban planning, ecological environment monitoring, and land use analysis. In this context, single-modal data often fails to fully characterize complex land cover features and has limited classification accuracy. In contrast, fused multimodal data can comprehensively utilize multi-source remote sensing information, and its classification performance is usually better than that of single-modal data, providing richer criteria for accurately identifying land cover types.

[0003] In recent years, multimodal remote sensing image classification methods based on deep neural networks have made significant progress, typically outperforming traditional single-modal methods. However, these methods often rely on an idealized assumption: the input modalities used in the training and testing phases must be completely consistent. In practical engineering and application scenarios, factors such as sensor malfunctions, atmospheric conditions, imaging cycles, or data transmission constraints can lead to unstable or complete loss of high-quality data for a particular modality. Once the input modality deviates from the training preset, the model's classification performance will significantly decline, making it difficult to fully realize its technical advantages in practice.

[0004] Existing multimodal remote sensing image classification methods mainly target full-modal scenes, with relatively little research on incomplete modalities. Incomplete multimodal remote sensing image classification methods primarily include generative and distillation-based methods. Generative methods use illusion networks or generative adversarial strategies to complete the missing modalities, then combine this with traditional multimodal classifiers to identify land cover types. While generative methods are simple to implement, they suffer from errors in pixel-level data reconstruction, affecting the final model's classification performance. Distillation-based methods focus on feature-level knowledge transfer, effectively avoiding reconstruction errors by directly aligning cross-modal representations. However, these methods typically rely on the premise that the types of missing modalities are known and fixed in advance during training and testing. This contradicts the random and unpredictable nature of modal missingness in real-world remote sensing scenarios. If the actual missing modalities deviate from the preset values, the model's classification performance drops sharply, resulting in low remote sensing image classification accuracy and limiting the applicability of these methods in complex real-world environments. Summary of the Invention

[0005] This application provides a multimodal remote sensing image classification method under modality loss, which can solve the problem of low classification accuracy of remote sensing images.

[0006] This application provides a method for classifying multimodal remote sensing images under modality loss, including: Obtain a multimodal remote sensing image dataset; the multimodal remote sensing image dataset includes multiple samples, including hyperspectral images and the corresponding lidar images; A teacher network was trained using a multimodal remote sensing image dataset to obtain the trained teacher network. The teacher network includes a feature encoder module, a spatial enhancement module, a spectral enhancement module, a Fourier cross-fusion module, and an output module. The feature encoder module is used to extract features from hyperspectral images and lidar images and output hyperspectral image features and lidar image features. The spatial enhancement module is used to process the hyperspectral image features and lidar image features to obtain enhanced hyperspectral image features. The spectral enhancement module is used to process the hyperspectral image features, lidar image features, enhanced hyperspectral image features, and enhanced lidar image features to obtain fused features. The output module is used to process the fused features to obtain the predicted classification results of the samples. A student network with the same network structure as the teacher network is constructed, and a modality missing module is set between the feature encoder module and the Fourier cross-fusion module of the student network. The modality missing module is used to randomly missing one of the hyperspectral image features and lidar image features output by the feature encoder module of the student network. Distill the multimodal information learned by the teacher network into the student network to complete the training of the student network under the condition of modality missing, and obtain the trained student network; The trained student network is used to classify hyperspectral and lidar images of the study area to obtain the classification results of the study area.

[0007] Optionally, the modal missing mode is used to perform the following steps: The hyperspectral image features and lidar image features output by the feature encoder module are projected and mapped into a common space of the same dimension through a linear layer. Normalization processing was performed on the hyperspectral image features and the lidar image features after projection mapping, respectively. Calculate the cosine similarity between the normalized hyperspectral image features and the normalized lidar image features; If the cosine similarity is greater than or equal to a preset threshold, then one of the hyperspectral image features and lidar image features output by the feature encoder module will be randomly missing.

[0008] Optionally, the missing rate for random missing handling is one of 0.3, 0.5, 0.8, and 1.

[0009] Optional, loss function for knowledge distillation for: ; in, Indicates the uncertainty weight. , This represents the sigmoid function. and They represent the first The first sample and the first The variance of a sample, , Represents the total dimension. Indicating the first in the teacher network The first sample Dimensional features, Indicating the first in the teacher network The average of all dimensions of a sample. This indicates temperature hyperparameters. express The Middle The first sample and the first The degree of similarity between samples This represents the normalized Gram matrix of the teacher network. express The Middle The first sample and the first The degree of similarity between samples This represents the normalized Gram matrix of the student network. Represents a constant. ; ; ; ; ; ; in, The Gram matrix representing the teacher network. This represents the global scale factor of the teacher network. Denotes the Frobenius norm of a matrix. Represents a constant term. , express The Middle Line number Column elements, This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the teacher network in the spatial and channel dimensions. The Gram matrix representing the student network. express The Middle Line number Column elements, This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the student network in the spatial and channel dimensions.

[0010] Optionally, the loss function used when training the teacher network. for: ; in, Represents the classification loss function. This indicates the predicted classification result of the sample. Indicates the true class label of the sample. Indicates weight, This represents the cross-modal consistency loss of the teacher network.

[0011] Optionally, cross-modal consistency loss of the teacher network. for: ; in, This indicates the number of modes contained in the multimodal input. This represents the single-mode prediction output. Indicates the first Features corresponding to each mode Indicates the true class label of the sample. Indicates missing weights. , Indicates the missing rate of the hyperspectral image. This indicates the missing rate of the LiDAR image. Represents the balance factor. This represents the mean squared error loss function. This represents the predicted classification results for hyperspectral modes. This represents the predicted classification results of the lidar modes.

[0012] Optionally, the loss function used when training the knowledge-distilled student network. for: ; in, Represents the classification loss function. This indicates the predicted classification result of the sample. Indicates the true class label of the sample. Indicates weight, The loss function represents knowledge distillation. This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the teacher network in the spatial and channel dimensions. This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the student network in the spatial and channel dimensions. Indicates weight, This represents the cross-modal consistency loss of the student network.

[0013] Optionally, the spatial enhancement module calculates the enhanced hyperspectral image features using the following formula. : ; ; ; ; ; in, Represents spatial attention weights. Indicates hyperspectral image features, This represents the sigmoid function. This indicates the squeeze operation. Represents the cross-modal affinity matrix. The feature matrix representing a hyperspectral image, The feature matrix representing the lidar image, Represents a 1×1 convolution. This indicates the features of a lidar image.

[0014] Optionally, the spectral enhancement module calculates the enhanced lidar image features using the following formula. : ; ; ; ; ; in, Represents the spectral attention weight matrix. Indicates the features of a lidar image. This represents the sigmoid function. This indicates the squeeze operation. Represents the cross-modal spectral affinity matrix. The feature matrix representing a hyperspectral image, The feature matrix representing the lidar image, This represents the features of a hyperspectral image.

[0015] Optionally, the Fourier cross-fusion module calculates the fusion features using the following formula. : ; ; ; in, , , , Indicates the number of feature layers in the structured flow. The first structured flow represents the first structured flow. The layer features and the structure flow are the result of stitching together hyperspectral image features and lidar image features. This represents the structure flow after interaction with global modeling in the frequency domain. This represents a multilayer perceptron. Representation layer normalization, This indicates a two-feature interaction operation. This represents the compensation flow after interaction with global modeling in the frequency domain. The first term representing the compensation flow The layer features, and the compensation flow is the result of stitching together the enhanced hyperspectral image features and the enhanced lidar image features; ; ; The first structured flow represents the first structured flow. Features of the layer ,when hour, , The first term representing the compensation flow The characteristics of the layer, when hour, , This represents the two-dimensional inverse fast Fourier transform. This represents a two-dimensional Fast Fourier Transform. It is a learnable frequency domain filter. This indicates a flattening operation.

[0016] The above-mentioned solution in this application has the following beneficial effects: In the embodiments of this application, by introducing a modality missing module into the student network that can randomly handle the missing features of either hyperspectral image features or lidar image features, the student network can simulate different modality missing situations. At the same time, by first training the teacher network and then using the teacher network to perform knowledge distillation on the student network, the student network retains the unique discriminative information of each modality by distilling the feature diversity of the teacher network, and transfers the knowledge to the student network for learning. As a result, the student network can still accurately output classification results when faced with random and unpredictable modality missing, thereby improving the classification accuracy of remote sensing images.

[0017] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a multimodal remote sensing image classification method under modality loss provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a teacher network provided in one embodiment of this application; Figure 3 The image shows the classification results of different methods on the Trento dataset in an experiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0026] To make the classification accuracy of modality-deficient networks closer to that of full-modality networks, this application provides a multimodal remote sensing image classification method under modality-deficient conditions. This method introduces a modality-deficient module into the student network that can randomly handle the loss of either hyperspectral image features or lidar image features. This allows the student network to simulate different modality-deficient situations. Simultaneously, by first training the teacher network and then using the teacher network to perform knowledge distillation on the student network, the student network retains the unique discriminative information of each modality by distilling the feature diversity of the teacher network. This knowledge is then transferred to the student network for learning. As a result, the student network can still accurately output classification results when faced with random and unpredictable modality loss, thereby improving the classification accuracy of remote sensing images.

[0027] The following is an exemplary description of the multimodal remote sensing image classification method under modality loss conditions proposed in this application, with specific implementation examples.

[0028] like Figure 1 As shown in the embodiments of this application, the multimodal remote sensing image classification method under modality loss includes the following steps: Step 11: Obtain the multimodal remote sensing image dataset.

[0029] A multimodal remote sensing image dataset comprises multiple samples, each encompassing images of various modalities, such as hyperspectral images and lidar images (i.e., the lidar image corresponding to the hyperspectral image). These images of different modalities were acquired by different sensors over the same area. As an example, hyperspectral images can be acquired using a hyperspectral imager, and lidar images can be acquired using a lidar (LiDAR) device.

[0030] It is understandable that the above samples are all different. To facilitate model training, each sample has a true class label, which indicates the land cover classification result of the corresponding area, such as roads, forests, buildings, vineyards, etc.

[0031] In practical applications, to facilitate data processing, preprocessing is required after acquiring the raw images (i.e., hyperspectral images and lidar images). Specifically, the spatial dimension of the raw images is denoted as... ( Indicates the height of the original image. The width of the original image is represented by , and the number of spectral bands in the hyperspectral image is denoted as . The number of channels in a LiDAR image is denoted as... First, edge filling is performed on the original image, then pixel-wise... The size obtained by cropping to the center is The image patch, the cropped result of the hyperspectral image is denoted as... The cropping result of the LiDAR image is denoted as : ; ; in, Indicates the first in the image line, number Column of pixels; Represents a hyperspectral image. Represents a lidar image; Hyperspectral images are represented in pixels Cut into the center Image patches of various sizes Representing LiDAR images in pixels Cut into the center Image blocks of varying sizes; Represents the set of real numbers; Indicates all The number of channels, with both height and width being [value]. Three-dimensional space; Indicates all The number of channels, with both height and width being [value]. The three-dimensional space. It should be noted that the sample in the embodiments of this application refers to... and .

[0032] Step 12: Train the teacher network using the multimodal remote sensing image dataset to obtain the trained teacher network.

[0033] like Figure 2 As shown, the aforementioned teacher network includes a feature encoder module, a spatial enhancement module, a spectral enhancement module, a Fourier cross-fusion module, and an output module. The inputs to the feature encoder module are a hyperspectral image and a LiDAR image. The output of the feature encoder module is connected to the inputs of the spatial enhancement module, the spectral enhancement module, and the Fourier cross-fusion module, respectively. The outputs of the spatial enhancement module and the spectral enhancement module are both connected to the input of the Fourier cross-fusion module. The output of the Fourier cross-fusion module is connected to the input of the output module. The output of the output module is the predicted classification result (i.e., the land cover classification result for the region corresponding to the input hyperspectral image and LiDAR image).

[0034] The aforementioned feature encoder module is used to extract features from hyperspectral images and lidar images, and output hyperspectral image features and lidar image features; the spatial enhancement module is used to process the hyperspectral image features and lidar image features to obtain enhanced hyperspectral image features; the spectral enhancement module is used to process the hyperspectral image features and lidar image features to obtain enhanced lidar image features; the Fourier cross-fusion module is used to process the hyperspectral image features, lidar image features, enhanced hyperspectral image features, and enhanced lidar image features to obtain fused features; the output module is used to process the fused features to obtain the predicted classification results of the samples.

[0035] The feature encoder module consists of two parallel feature encoders. One feature encoder extracts features from the hyperspectral image, outputting hyperspectral image features, while the other extracts features from the LiDAR image, outputting LiDAR image features. Specifically, the two feature encoders have identical structures, both employing a lightweight three-layer convolutional structure, i.e., performing three consecutive convolutions, normalization, and ReLU activation function operations. The first and second convolutional layers progressively increase the number of channels while maintaining the spatial scale, thereby extracting local texture and spatial-spectral relationships. The third layer achieves downsampling through a convolution operation with a stride of 2, thus obtaining more discriminative shallow features. After feature extraction, the number of channels is obtained. Multimodal shallow features (i.e., hyperspectral image features, lidar image features).

[0036] Output of the feature encoder It can be represented as: ; in, Indicates the network to which it belongs. For teachers' network, For student networks; Indicates the corresponding mode. For hyperspectral image modes, For lidar image modes; Indicates the convolution operation; This indicates a normalization operation; Activation function operation.

[0037] For ease of description later, the hyperspectral image features will be denoted as... The features of a lidar image are denoted as follows: .

[0038] In some embodiments of this application, the multimodal shallow features (i.e., hyperspectral image features and lidar image features) extracted by the feature encoder module can be input into the spatial-spectral joint enhancement module to fully exploit the complementary information between modes. The spatial-spectral joint enhancement module consists of two parts: a spatial enhancement module and a spectral enhancement module.

[0039] The spatial enhancement module calculates the enhanced hyperspectral image features using the following formula. : ; ; ; ; ; in, Represents spatial attention weights. Indicates hyperspectral image features, This represents the sigmoid function. This indicates the squeeze operation. Represents the cross-modal affinity matrix. The feature matrix representing a hyperspectral image, The feature matrix representing the lidar image, Represents a 1×1 convolution. Indicates the features of a lidar image. Indicates matrix transpose. This represents the matrix inner product operation. This represents matrix multiplication.

[0040] It should be noted that, due to the limited spatial resolution of hyperspectral images, the spatial structural information of lidar images can be utilized to enhance their spatial representation capabilities. Specifically, by first performing feature compression on the features of both the hyperspectral image and the lidar image, the feature matrix of the hyperspectral image is obtained. Feature matrix of lidar image Then, by performing an inner product operation on the characteristic matrices of the two modes, the cross-modal affinity matrix is ​​obtained. Finally, the cross-modal affinity matrix was analyzed. A learnable squeeze operation is introduced along the spatial location dimension of the LiDAR image, and the affinity matrix is ​​mapped using the sigmoid function to obtain the spatial attention weight matrix corresponding to each spatial location in the hyperspectral image. Then, the enhanced hyperspectral image features are obtained by applying the weights to the original hyperspectral image features position by position. .

[0041] The spectral enhancement module calculates the enhanced lidar image features using the following formula. : ; ; ; ; ; in, Represents the spectral attention weight matrix. Indicates the features of a lidar image. This represents the sigmoid function. This indicates the squeeze operation. Represents the cross-modal spectral affinity matrix. The feature matrix representing a hyperspectral image, The feature matrix representing the lidar image, This represents the features of a hyperspectral image.

[0042] It should be noted that the rich spectral information of the hyperspectral image is used to supplement and enhance the features of the lidar image. Similar to the spatial enhancement branch, feature compression is first performed on the features of the hyperspectral image and the lidar image to obtain the feature matrix of the hyperspectral image. Feature matrix of lidar image Secondly, the cross-modal spectral affinity matrix is ​​obtained by performing an inner product operation on the characteristic matrices of the two modes. Then, a learnable squeeze operation is performed along the spectral position dimension of the hyperspectral image to obtain the spectral attention weight matrix of the LiDAR image. Finally, the weight matrix is ​​applied channel-by-channel to the original LiDAR image features to obtain the enhanced LiDAR image features. .

[0043] In some embodiments of this application, the aforementioned Fourier cross-fusion module achieves high-frequency noise suppression and dual-stream deep feature fusion through frequency domain modeling and cross-modal feature interaction. Specifically, the hyperspectral image features before and after enhancement, along with the LiDAR image features, are concatenated along the channel dimension to form two parallel feature streams. The concatenated hyperspectral image features before enhancement and the LiDAR image features constitute the structure stream, while the concatenated hyperspectral image features after enhancement and the LiDAR image features constitute the compensation stream. The two-dimensional feature maps of the structure stream and the compensation stream are flattened along the spatial dimension into a token sequence. A learnable positional code is introduced into each token, and it is added element-wise with the original features. Then, the features are regularized using a dropout operation to reduce overfitting and improve the model's generalization ability.

[0044] The token sequence is then rearranged into a two-dimensional feature map according to its original spatial layout. A Fourier transform is used to map the original time-domain features to the frequency domain. A learnable filter is then introduced and multiplied element-wise with the frequency-domain features to achieve frequency-domain filtering. After frequency-domain filtering, an inverse Fourier transform is used to map the features back to the time domain.

[0045] That is, the fusion features can be calculated using the following formula by the above Fourier cross-fusion module. : ; ; ; in, , , , This represents the number of characteristic layers in the structured flow (i.e., the number of characteristic layers in the compensated flow). The first structured flow represents the first structured flow. The layer features and the structure flow are the result of stitching together hyperspectral image features and lidar image features. This represents the structure flow after interaction with global modeling in the frequency domain. This represents a multilayer perceptron. Representation layer normalization, This indicates a two-feature interaction operation. This represents the compensation flow after interaction with global modeling in the frequency domain. The first term representing the compensation flow The layer features, and the compensation flow is the result of stitching together the enhanced hyperspectral image features and the enhanced lidar image features.

[0046] ; ; The first structured flow represents the first structured flow. Features of the layer ,when hour, , The first term representing the compensation flow The characteristics of the layer, when hour, , This represents the two-dimensional inverse fast Fourier transform. This represents a two-dimensional Fast Fourier Transform. It is a learnable frequency domain filter. This indicates a flattening operation.

[0047] As can be seen from the above calculation formula, the Fourier cross-fusion module enables information exchange between the structure flow and the compensation flow at each level through cross-branch fusion, achieving deep interaction between the two parallel feature flows. Specifically, the Dual Feature Interaction (DFI) method is adopted to adjust the interaction weights according to the input features of different modalities, achieving more adaptive feature fusion.

[0048] In some embodiments of this application, the output module includes a fully connected layer and a softmax activation function connected in sequence. The fused features output by the Fourier cross-fusion module are processed by the fully connected layer and the softmax activation function in sequence to output the predicted classification result.

[0049] In some embodiments of this application, the teacher network can be trained using conventional deep learning training methods, but the loss function used when training the teacher network... for: ; in, Represents the classification loss function. This represents the predicted classification result of the sample (i.e., the classification result output by the teacher network). Indicates the true class label of the sample. This represents the weights, used to adjust the weights of the two loss terms in the overall model optimization. The specific values ​​can be set according to the actual situation. This represents the cross-modal consistency loss of the teacher network.

[0050] Cross-modal consistency loss of the aforementioned teacher network for: ; in, This indicates the number of modes included in the multimodal input (2 in this application). This represents the single-modal prediction output (i.e., the output of the teacher network when a single-modal hyperspectral image / LiDAR image is input to the teacher network). Indicates the first Features corresponding to each mode Indicates the true class label of the sample. Indicates missing weights. , Indicates the missing rate of the hyperspectral image. This represents the missing rate of LiDAR images in the teacher network. 1 indicates no missing modes. This represents the balancing factor, used to adjust the weights. The specific value can be set according to the actual situation. This represents the mean squared error loss function. This represents the predicted classification result of the hyperspectral modality (i.e., the output of the teacher network when a single-modality hyperspectral image is input to the teacher network). This represents the predicted classification result of the lidar mode (i.e., the output of the teacher network when a single-mode lidar image is input to the teacher network).

[0051] It should be noted that the cross-modal consistency loss, by minimizing the classification loss between each single-modal prediction and the true label, constrains different modal branches to maintain basic discriminative ability during joint training. This application uses multimodal remote sensing image data with complete modalities as input, and trains the teacher network by combining the classification loss and cross-modal consistency loss. This iterative optimization of the teacher network ensures that its extracted fusion features are rich in complementary modality-specific information and shared semantic information.

[0052] Step 13: Construct a student network with the same network structure as the teacher network, and set up a modality missing module between the feature encoder module and the Fourier cross-fusion module of the student network. This modality missing module is used to randomly missing one of the hyperspectral image features and lidar image features output by the feature encoder module of the student network.

[0053] In some embodiments of this application, the student network and the teacher network have the same network structure, but the difference lies in that a mode-deficient module is provided between the feature encoder module and the Fourier cross-fusion module of the student network. That is, a mode-deficient module is provided between the output of the feature encoder module and the input of the Fourier cross-fusion module. This mode-deficient module is used to perform steps 13.1 to 13.3: Step 13.1: Project and map the hyperspectral image features and LiDAR image features output by the feature encoder module to a common space of the same dimension through a linear layer. middle.

[0054] Step 13.2: Normalize the hyperspectral image features and the lidar image features after projection mapping, respectively. For example, this normalization process can be an L2 normalization operation.

[0055] Step 13.3: Calculate the cosine similarity between the normalized hyperspectral image features and the normalized lidar image features. If the cosine similarity is greater than or equal to a preset threshold (e.g., 0.85), randomly perform missing processing on one of the hyperspectral image features and lidar image features output by the feature encoder module. If the cosine similarity is less than the preset threshold (e.g., 0.85), no missing processing is required, and the output of the feature encoder module is directly given to the Fourier cross-fusion module.

[0056] In some embodiments of this application, the normalized cosine similarity can be calculated using dot product. : ; in, This represents the features of the hyperspectral image after normalization. This represents the features of the lidar image after normalization.

[0057] It should be noted that when the cosine similarity is greater than or equal to a preset threshold (e.g., 0.85), it indicates that the feature redundancy of the two modes in this region is extremely high. Random deletion processing can be applied to either the hyperspectral image features or the lidar image features output by the feature encoder module.

[0058] In some embodiments of this application, the missing rate for random missing data processing is one of 0.3, 0.5, 0.8, and 1. Here, 1 indicates that the modality is completely missing, and 0.3, 0.5, and 0.8 represent that 30%, 50%, and 80% of the available information for the modality are missing, respectively. The complete multimodal remote sensing image data (i.e., hyperspectral images and lidar images) input into the student network is transformed into incomplete multimodal remote sensing image data through a cross-modal feature distance missing rate mechanism, simulating the modality missing situation caused by various factors in real-world application scenarios.

[0059] Step 14: Distill the multimodal information knowledge learned by the teacher network into the student network to complete the training of the student network under the condition of modality missing, and obtain the trained student network.

[0060] In some embodiments of this application, the student network and teacher network have the same main structure, the difference being the introduction of a cross-modal feature distance missing rate mechanism. In this application, knowledge distillation is performed on the student network by fixing the teacher network model parameters. To effectively transfer knowledge from the teacher network to the student network, this application designs a dual adaptive similarity-preserving distillation method. Specifically, the Gram matrices of the teacher and student networks are first constructed, then an adaptive temperature normalization strategy is introduced to alleviate the gradient imbalance problem caused by the difference in amplitude between teacher and student features. The Frobenius norm of the teacher Gram matrix is ​​used as a global scaling factor, and the Gram matrices of both the teacher and student networks are normalized. Finally, the loss function for knowledge distillation is defined based on this.

[0061] Specifically, the loss function of the above knowledge distillation for: ; in, Indicates the uncertainty weight. It is in the form of sample pairs, used to dynamically adjust the distillation intensity. , This represents the sigmoid function. and They represent the first The first sample and the first The variance of a sample, , Represents the total dimension. Indicating the first in the teacher network The first sample Dimensional features, Indicating the first in the teacher network The average of all dimensions of a sample. This indicates temperature hyperparameters. express The Middle The first sample and the first The degree of similarity between samples This represents the normalized Gram matrix of the teacher network. express The Middle The first sample and the first The degree of similarity between samples This represents the normalized Gram matrix of the student network. Represents a constant. .in, and The calculation method is the same, just make sure that it is related to the first one. The data of the nth sample is replaced with the data of the th... Data from one sample is sufficient.

[0062] ; ; ; ; ; in, The Gram matrix representing the teacher network. This represents the global scale factor of the teacher network. Denotes the Frobenius norm of a matrix. This represents a constant term, used to prevent numerical instability. , express The Middle Line number Column elements, This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the teacher network in the spatial and channel dimensions. The Gram matrix representing the student network. express The Middle Line number Column elements, This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the student network in the spatial and channel dimensions.

[0063] In some embodiments of this application, the multimodal information learned by the teacher network after training is distilled into the student network to complete the training of the student network under modality-deficient conditions (it should be noted that knowledge distillation of the student network using the trained teacher network is part of the overall training process of the student network). Specifically, conventional deep learning training methods can be used to train the student network, but the loss function during student network training... for: ; in, This represents the classification loss function, which calculates the classification loss predicted by the student network itself based on the multimodal input after considering the cross-modal feature distance missing rate. This represents the predicted classification result of the sample (i.e., the classification result output by the student network). Indicates the true class label of the sample. Indicates weight, The loss function represents knowledge distillation. This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the teacher network in the spatial and channel dimensions. This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the student network in the spatial and channel dimensions. Indicates weight, This represents the cross-modal consistency loss of the student network. and All settings can be customized according to the actual situation.

[0064] It should be noted that the cross-modal consistency loss of the student network is calculated in the same way as that of the teacher network. The difference is that the output of the teacher network is replaced with the output of the student network, and the missing weight is calculated based on the missing rate of random missing handling.

[0065] It is worth mentioning that, For the student network, the cross-modal consistency loss is the same as that of the teacher network, which continues to promote equitable learning across all modalities and lays a good foundation for handling any single modal input; The dual adaptive similarity matrix distillation loss is used to transfer the complete multimodal knowledge learned in the teacher network to the student network, compensating for the lack of information caused by the absence of random modalities. Overall, the student network is iteratively trained using incomplete multimodal data incorporating cross-modal feature distance missing rates as input, combined with the above three losses.

[0066] Step 15: Use the trained student network to classify the hyperspectral images and lidar images of the study area to obtain the classification results of the study area.

[0067] In some embodiments of this application, the aforementioned study area is an area requiring remote sensing image classification. Specifically, hyperspectral images and lidar images of the study area can be input into a trained student network for processing to obtain the classification results (i.e., land cover classification results) for the study area, such as vineyards, buildings, forests, roads, etc.

[0068] The following section provides an exemplary illustration of the multimodal remote sensing image classification method under modality loss conditions proposed in this application, using experimental data.

[0069] The effectiveness was validated using the publicly available Trento remote sensing dataset. Table 1 shows the settings for the training and test samples. Furthermore, the comparison methods selected included classification methods applicable to incomplete multimodal remote sensing images, such as Deep Gradient Network (DGDNet), Multiscale Head Network (MSHNet), Latent Denoising Sparse Stacked Autoencoder (LDS2AE), Shape Spectrum Transformer (ShaSpec), and Multimodal Medical Transformer (mmFormer).

[0070] Table 1. Sample splitting settings for the Trento dataset

[0071] The experimental results of different methods are shown in Table 2 and Figure 3 As shown, the results demonstrate that, compared with various advanced methods, the method proposed in this application achieves the best results in terms of overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Ka). This proves that the method proposed in this application effectively improves the classification accuracy of land cover under different modality combinations. The average values ​​in Table 2 represent the average classification performance evaluation indicators of different methods under the same training samples, with and without missing modality data.

[0072] Table 2. Comparative experimental results of different methods on the Trento dataset.

[0073] In summary, the multimodal remote sensing image classification method under modality missing conditions provided in this application solves the problem of the performance of multimodal remote sensing classification models being drastically reduced due to random modality missing in actual deployment by introducing a cross-modal feature distance missing rate mechanism training strategy, a dual adaptive similarity matrix distillation, and a synergistic mechanism of cross-modal consistency loss.

[0074] The model enhances robustness by simulating different modality loss scenarios. The proposed cross-modal consistency loss strategy effectively balances the multimodal training process, improves the ability to utilize weak modalities, and further preserves the unique discriminative information of each modality by distilling the feature diversity of the teacher network. This knowledge is then transferred to the student network for learning, thereby improving the prediction accuracy of the student network in the face of modality loss scenarios.

[0075] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multimodal remote sensing image classification method under modality loss, characterized in that, include: Obtain a multimodal remote sensing image dataset; the multimodal remote sensing image dataset includes multiple samples, the samples include hyperspectral images and corresponding lidar images; A teacher network is trained using a multimodal remote sensing image dataset to obtain the trained teacher network. The teacher network includes a feature encoder module, a spatial enhancement module, a spectral enhancement module, a Fourier cross-fusion module, and an output module. The feature encoder module is used to extract features from hyperspectral images and lidar images and output hyperspectral image features and lidar image features. The spatial enhancement module is used to process the hyperspectral image features and lidar image features to obtain enhanced hyperspectral image features. The spectral enhancement module is used to process the hyperspectral image features and lidar image features to obtain enhanced lidar image features. The Fourier cross-fusion module is used to process the hyperspectral image features, lidar image features, enhanced hyperspectral image features, and enhanced lidar image features to obtain fused features. The output module is used to process the fused features to obtain the predicted classification results of the samples. A student network with the same network structure as the teacher network is constructed, and a modality missing module is provided between the feature encoder module and the Fourier cross-fusion module of the student network. The modality missing module is used to randomly missing one of the hyperspectral image features and lidar image features output by the feature encoder module of the student network. Distill the multimodal information learned by the teacher network into the student network to complete the training of the student network under the condition of modality missing, and obtain the trained student network; The trained student network is used to classify the hyperspectral and lidar images of the study area to obtain the classification results of the study area.

2. The multimodal remote sensing image classification method according to claim 1, characterized in that, The missing modality is used to perform the following steps: The hyperspectral image features and lidar image features output by the feature encoder module are projected and mapped into a common space of the same dimension through a linear layer. Normalization processing was performed on the hyperspectral image features and the lidar image features after projection mapping, respectively. Calculate the cosine similarity between the normalized hyperspectral image features and the normalized lidar image features; If the cosine similarity is greater than or equal to a preset threshold, then one of the hyperspectral image features and lidar image features output by the feature encoder module will be randomly missing.

3. The multimodal remote sensing image classification method according to claim 2, characterized in that, The missing rate for random missing handling is one of 0.3, 0.5, 0.8, and 1.

4. The multimodal remote sensing image classification method according to claim 1, characterized in that, Loss function of knowledge distillation for: ； in, Indicates the uncertainty weight. , This represents the sigmoid function. and They represent the first The first sample and the first The variance of a sample, , Represents the total dimension. Indicating the first in the teacher network The first sample Dimensional features, Indicating the first in the teacher network The average of all dimensions of a sample. This indicates temperature hyperparameters. express The Middle The first sample and the first The degree of similarity between samples This represents the normalized Gram matrix of the teacher network. express The Middle The first sample and the first The degree of similarity between samples This represents the normalized Gram matrix of the student network. Represents a constant. ; ； ； ； ； ； in, The Gram matrix representing the teacher network. This represents the global scale factor of the teacher network. Denotes the Frobenius norm of a matrix. Represents a constant term. , express The Middle Line 1 Column elements, This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the teacher network in the spatial and channel dimensions. The Gram matrix representing the student network. express The Middle Line 1 Column elements, This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the student network in the spatial and channel dimensions.

5. The multimodal remote sensing image classification method according to claim 1, characterized in that, Loss function when training teacher network for: ； in, Represents the classification loss function. This indicates the predicted classification result of the sample. Indicates the true class label of the sample. Indicates weight, This represents the cross-modal consistency loss of the teacher network.

6. The multimodal remote sensing image classification method according to claim 5, characterized in that, Cross-modal consistency loss of teacher networks for: ； in, This indicates the number of modes contained in the multimodal input. This represents the single-mode prediction output. Indicates the first Features corresponding to each mode Indicates the true class label of the sample. Indicates missing weights. , Indicates the missing rate of the hyperspectral image. This indicates the missing rate of the LiDAR image. Represents the balance factor. This represents the mean squared error loss function. This represents the predicted classification results for hyperspectral modes. This represents the predicted classification results of the lidar modes.

7. The multimodal remote sensing image classification method according to claim 1, characterized in that, Loss function when training a student network after knowledge distillation for: ； in, Represents the classification loss function. This indicates the predicted classification result of the sample. Indicates the true class label of the sample. Indicates weight, The loss function represents knowledge distillation. This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the teacher network in the spatial and channel dimensions. This represents the result of flattening the fusion features output by the Fourier cross-fusion module of the student network in the spatial and channel dimensions. Indicates weight, This represents the cross-modal consistency loss of the student network.

8. The multimodal remote sensing image classification method according to claim 1, characterized in that, The spatial enhancement module calculates the enhanced hyperspectral image features using the following formula. : ； ； ； ； ； in, Represents spatial attention weights. Indicates hyperspectral image features, This represents the sigmoid function. This indicates the squeeze operation. Represents the cross-modal affinity matrix. The feature matrix representing a hyperspectral image, The feature matrix representing the lidar image, Represents a 1×1 convolution. This indicates the features of a lidar image.

9. The multimodal remote sensing image classification method according to claim 1, characterized in that, The spectral enhancement module calculates the enhanced lidar image features using the following formula. : ； ； ； ； ； in, Represents the spectral attention weight matrix. Indicates the features of a lidar image. This represents the sigmoid function. This indicates the squeeze operation. Represents the cross-modal spectral affinity matrix. The feature matrix representing a hyperspectral image, The feature matrix representing the lidar image, This represents the features of a hyperspectral image.

10. The multimodal remote sensing image classification method according to claim 1, characterized in that, The Fourier cross-fusion module calculates the fusion features using the following formula. : ； ； ； in, , , , Indicates the number of feature layers in the structured flow. The first character representing the structured flow The layer features and the structure flow are the result of stitching together hyperspectral image features and lidar image features. This represents the structure flow after interaction with global modeling in the frequency domain. This represents a multilayer perceptron. Representation layer normalization, This indicates a two-feature interaction operation. This represents the compensation flow after interaction with global modeling in the frequency domain. The first one represents the compensation flow. The layer features, and the compensation flow is the result of stitching together the enhanced hyperspectral image features and the enhanced lidar image features; ； ； The first character representing the structured flow Features of the layer ,when hour, , The first one represents the compensation flow. The characteristics of the layer, when hour, , This represents the two-dimensional inverse fast Fourier transform. This represents a two-dimensional Fast Fourier Transform. It is a learnable frequency domain filter. This indicates a flattening operation.

Images