A method for joint classification of hyperspectral images and lidar data

Abstract

This invention relates to the field of deep learning technology, and in particular provides a joint classification method for hyperspectral images and LiDAR data. The method first extracts multi-scale features from multimodal data by decoupling a three-branch encoder. Then, a self-designed feature fusion module performs attention-guided cross-modal fusion on three initial features at the same scale, resulting in three multimodal fused features at different scales. The deepest fused feature is input to a weight-shared decoder, where pseudo-data is obtained through a reconstruction task. A structure consistency-aware discriminator identifies real data from pseudo-data and returns a discrimination loss, thereby updating the encoder parameters and iterating until optimal. Finally, the fused features at the three scales obtained based on the optimal parameters are input into a multi-level feature fusion classifier to complete the classification. This method effectively improves classification accuracy and achieves adaptive fusion and enhancement of multimodal features.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a joint classification method for hyperspectral images and lidar data. Background Technology

[0002] With the rapid development of airborne and spaceborne remote sensing technologies, multi-source remote sensing data plays an increasingly important role in fields such as natural resource surveys, ecological environment monitoring, urban planning and construction, disaster prevention and control, and agricultural production. However, a single data source often struggles to comprehensively characterize the features of ground objects in complex environments. While hyperspectral (HIS) image data can distinguish material composition, it falls short in spatial structure characterization; and while lidar (LiDAR) image data can accurately represent three-dimensional geometry, it lacks fine-grained spectral attributes. This complementary relationship makes the joint classification of hyperspectral and lidar data a cutting-edge direction in current remote sensing research. Although significant progress has been made in the fusion classification technology of hyperspectral and lidar data, existing technologies still face the following shortcomings:

[0003] (1) Modal feature entanglement and independence issues: Existing deep frameworks generally suffer from cross-interference of modal information in the feature extraction stage, lacking effective separation of spectral, spatial and elevation information, which affects the purity and efficiency of subsequent fusion; (2) Paradigm limitations of cross-modal guidance: The current attention interaction mechanism is not refined enough and fails to fully realize the robust and accurate guidance of LiDAR structure priors on HSI spectral features; (3) Stability and semantic preservation of adversarial training: When using Generative Adversarial Networks (GANs) for feature enhancement or domain adaptation, the design of the discriminator is difficult to balance training stability and sensitivity to high-level semantic information, which restricts the quality of generated features. Summary of the Invention

[0004] In view of this, the present invention provides a joint classification method for hyperspectral images and lidar data to improve the classification accuracy of the network and achieve adaptive fusion and enhancement of multimodal features.

[0005] In a first aspect, the present invention provides a joint classification method for hyperspectral images and lidar data, the method comprising:

[0006] Step 1: Extract multi-scale features from the original image data of hyperspectral image (HSI) and lidar (LiDAR) by decoupling the three-branch encoder to obtain three initial features at different scales.

[0007] Step 2: Use the feature fusion module to perform attention-guided cross-modal fusion of three initial features at the same scale to obtain multimodal fused features;

[0008] Step 3: Input the deepest feature from the multimodal fusion features into the weight-sharing decoder to perform the data reconstruction task and obtain the reconstructed pseudo data;

[0009] Step 4: Input the original real data and the reconstructed pseudo data into the structure consistency perception discriminator for identification, distinguish between real data and pseudo data, and return the discrimination loss;

[0010] Step 5: Return the discrimination loss to the decoupled three-branch encoder, update the parameters, and repeat this process multiple times to obtain the optimal parameters;

[0011] Step 6: Input the multimodal fusion features obtained based on the optimal parameters at three different scales into the multi-level feature fusion classifier to perform the final classification task.

[0012] Optionally, step 1 includes:

[0013] Multi-scale feature extraction is performed through a decoupled three-branch encoder in the decoupled guided fusion generator. The decoupled three-branch encoder employs a three-branch parallel coding architecture to extract three initial features at different scales from the raw image data of hyperspectral images and LiDAR. The three initial features are spectral features. Spatial features Elevation characteristics The three branches are spectral feature extraction, spatial feature extraction, and elevation feature extraction; each branch employs a hierarchical residual learning framework for multi-scale feature extraction.

[0014] a. The spectral feature extraction branch processes the spectral information in the raw image data of the hyperspectral image (HSI), and consists of three cascaded spectral feature convolutional blocks. It extracts spectral features at different levels step by step through a three-level residual network (ResidualBlock). Each convolutional block addresses the vanishing gradient problem in deep networks through skip connections in the residual network, improving the depth and accuracy of feature extraction. Batch normalization is used to improve the training stability and convergence speed of the model. Then, 2×2 max pooling is applied after each level to downsample the spatial dimension. At the same time, the ReLU activation function is embedded in the residual network to enhance the expressive power of the model.

[0015] b. The spatial feature extraction branch focuses on extracting spatial structure features from hyperspectral images. It consists of three concatenated spatial feature convolutional blocks. The first convolutional block extracts local spatial features through 2D convolution, is then stabilized through batch normalization, and nonlinearity is introduced by a linear rectified unit. The second and third convolutional blocks have the same structure as the spectral feature convolutional blocks. They address the gradient vanishing problem in deep networks through a residual network, and further extract deep spatial features by combining batch normalization and max pooling. Finally, they output spatial features at different scales. ;

[0016] c. The elevation feature extraction branch processes the raw image data from the lidar and extracts elevation information features. It consists of three concatenated elevation feature convolutional blocks, employing the same residual network structure as the spectral feature extraction branch to ensure feature hierarchy compatibility. This three-level residual network extracts multi-scale terrain features, i.e., elevation information features, from the raw LiDAR image data. .

[0017] Optionally, step 2 includes:

[0018] The feature fusion module in the decoupled guided fusion generator is used to process three initial features at the same scale. Attention-guided cross-modal fusion to obtain multimodal fusion features ,Right now The feature fusion module employs a hierarchical attention fusion mechanism to adaptively fuse multimodal features at each feature level. The feature fusion module performs the following processing flow:

[0019] d. Spectral-spatial feature fusion;

[0020] d1. Feature stitching: Combining spectral features at the same level Spatial features By splicing along the channel dimension, a joint feature is formed. ;

[0021] d2. Feature projection: Dimensionality reduction and fusion are performed using 1×1 convolutional layers, batch normalization, and linear rectified units.

[0022] ;

[0023] in, ReLU represents the activation function, and BN represents batch normalization. Indicates the convolution kernel parameters;

[0024] d3. Channel Attention Weighting: The channel attention mechanism is applied to the features to calculate the weight of each channel, identify and highlight the information of important channels, and generate a channel attention weight map;

[0025] ;

[0026] in, This indicates multiplication by channel. This represents the Sigmoid activation function. An MLP consists of two fully connected layers.

[0027] e. LiDAR-guided cross-modal augmentation;

[0028] e1. Dual-path feature extraction: Extraction of elevation information features Multi-scale statistical properties of elevation features are extracted using average pooling and max pooling:

[0029] ;

[0030] e2. Spatial Attention Generation: First, the results of average pooling and max pooling are concatenated to integrate multi-scale statistical information; then, the concatenated features are convolved, and finally converted into a spatial attention weight map through a mapping function.

[0031] ;

[0032] in, This represents a 1×1 convolution kernel. This represents the Sigmoid activation function;

[0033] f. Modal interaction enhancement: The spectral-spatial weighted features processed by the upper branch and the elevation weighted features processed by the lower branch are multiplied element-wise again to obtain the fused features. :

[0034] ;

[0035] The final output is multimodal features. It contains the spectral-spatial features of HSI and the elevation information features of LiDAR.

[0036] Optionally, step 3 includes:

[0037] A dual-path decoding architecture with shared weights is adopted to achieve cross-modal feature collaborative reconstruction; the decoder fuses features at the deepest level of the encoder. As input, the spatial resolution of the feature map is first improved through upsampling. Then, local correlation information is extracted through convolutional layers to refine the feature representation. Batch normalization accelerates training and improves stability. Finally, a leaky linear rectified unit introduces nonlinearity, and this operation is repeated multiple times to reconstruct the original input data. The weights of the convolutional layers in both paths are shared, allowing the model to learn common feature representations of HSI and LiDAR. The decoder outputs the reconstructed data. It forms an adversarial learning loop with the original image data, while providing regularization constraints for the classification task, thus forming a multi-task collaborative optimization framework.

[0038] Optionally, step 4 includes:

[0039] Data discrimination is achieved through a structure consistency-aware discriminator. This module employs a dual-discriminator architecture, designing independent and structurally identical discriminators for HSI and LiDAR modalities, respectively. First, the two branches input the original real image data and the reconstructed pseudo data, respectively. Next, the data passes through convolutional layers to extract local features. Batch normalization is applied to the convolutional output features to accelerate model training and improve stability. Then, a leaky linear rectifier unit introduces nonlinearity to enhance the ability to express complex patterns. Furthermore, a self-attention mechanism captures long-distance dependencies of features, enhancing the perception of structural differences. Finally, the features are mapped to the discriminant space through a fully connected layer, and the probability of real or pseudo data is output through a sigmoid function. The dual-discriminator architecture integrates two core technologies to improve the stability and discriminative ability of adversarial training:

[0040] h. Spectral normalization: All convolutional and fully connected layers in the discriminator are spectral normalized by constraining the Lipschitz constant of the discriminator function.

[0041] i. Self-attention mechanism: A self-attention module is embedded after the deep feature map of the discriminator. The self-attention module is used to capture the long-distance dependency between any two locations in the feature map, so that the discriminator can evaluate the realism of local texture and judge the global structural consistency of the generated image, thereby providing gradient feedback to the generator. The expression for the feature enhancement of the self-attention mechanism is:

[0042] ;

[0043] Among them, query ,key ,value All are feature projections. , , respectively query ,key ,value The weights; Let K be the dimension of the key.

[0044] Optionally, step 6 includes:

[0045] To leverage feature information at different scales, a multi-level feature fusion classifier constructs a hierarchical adaptive decision framework. This framework enhances classification robustness through multi-scale feature collaborative optimization, receiving fused features from different scales of the encoder in parallel. Each feature is input into a dedicated classification path. First, it passes through convolutional layers to extract key classification information. Then, batch normalization improves stability, and leaky linear rectifier units introduce nonlinearity to refine feature representation. A combined module is then added to enhance feature discriminative power. Finally, a normalization function maps each level of feature to its corresponding class probability, the expression of which is:

[0046] ;

[0047] in, Activated for softmax. ReLU nonlinear activation, The parameter r is a learnable parameter, and its values ​​are 1 and 2.

[0048] Finally, the decision information from the three category probability maps is fused by weighted summation to compensate for the limitations of single-level features, ultimately outputting a globally optimal classification result that balances fine-grained details with global structural consistency.

[0049] ;

[0050] Among them, the weights δ1, δ2, and δ3 are all learnable parameters that are automatically optimized during training based on the classification ability of the three-level features, and satisfy that δ1+δ2+δ3 approaches 1.

[0051] The final classification result R i By joint classification probability P i The maximum value index is determined by the expression:

[0052] ;

[0053] Where N is the total number of land cover categories.

[0054] In a second aspect, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform a joint classification method for hyperspectral images and lidar data, as described in the first aspect or any possible implementation thereof.

[0055] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the joint classification method for hyperspectral images and lidar data in the first aspect or any possible implementation of the first aspect.

[0056] The technical solution provided by this invention includes a method that extracts multi-scale features from multimodal data by decoupling a three-branch encoder, then uses a feature fusion module to perform attention-guided cross-modal fusion of three initial features at the same scale to obtain multimodal fused features at three scales; the deepest feature in the multimodal fused features is input into a weight-shared decoder to perform a data reconstruction task and obtain reconstructed pseudo-data; the original real data and the reconstructed pseudo-data are input into a structure consistency-aware discriminator for identification to distinguish between real and pseudo-data, and a discrimination loss is returned; the discrimination loss is input into the decoupling three-branch encoder for parameter updates, repeated multiple times to obtain optimal parameters; and the three multimodal fused features at different scales obtained based on the optimal parameters are input into a multi-level feature fusion classifier to perform the final classification task. This method improves the classification accuracy of the network and achieves adaptive fusion and enhancement of multimodal features. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart of a joint classification method for hyperspectral images and lidar data provided in an embodiment of the present invention;

[0059] Figure 2 A schematic diagram illustrating the acquisition of three initial features at different scales, provided for embodiments of the present invention;

[0060] Figure 3 This is a schematic diagram illustrating the acquisition of multimodal fusion features according to an embodiment of the present invention;

[0061] Figure 4a This is a schematic diagram of the experimental results of the weight coefficient λ of the Houston2013 dataset provided in this embodiment of the invention;

[0062] Figure 4b This is a schematic diagram of the experimental results of the weight coefficient λ of the MUUFL dataset provided in this embodiment of the invention;

[0063] Figure 4c This is a schematic diagram of the experimental results of the weight coefficient λ of the Augsburg dataset provided in this embodiment of the invention;

[0064] Figure 5a This is a schematic diagram of the learning rate LR experiment results for the Houston2013 dataset provided in this embodiment of the invention;

[0065] Figure 5b This is a schematic diagram of the learning rate LR experiment results for the MUUFL dataset provided in this embodiment of the invention;

[0066] Figure 5c This is a schematic diagram of the learning rate LR experiment results for the Augsburg dataset provided in this embodiment of the invention;

[0067] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0070] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0071] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0072] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0073] Figure 1 A flowchart of a joint classification method for hyperspectral images and lidar data provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0074] This invention proposes a multi-path deep learning network architecture to solve the multimodal joint classification task of hyperspectral imaging (HSI) and lidar (LiDAR). This architecture achieves adaptive fusion and enhancement of multimodal features through collaborative optimization of feature representation learning and joint distribution modeling. Figure 1 As shown, the network architecture mainly consists of two parts: a generative adversarial network (GAN) for feature learning and data reconstruction, which includes a decoupled guided fusion generator and a structure consistency-aware discriminator; and a multi-level feature fusion classifier for the final classification decision. The GAN part drives feature learning through an unsupervised image reconstruction task, while the multi-level feature fusion classifier optimizes these features through a supervised classification task. These two parts are co-trained using a joint loss function.

[0075] The decoupled guided fusion generator is the core of the entire model framework, employing an encoder-decoder structure. The decoupled three-branch encoder extracts features at different scales from multimodal data, while the feature fusion module adaptively fuses three different features (spectral features, spatial features, and elevation features) at the same scale across modalities. The weight-shared decoder then utilizes the deepest features obtained from the fusion to acquire the reconstructed pseudo-data.

[0076] Step 1: Multi-scale feature extraction is performed on the original image data of hyperspectral image (HSI) and lidar (LiDAR) by decoupling the three-branch encoder to obtain three initial features at different scales.

[0077] In embodiments of the present invention, such as Figure 2 As shown, step 1 includes:

[0078] Multi-scale feature extraction is performed through a decoupled three-branch encoder in the decoupled guided fusion generator. The decoupled three-branch encoder employs a three-branch parallel coding architecture to extract three initial features at different scales from the raw image data of hyperspectral images and LiDAR. The three initial features are spectral features. Spatial features Elevation characteristics The three branches are spectral feature extraction, spatial feature extraction, and elevation feature extraction; each branch employs a hierarchical residual learning framework for multi-scale feature extraction.

[0079] a) The spectral feature extraction branch processes the spectral information in the raw image data of the hyperspectral image (HSI). It consists of three cascaded spectral feature convolutional blocks, which progressively extract spectral features at different levels through a three-level residual network (ResidualBlock). Each convolutional block addresses the vanishing gradient problem in deep networks through skip connections in the residual network, improving the depth and accuracy of feature extraction. Batch normalization is used to improve the training stability and convergence speed of the model. Then, 2×2 max pooling is applied after each level to downsample the spatial dimension. At the same time, the ReLU activation function is embedded in the residual network to enhance the expressive power of the model.

[0080] b. The spatial feature extraction branch focuses on extracting spatial structure features from hyperspectral images. It consists of three concatenated spatial feature convolutional blocks. The first convolutional block extracts local spatial features through 2D convolution, is then stabilized through batch normalization, and nonlinearity is introduced by a linear rectified unit. The second and third convolutional blocks have the same structure as the spectral feature convolutional blocks. They address the gradient vanishing problem in deep networks through a residual network, and further extract deep spatial features by combining batch normalization and max pooling. Finally, they output spatial features at different scales. ;

[0081] c. The elevation feature extraction branch processes the raw image data from the lidar and extracts elevation information features. It consists of three concatenated elevation feature convolutional blocks, employing the same residual network structure as the spectral feature extraction branch to ensure feature hierarchy compatibility. This three-level residual network extracts multi-scale terrain features, i.e., elevation information features, from the raw LiDAR image data. .

[0082] Step 2: Use the feature fusion module to perform attention-guided cross-modal fusion of three initial features at the same scale to obtain multimodal fused features.

[0083] In this embodiment of the invention, the feature fusion module is a self-designed module.

[0084] In embodiments of the present invention, such as Figure 3 As shown, step 2 includes:

[0085] The feature fusion module in the decoupled guided fusion generator is used to process three initial features at the same scale. Attention-guided cross-modal fusion to obtain multimodal fusion features ,Right now The feature fusion module employs a hierarchical attention fusion mechanism to adaptively fuse multimodal features at each feature level. The feature fusion module performs the following processing flow:

[0086] d. Spectral-spatial feature fusion;

[0087] d1. Feature stitching: Combining spectral features at the same level Spatial features By splicing along the channel dimension, a joint feature is formed. ;

[0088] d2. Feature projection: Dimensionality reduction and fusion are performed using 1×1 convolutional layers, batch normalization, and linear rectified units.

[0089] ;

[0090] in, ReLU represents the activation function, and BN represents batch normalization. Indicates the convolution kernel parameters;

[0091] d3. Channel Attention Weighting: The channel attention mechanism is applied to the features to calculate the weight of each channel, identify and highlight the information of important channels, and generate a channel attention weight map;

[0092] ;

[0093] in, This indicates multiplication by channel. This represents the Sigmoid activation function. An MLP consists of two fully connected layers.

[0094] e. LiDAR-guided cross-modal augmentation;

[0095] e1. Dual-path feature extraction: Extraction of elevation information features Multi-scale statistical properties of elevation features are extracted using average pooling and max pooling:

[0096] ;

[0097] e2. Spatial Attention Generation: First, the results of average pooling and max pooling are concatenated to integrate multi-scale statistical information; then, the concatenated features are convolved, and finally converted into a spatial attention weight map through a mapping function.

[0098] ;

[0099] in, This represents a 1×1 convolution kernel. This represents the Sigmoid activation function;

[0100] f. Modal interaction enhancement: The spectral-spatial weighted features processed by the upper branch and the elevation weighted features processed by the lower branch are multiplied element-wise again to obtain the fused features. :

[0101] ;

[0102] The final output is multimodal features. It incorporates the spectral-spatial features of HSI and the elevation information features of LiDAR. This fused feature is input into a hierarchical weighted classifier for final decision-making, achieving end-to-end joint optimization.

[0103] Step 3: Input the deepest feature from the multimodal fusion features into the weight-sharing decoder to perform the data reconstruction task and obtain the reconstructed pseudo data.

[0104] In this embodiment of the invention, step 3 includes:

[0105] A dual-path decoding architecture with shared weights is adopted to achieve cross-modal feature collaborative reconstruction; the decoder fuses features at the deepest level of the encoder. As input, the spatial resolution of the feature map is first improved through upsampling. Then, local correlation information is extracted through convolutional layers to refine the feature representation. Batch normalization accelerates training and improves stability. Finally, a leaky linear rectified unit introduces nonlinearity, and this operation is repeated multiple times to reconstruct the original input data. The weights of the convolutional layers in both paths are shared, allowing the model to learn common feature representations of HSI and LiDAR. The decoder outputs the reconstructed data. It forms an adversarial learning loop with the original image data, while providing regularization constraints for the classification task, thus forming a multi-task collaborative optimization framework.

[0106] Step 4: Input the original real data and the reconstructed pseudo data into the structure consistency perception discriminator for identification, distinguish between real data and pseudo data, and return the discrimination loss.

[0107] In this embodiment of the invention, step 4 includes:

[0108] Data discrimination is achieved through a structure consistency-aware discriminator. This module employs a dual-discriminator architecture, designing independent and structurally identical discriminators for HSI and LiDAR modalities, respectively. First, the two branches input the original real image data and the reconstructed pseudo-data, respectively. Next, the data passes through convolutional layers to extract local features. Batch normalization is applied to the convolutional output features to accelerate model training and improve stability. Then, a leaky linear rectifier unit introduces nonlinearity to enhance the ability to express complex patterns. A self-attention mechanism captures long-distance dependencies of features, enhancing the perception of structural differences. Finally, the features are mapped to the discriminant space through a fully connected layer, and the probability of real or pseudo-data is output through a sigmoid function, thus achieving data discrimination. The dual-discriminator architecture integrates two core technologies to improve the stability and discriminative capability of adversarial training:

[0109] h. Spectral Normalization: All convolutional and fully connected layers in the discriminator are normalized using spectral normalization. By constraining the Lipschitz constant of the discriminator function, spectral normalization effectively prevents gradient explosion and mode collapse during training, making adversarial training more stable.

[0110] i. Self-Attention Mechanism: A self-attention module is embedded after the deep feature map of the discriminator. This module captures long-distance dependencies between any two locations in the feature map, enabling the discriminator to evaluate the realism of local textures and assess the global structural consistency of the generated image. This provides the generator with more comprehensive and accurate gradient feedback, prompting the decoupled three-branch encoder in the generator to extract more discriminative features. The expression for self-attention feature enhancement is:

[0111] ;

[0112] Among them, query ,key ,value All are feature projections. , , respectively query ,key ,value The weights; Let K be the dimension of the key.

[0113] Step 5: Return the discrimination loss to the decoupled three-branch encoder, update the parameters, and repeat this process multiple times to obtain the optimal parameters;

[0114] Step 6: Input the multimodal fusion features obtained based on the optimal parameters at three different scales into the multi-level feature fusion classifier to perform the final classification task.

[0115] In this embodiment of the invention, step 6 includes:

[0116] To leverage feature information at different scales, the Multi-Level Feature Fusion Classifier (MFFC) constructs a hierarchical adaptive decision framework. It enhances classification robustness through multi-scale feature collaborative optimization, receiving fused features from different scales of the encoder in parallel. Each feature is input into a dedicated classification path. First, it passes through convolutional layers to extract key classification information. Then, batch normalization improves stability, and leaky linear rectifier units introduce nonlinearity to refine feature representation. A combined module is then added to enhance feature discriminative power. Finally, a normalization function maps each level of feature to its corresponding class probability, the expression of which is:

[0117] ;

[0118] in, Activated for softmax. ReLU nonlinear activation, The parameter r is a learnable parameter, and its values ​​are 1 and 2.

[0119] Finally, the decision information from the three category probability maps is fused by weighted summation to compensate for the limitations of single-level features, ultimately outputting a globally optimal classification result that balances fine-grained details with global structural consistency.

[0120] ;

[0121] Among them, the weights δ1, δ2, and δ3 are all learnable parameters that are automatically optimized during training based on the classification ability of the three-level features, and satisfy that δ1+δ2+δ3 approaches 1.

[0122] The final classification result R i By joint classification probability P i The maximum value index is determined by the expression:

[0123] ;

[0124] Where N is the total number of land cover categories.

[0125] The training objective of the model in this invention is achieved through a multi-task optimization framework consisting of adversarial loss and classification loss.

[0126] (1) Adversarial loss function: A GAN loss based on Wasserstein distance is used to improve training stability. For the discriminator (include and The optimization objective is to maximize the difference between the scores of real samples and generated samples. For the generator... Its antagonistic losses The aim is to minimize the degree to which its generated samples are judged as fake data, and its expression is:

[0127] ;

[0128] in, This refers to the batch size;

[0129] (2) Hierarchical classification loss function: classification loss It combines cross-entropy loss and mean squared error loss. Cross-entropy loss is used to constrain the classification results of the highest-level features:

[0130] ;

[0131] in, One-hot encoding of the actual label;

[0132] The mean squared error loss is used to constrain the consistency of classification results at different levels:

[0133] ;

[0134] This enables shallow and mid-level networks to learn discriminative features that are just as effective as those of deep networks;

[0135] The final classification loss is: ;

[0136] (3) Overall Loss: Considering the balance between image reconstruction and classification tasks in the generator, the overall loss function of the generator is expressed as follows:

[0137] ;

[0138] in, The weights are used to balance the importance of the two tasks, and an alternating optimization strategy is employed to train the entire network.

[0139] In this embodiment of the invention, the three datasets used in the parameter experiments are the Houston2013 dataset, the MUUFL dataset, and the Augsburg dataset. For these three datasets, the batch size, patch size, and number of training sessions were set to 64, 16×16, and 250, respectively, during the experiments. In addition, two important parameters need to be pre-determined to their optimal values: the weight coefficient λ and the learning rate LR. The weight coefficient λ is used to balance the adversarial loss and classification loss in the generative adversarial network (GAN); the learning rate LR is used to control the step size of the optimizer (Adam), affecting the update speed of the model parameters. When selecting the optimal values ​​of λ and LR, the values ​​of λ or LR are changed individually in the model, and the impact of different parameter values ​​on classification accuracy is observed to determine the optimal parameter values. The experimental results for the two parameters are as follows: Figure 4a , Figure 4b and Figure 4c as well as Figure 5a , Figure 5b and Figure 5c As shown, Overall Accuracy (OA) represents the proportion of correctly classified pixels out of the total number of pixels. It is the most intuitive and commonly used indicator. Average Accuracy (AA) first calculates the classification accuracy (i.e., mapping accuracy or user accuracy) for each land cover class, and then averages the accuracy of all classes. Kappa Statistics measures the consistency between the classification results and random classification. It considers not only correctly classified pixels but also the consistency caused by random chance. The higher the Kappa coefficient value, the better the classifier performs compared to random classification.

[0140] In this embodiment of the invention, to verify the effectiveness of the method, comparative experiments were conducted using three widely used HSI and LiDAR datasets to evaluate the performance of the proposed model (as shown in the table). The model was also compared with other state-of-the-art methods (Coupled CNN, CALC, M2FNet, DSHFNet, HLMamba, MICFNet, FDNet). To objectively evaluate classification performance, three evaluation metrics were used: overall accuracy (OA), average accuracy (AA), and Kappa coefficient. To ensure fairness and objectivity, all experiments were conducted under uniform conditions. The same training and test samples were used for all methods, and the network parameters of the compared methods followed the optimal settings in their original papers.

[0141] Tables 1, 2, and 3 show the overall classification accuracy (OA), average classification accuracy (AA), and Kappa coefficient (κ) of all the comparison methods on the three real remote sensing datasets: Houston2013, MUUFL, and Augsburg. The comparison results clearly show that the method of this invention achieves significant improvements in all evaluation metrics.

[0142] Table 1 Comparison of evaluation metrics on the Houston2013 dataset

[0143] ;

[0144] Table 2 Comparison of evaluation metrics on the MUUFL dataset

[0145] ;

[0146] Table 3 Comparison of evaluation metrics on the Augsburg dataset

[0147] .

[0148] In this embodiment of the invention, to verify the effectiveness of the module, a series of corresponding ablation experiments were constructed on three datasets. The experimental results are shown in Table 4. The classification performance of the spectral-spatial separation residual module (SSSRM), the spectral-spatial-elevation feature integration module (SSEFI), and the spectral normalization self-attention discriminator (SNSAD) were compared under different combination configurations (× in the table indicates not configured, √ indicates configured). The impact of each module on the overall performance of the model was analyzed one by one, thereby clarifying the core role of the above modules in the joint classification task.

[0149] Table 4 Ablation Experiment Results

[0150] .

[0151] In this invention, to effectively fuse hyperspectral image (HSI) and lidar (LiDAR) data to improve the accuracy of land cover classification, a deep fusion framework based on generative adversarial networks (GANs) is proposed. This framework aims to efficiently fuse HSI and LiDAR data to enhance land cover classification accuracy. Through three innovative designs—a decoupled three-branch encoder, a feature fusion module, and a structure consistency-aware discriminator—this framework systematically addresses key challenges such as feature extraction, multimodal fusion, and training stability. First, the encoder employs a decoupled three-branch encoder, extracting and enhancing the spectral and spatial features of HSI and the elevation features of LiDAR data through three branches. Second, the feature fusion module utilizes attention mechanisms at multiple scales to perform hierarchical deep fusion of the spectral-spatial features of HSI and the elevation features of LiDAR, fully leveraging the complementary advantages of the data. Finally, spectral normalization and self-attention mechanisms are introduced into the structure consistency-aware discriminator. Spectral normalization ensures the stability of GAN training, while the self-attention mechanism enables the model to focus on the most informative feature regions, thereby guiding the network to learn more discriminative features. Through an end-to-end GAN structure, the challenges of multimodal data feature extraction, deep fusion, and stable training are systematically solved, providing an effective solution for high-precision remote sensing image classification and improving the training stability and classification accuracy of generative adversarial networks (GANs).

[0152] The technical solution provided by this invention includes a method that extracts multi-scale features from multimodal data by decoupling a three-branch encoder, then uses a feature fusion module to perform attention-guided cross-modal fusion of three initial features at the same scale to obtain multimodal fused features at three scales; the deepest feature in the multimodal fused features is input into a weight-shared decoder to perform a data reconstruction task and obtain reconstructed pseudo-data; the original real data and the reconstructed pseudo-data are input into a structure consistency-aware discriminator for identification to distinguish between real and pseudo-data, and a discrimination loss is returned; the discrimination loss is input into the decoupling three-branch encoder for parameter updates, repeated multiple times to obtain optimal parameters; and the three multimodal fused features at different scales obtained based on the optimal parameters are input into a multi-level feature fusion classifier to perform the final classification task. This method improves the classification accuracy of the network and achieves adaptive fusion and enhancement of multimodal features.

[0153] The various steps in the embodiments of the present invention can be performed by an electronic device. This electronic device includes, but is not limited to, tablet computers, portable PCs, and desktop computers.

[0154] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the electronic device containing the computer-readable storage medium to perform the above-described embodiment of the joint classification method for hyperspectral images and lidar data.

[0155] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 6 As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the joint classification method for hyperspectral images and lidar data in the embodiment. To avoid repetition, it will not be described in detail here.

[0156] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0157] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0158] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A joint classification method for hyperspectral images and lidar data, characterized in that, The method includes: Step 1: Extract multi-scale features from the original image data of hyperspectral image (HSI) and lidar (LiDAR) by decoupling the three-branch encoder to obtain three initial features at different scales. Step 2: Use the feature fusion module to perform attention-guided cross-modal fusion of three initial features at the same scale to obtain multimodal fused features; Step 3: Input the deepest feature from the multimodal fusion features into the weight-sharing decoder to perform the data reconstruction task and obtain the reconstructed pseudo data; Step 4: Input the original real data and the reconstructed pseudo data into the structure consistency perception discriminator for identification, distinguish between real data and pseudo data, and return the discrimination loss; Step 5: Return the discrimination loss to the decoupled three-branch encoder, update the parameters, and repeat this process multiple times to obtain the optimal parameters; Step 6: Input the multimodal fusion features obtained based on the optimal parameters at three different scales into the multi-level feature fusion classifier to perform the final classification task; Multi-scale feature extraction is performed through a decoupled three-branch encoder in the decoupled guided fusion generator. The decoupled three-branch encoder employs a three-branch parallel coding architecture to extract three initial features at different scales from the raw image data of hyperspectral images and LiDAR. The three initial features are spectral features. Spatial features Elevation characteristics The three branches are spectral feature extraction, spatial feature extraction, and elevation feature extraction; each branch uses a hierarchical residual learning framework for multi-scale feature extraction. Step 2 includes: The feature fusion module in the decoupled guided fusion generator is used to process three initial features at the same scale. Attention-guided cross-modal fusion to obtain multimodal fusion features ,Right now The feature fusion module employs a hierarchical attention fusion mechanism to adaptively fuse multimodal features at each feature level. The feature fusion module performs the following processing flow: d. Spectral-spatial feature fusion; d1. Feature stitching: Combining spectral features at the same level Spatial features By splicing along the channel dimension, a joint feature is formed. ,in, Indicates the number of spectral channels. Represents the spatial height of the image. Indicates the spatial width of the image; d2. Feature projection: Dimensionality reduction and fusion are performed using 1×1 convolutional layers, batch normalization, and linear rectified units. ； in, ReLU represents the activation function, and BN represents batch normalization. Indicates the convolution kernel parameters; d3. Channel Attention Weighting: The channel attention mechanism is applied to the features to calculate the weight of each channel, identify and highlight the information of important channels, and generate a channel attention weight map; ； in, This indicates multiplication by channel. This represents the Sigmoid activation function. An MLP consists of two fully connected layers. e. LiDAR-guided cross-modal augmentation; e1. Dual-path feature extraction: Extraction of elevation information features Multi-scale statistical properties of elevation features are extracted using average pooling and max pooling. and : e2. Spatial Attention Generation: First, the results of average pooling and max pooling are concatenated to integrate multi-scale statistical information; then, the concatenated features are convolved, and finally converted into a spatial attention weight map through a mapping function. ； in, This represents a 1×1 convolution kernel. This represents the Sigmoid activation function; f. Modal interaction enhancement: The spectral-spatial weighted features processed by the upper branch and the elevation weighted features processed by the lower branch are multiplied element-wise again to obtain the fused features. : ； The final output is multimodal features. It contains the spectral-spatial features of HSI and the elevation information features of LiDAR.

2. The method according to claim 1, characterized in that, Step 1 includes: a. The spectral feature extraction branch processes the spectral information in the raw image data of the hyperspectral image (HSI), and consists of three cascaded spectral feature convolutional blocks. It extracts spectral features at different levels step by step through a three-level residual network (ResidualBlock). Each convolutional block addresses the vanishing gradient problem in deep networks through skip connections in the residual network, improving the depth and accuracy of feature extraction. Batch normalization is used to improve the training stability and convergence speed of the model. Then, 2×2 max pooling is applied after each level to downsample the spatial dimension. At the same time, the ReLU activation function is embedded in the residual network to enhance the expressive power of the model. b. The spatial feature extraction branch focuses on extracting the spatial structural features of hyperspectral images. It consists of three concatenated spatial feature convolutional blocks. The first convolutional block extracts local spatial features through 2D convolution, is then stabilized through batch normalization, and nonlinearity is introduced by a linear rectified unit. The second and third convolutional blocks have the same structure as the spectral feature convolutional blocks. They address the gradient vanishing problem in deep networks through a residual network, and further extract deep spatial features by combining batch normalization and max pooling. The final output is spatial features at different scales. ; c. The elevation feature extraction branch processes the raw image data from the lidar and extracts elevation information features. It consists of three concatenated elevation feature convolutional blocks, employing the same residual network structure as the spectral feature extraction branch to ensure feature hierarchy compatibility. This three-level residual network extracts multi-scale terrain features, i.e., elevation information features, from the raw LiDAR image data. .

3. The method according to claim 1, characterized in that, Step 3 includes: A dual-path decoding architecture with shared weights is adopted to achieve cross-modal feature collaborative reconstruction; the decoder fuses features at the deepest level of the encoder. As input, the spatial resolution of the feature map is first improved through upsampling. Then, local correlation information is extracted through convolutional layers to refine the feature representation. Batch normalization accelerates training and improves stability. Finally, a leaky linear rectified unit introduces nonlinearity, and this operation is repeated multiple times to reconstruct the original input data. The weights of the convolutional layers in both paths are shared, allowing the model to learn common feature representations of HSI and LiDAR. The decoder outputs the reconstructed data. It forms an adversarial learning loop with the original image data, while providing regularization constraints for the classification task, thus forming a multi-task collaborative optimization framework.

4. The method according to claim 3, characterized in that, Step 4 includes: Data discrimination is achieved through a structure consistency-aware discriminator. This module employs a dual-discriminator architecture, designing independent and structurally identical discriminators for HSI and LiDAR modalities, respectively. First, the two branches input the original real image data and the reconstructed pseudo data, respectively. Next, the data passes through convolutional layers to extract local features. Batch normalization is applied to the convolutional output features to accelerate model training and improve stability. Then, a leaky linear rectifier unit introduces nonlinearity to enhance the ability to express complex patterns. Furthermore, a self-attention mechanism captures long-distance dependencies of features, enhancing the perception of structural differences. Finally, the features are mapped to the discriminant space through a fully connected layer, and the probability of real or pseudo data is output through a sigmoid function. The dual-discriminator architecture integrates two core technologies to improve the stability and discriminative ability of adversarial training: h. Spectral normalization: All convolutional and fully connected layers in the discriminator are spectral normalized by constraining the Lipschitz constant of the discriminator function. i. Self-attention mechanism: A self-attention module is embedded after the deep feature map of the discriminator. The self-attention module is used to capture the long-distance dependency between any two locations in the feature map, so that the discriminator can evaluate the realism of local texture and judge the global structural consistency of the generated image, thereby providing gradient feedback to the generator. The expression for the feature enhancement of the self-attention mechanism is: ； Among them, query ,key ,value All are feature projections. , , respectively query ,key ,value The weights; Let K be the dimension of the key.

5. The method according to claim 4, characterized in that, Step 6 includes: To leverage feature information at different scales, a multi-level feature fusion classifier constructs a hierarchical adaptive decision framework. This framework enhances classification robustness through multi-scale feature collaborative optimization, receiving fused features from different scales of the encoder in parallel. Each feature is input into a dedicated classification path. First, it passes through convolutional layers to extract key classification information. Then, batch normalization improves stability, and leaky linear rectifier units introduce nonlinearity to refine feature representation. A combined module is then added to enhance feature discriminative power. Finally, a normalization function maps each level of feature to its corresponding class probability, the expression of which is: ； in, Activated for softmax. ReLU nonlinear activation, The parameter r is a learnable parameter, and its values ​​are 1 and 2. Finally, the decision information from the three category probability maps is fused by weighted summation to compensate for the limitations of single-level features, ultimately outputting a globally optimal classification result that balances fine-grained details with global structural consistency. ； Among them, the weights δ1, δ2, and δ3 are all learnable parameters that are automatically optimized during training based on the classification ability of the three-level features, and satisfy that δ1+δ2+δ3 approaches 1. The final classification result R i By joint classification probability P i The maximum value index is determined by the expression: ； Where N is the total number of land cover categories.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the joint classification method for hyperspectral image and lidar data as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the joint classification method for hyperspectral image and lidar data as described in any one of claims 1 to 5.

Images