Joint classification method for hyperspectral image and laser radar data

Abstract

The invention relates to the technical field of deep learning, and particularly provides a joint classification method for hyperspectral images and laser radar data. The method comprises the following steps: firstly, extracting multi-scale features of multi-modal data through a decoupling three-branch encoder, and then carrying out attention guidance cross-modal fusion on three initial features of the same scale through an autonomously designed feature fusion module to obtain three multi-modal fusion features of different scales; inputting the deepest fusion feature into a weight sharing decoder, and obtaining pseudo data through a reconstruction task; and the real data and the pseudo data are identified by the structure consistency perception discriminator, and the discrimination loss is returned, so that the encoder parameters are updated, and iteration is performed to the optimal. And finally, inputting the three-scale fusion features obtained based on the optimal parameters into a multi-level feature fusion classifier to complete classification, thereby effectively improving the classification precision, and realizing the adaptive fusion and enhancement of the multi-modal features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a joint classification method for hyperspectral image and laser radar data. BACKGROUND

[0002] With the rapid development of aviation and space remote sensing technology, multi-source remote sensing data plays an increasingly important role in natural resource investigation, ecological environment monitoring, urban planning and construction, disaster prevention and control, and agricultural production. However, a single data source is often difficult to fully depict the features of ground objects in complex environments. Although hyperspectral (HIS) image data can distinguish material components, it has limitations in spatial structure description. Laser radar (LiDAR) image data can accurately represent three-dimensional geometric shapes, but lacks fine-grained spectral properties. This complementary relationship makes the joint classification of hyperspectral and laser radar a frontier direction of current remote sensing research. Although the fusion classification technology of hyperspectral and laser radar has made great progress, the existing technology still has the following shortcomings: (1) Modality feature entanglement and independence problem: the existing deep framework generally has cross interference of modality information in the feature extraction stage, lacks effective separation of spectral, spatial and elevation information, and affects the purity and efficiency of subsequent fusion; (2) Limitations of cross-modal guidance paradigm: the current attention interaction mechanism is not fine enough, and cannot fully realize the robust and accurate guidance of LiDAR structure prior to HSI spectral features; (3) Stability and semantic preservation of adversarial training: when using generative adversarial networks (GAN) for feature enhancement or domain adaptation, the design of the discriminator is difficult to balance the training stability and sensitivity to high-level semantic information, which restricts the quality of the generated features. SUMMARY

[0003] Therefore, the present application provides a joint classification method for hyperspectral image and laser radar data to improve the classification accuracy of the network and realize adaptive fusion and enhancement of multi-modal features.

[0004] In a first aspect, the present application provides a joint classification method for hyperspectral image and laser radar data, which comprises: Step 1: performing multi-scale feature extraction on the original image data of hyperspectral image HSI and laser radar LiDAR by decoupling a three-branch encoder to obtain three initial features at different scales; Step 2: performing attention-guided cross-modal fusion on the three initial features at the same scale by using a feature fusion module to obtain multi-modal fusion features; Step 3, input the deepest feature in the multi-modal fusion feature into the weight-shared decoder to perform a data reconstruction task, and obtain reconstructed pseudo data; Step 4, input the original image real data and the reconstructed pseudo data into the structure consistency perception discriminator to distinguish the real data and the pseudo data, and return a discrimination loss; Step 5, return the discrimination loss to the decoupled three-branch encoder for parameter updating, and repeat multiple times to obtain optimal parameters; Step 6, input the three multi-modal fusion features of different scales obtained based on the optimal parameters into the multi-level feature fusion classifier to perform a final classification task.

[0005] Optionally, the step 1 comprises: The decoupled three-branch encoder in the decoupled guided fusion generator is used for multi-scale feature extraction, and the decoupled three-branch encoder adopts a three-branch parallel encoding architecture to extract three initial features of different scales from the original image data of the hyperspectral image and the laser radar The three initial features are respectively a spectral feature , a spatial feature , and an elevation feature ; the three branches are respectively a spectral feature extraction branch, a spatial feature extraction branch, and an elevation feature extraction branch; each branch adopts a hierarchical residual learning framework for multi-scale feature extraction: a. The spectral feature extraction branch is used for processing spectral information in the original image data of the hyperspectral image HSI, and is composed of 3 spectral feature convolution blocks in series; it extracts spectral features of different levels step by step through a three-level residual network ResidualBlock Each convolution block solves the gradient disappearance problem of the deep network through the skip connection structure in the residual network, improves the depth and accuracy of feature extraction, and uses batch normalization to improve the training stability and convergence speed of the model, followed by 2x2 max pooling for spatial dimension down-sampling, and the residual network is embedded with an activation function ReLU to enhance the expression ability of the model; b. The spatial feature extraction branch focuses on extracting spatial structure features from the hyperspectral image, and is composed of 3 spatial feature convolution blocks in series, wherein the first convolution block extracts spatial local features through two-dimensional convolution, and then stabilizes the training through batch normalization and introduces nonlinearity through a linear rectifier unit; the second and third convolution blocks are consistent with the structure of the spectral feature convolution block, and solve the gradient disappearance problem of the deep network through the residual network, and further extract deep spatial features in combination with batch normalization and max pooling, and finally output spatial features of different scales ; c. The elevation feature extraction branch is used for processing the original image data of the laser radar to extract elevation information features ; it consists of three elevation feature convolutional blocks in series, adopts the same residual network structure as the spectral feature extraction branch to ensure feature level compatibility, and uses a three-level residual network to extract multi-scale terrain features, i.e. elevation information features, from the original image data of LiDAR .

[0006] Optionally, the step 2 comprises: using a feature fusion module in the decoupled guided fusion generator to fuse the three initial features at the same scale performing attention-guided cross-modal fusion to obtain multi-modal fusion features , i.e. ; the feature fusion module adopts a hierarchical attention fusion mechanism to perform adaptive fusion of multi-modal features, at each feature level , the feature fusion module performs the following processing flow: d. spectral-spatial feature fusion d1. feature concatenation: concatenate the spectral features and spatial features at the same level in the channel dimension to form joint features ; d2. feature projection: dimension reduction fusion is performed through a 1x1 convolutional layer, batch normalization and a linear rectifier unit: ; wherein, ReLU represents the ReLU activation function, BN represents batch normalization, and represents the convolution kernel parameter; d3. channel attention weighting: applying the channel attention mechanism to the features calculates the weight of each channel, identifies and highlights the information of important channels, and generates a channel attention weight map; ; wherein, represents channel-wise multiplication, Sigmoid represents the Sigmoid activation function, and MLP consists of two fully connected layers; e. LiDAR-guided cross-modal enhancement e1. dual-channel feature extraction: for the elevation information features extract the multi-scale statistical characteristics of the elevation features through average pooling and maximum pooling: ; e2. spatial attention generation: first, concatenate the results of average pooling and maximum pooling to integrate multi-scale statistical information; then, perform convolution operation on the concatenated features, and convert them into a spatial attention weight map through a mapping function: ; wherein, represents a 1x1 convolution kernel, represents a Sigmoid activation function; f, modal interaction enhancement: the spectral-spatial weighted features processed by the upper branch are multiplied with the elevation weighted features processed by the lower branch again, and finally the fused features are obtained : ; the final output multi-modal features , which include the spectral-spatial features of HSI and the elevation information features of LiDAR.

[0007] Optionally, the step 3 comprises: a dual-path decoding architecture with shared weights is adopted to realize cross-modal feature collaborative reconstruction; the decoder takes the deepest level fused features of the encoder as input first, the spatial resolution of the feature map is improved through upsampling operation, then local correlation information is extracted through convolution layer to refine feature expression, and then batch normalization is used to accelerate training and improve stability, finally, the nonlinearity is introduced by the leaky linear rectifier unit, and this operation is repeated several times to realize the reconstruction of the original input data, and the weights of the convolution layers in the two paths are shared, and the shared layers are used to make the model learn the common feature representation of HSI and LiDAR; the reconstructed data output by the decoder forms an adversarial learning loop with the original image data, and provides regularization constraints for the classification task, forming a multi-task collaborative optimization framework.

[0008] Optionally, the step 4 comprises: data discrimination is performed by a structure consistency perception discriminator, which adopts a dual discriminator architecture, and independently designs discriminators with the same structure for HSI and LiDAR modalities; first, the two branches input the original image real data and the pseudo data generated by reconstruction respectively; then, the data is extracted through the convolution layer to extract the local features, and the batch normalization is performed on the convolution output features to accelerate the model training and improve the stability, then the nonlinearity is introduced through the leaky linear rectifier unit to enhance the expression ability of complex patterns; then the long-distance dependence of the features is captured through the self-attention mechanism to enhance the perception of structural differences; finally, the features are mapped to the discrimination space through the full connection layer, and the probability of real data or pseudo data is output through the S-shaped function; the dual discriminator architecture integrates two core technologies to improve the stability and discrimination ability of adversarial training: h, spectral normalization: all convolution layers and full connection layers in the discriminator adopt spectral normalization, which constrains the Lipschitz constant of the discriminator function; i. Self-attention mechanism: after the deep feature map of the discriminator, a self-attention module is embedded; the self-attention module is used to capture the long-distance dependence between any two positions in the feature map, so that the discriminator evaluates the authenticity of the local texture and judges the global structural consistency of the generated image, thereby providing gradient feedback for the generator, and the expression of the self-attention mechanism feature enhancement is: ; Wherein, the query , the key , and the value are feature projections, 、 、 The weights of the query , the key , and the value are respectively; The dimension of the key K.

[0009] Optionally, the step 6 comprises: In order to utilize the feature information of different scales, the multi-level feature fusion classifier constructs a hierarchical adaptive decision framework, which improves the classification robustness through multi-scale feature collaborative optimization ; Each feature is input into a dedicated classification path, first through a convolutional layer to mine classification key information, then through batch normalization to improve stability, and a leaky linear rectifier unit to introduce nonlinearity to refine feature expression, and a superposition combination module to strengthen feature discriminability, and then through a normalization function to map each level of feature to a corresponding class probability, which is expressed as: ; Wherein, is the softmax activation, is the ReLU nonlinear activation, is a learnable parameter, and r takes the value of 1 or 2; Finally, the decision information of the three class probability maps is fused by weighted summation to make up for the limitations of single-level features, and the final output is the global optimal classification result that takes into account the fine-grained details and global structural consistency: ; Wherein, the weights δ1, δ2, and δ3 are all learnable parameters, which are automatically optimized in training according to the classification ability of the three levels of features, and satisfy δ1+δ2+δ3 tends to 1; The final classification result R i is determined by the maximum value index in the joint classification probability P i , which is expressed as: ; Wherein, N is the total number of ground object categories.

[0010] In a second aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium comprising a stored program, wherein the program, when executed, controls a device in which the computer readable storage medium is located to perform the method for joint classification of hyperspectral images and lidar data in the first aspect or any possible implementation manner of the first aspect.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs comprise instructions that, when executed by the device, cause the device to perform the method for joint classification of hyperspectral images and lidar data in the first aspect or any possible implementation manner of the first aspect.

[0012] The technical scheme provided by the present application, the method comprises: performing multi-scale feature extraction on multi-modal data by a decoupled three-branch encoder; then performing attention-guided cross-modal fusion on three initial features of the same scale by a feature fusion module to obtain multi-modal fusion features at three scales; inputting the deepest features in the multi-modal fusion features into a weight-shared decoder to perform a data reconstruction task and obtain reconstructed pseudo data; inputting the original image real data and the reconstructed pseudo data into a structure consistency perception discriminator for discrimination to distinguish the real data and the pseudo data and return a discrimination loss; inputting the discrimination loss into the decoupled three-branch encoder for parameter updating, and repeating multiple times to obtain optimal parameters; inputting the three multi-modal fusion features of different scales obtained based on the optimal parameters into a multi-level feature fusion classifier to perform a final classification task, which improves the classification accuracy of the network and realizes adaptive fusion and enhancement of multi-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 A flowchart of the method for joint classification of hyperspectral images and lidar data provided by the embodiment of the present application is provided. Figure 2 A schematic diagram of obtaining three initial features at different scales provided by the embodiment of the present application is provided. Figure 3A schematic diagram of obtaining a multi-modal fusion feature provided by an embodiment of the present application is shown in FIG. 1. Figure 4a A schematic diagram of weight coefficient λ experimental results of the Houston2013 dataset provided by an embodiment of the present application is shown in FIG. 2. Figure 4b A schematic diagram of weight coefficient λ experimental results of the MUUFL dataset provided by an embodiment of the present application is shown in FIG. 3. Figure 4c A schematic diagram of weight coefficient λ experimental results of the Augsburg dataset provided by an embodiment of the present application is shown in FIG. 4. Figure 5a A schematic diagram of learning rate LR experimental results of the Houston2013 dataset provided by an embodiment of the present application is shown in FIG. 5. Figure 5b A schematic diagram of learning rate LR experimental results of the MUUFL dataset provided by an embodiment of the present application is shown in FIG. 6. Figure 5c A schematic diagram of learning rate LR experimental results of the Augsburg dataset provided by an embodiment of the present application is shown in FIG. 7. Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0015] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0016] It should be clear that the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0017] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application are also intended to include the plural forms unless the context clearly indicates otherwise.

[0018] It should be understood that the term "and / or" as used herein merely describes an associated relationship between associated objects, and can represent three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0019] Depending on the context, the word "if" as used herein can be interpreted as meaning "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as meaning "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0020] Figure 1 A flowchart of a method for joint classification of hyperspectral image and LiDAR data provided by an embodiment of the present application is shown in Figure 1 , which comprises: The present application proposes a multi-path deep learning network architecture for solving the multi-modal joint classification task of hyperspectral image (HSI) and LiDAR. The architecture realizes adaptive fusion and enhancement of multi-modal features through collaborative optimization of feature representation learning and joint distribution modeling. As shown in Figure 1 , the network architecture mainly consists of two parts: a generative adversarial network for feature learning and data reconstruction, which includes a decoupled guided fusion generator and a structure consistency perception discriminator; and a multi-level feature fusion classifier for final classification decision. The generative adversarial network part drives feature learning through an unsupervised image reconstruction task, while the multi-level feature fusion classifier optimizes these features through a supervised classification task. The two parts are collaboratively trained through a joint loss function.

[0021] The decoupled guided fusion generator is the core of the entire model framework, which adopts an encoder-decoder structure. The decoupled three-branch encoder is used to extract features of different scales from multi-modal data, and the feature fusion module is used to adaptively fuse three different features (spectral features, spatial features and elevation features) of the same scale. The weight sharing decoder uses the deepest fused features to obtain the reconstructed pseudo data.

[0022] Step 1: Multi-scale feature extraction of original image data of hyperspectral image HSI and LiDAR LiDAR through decoupled three-branch encoder, to obtain three initial features of different scales.

[0023] In the embodiment of the present application, as shown in Figure 2 , step 1 comprises: The decoupling three-branch encoder in the decoupled guidance fusion generator is used for multi-scale feature extraction, and the decoupled three-branch encoder adopts a three-branch parallel encoding architecture to extract three initial features at different scales from the original image data of the hyperspectral image and the laser radar The three initial features are respectively spectral features , spatial features , and elevation features The three branches are respectively a spectral feature extraction branch, a spatial feature extraction branch, and an elevation feature extraction branch, and each branch adopts a hierarchical residual learning framework for multi-scale feature extraction a. The spectral feature extraction branch is used for processing spectral information in the original image data of the hyperspectral image HSI, and is composed of three spectral feature convolution blocks in series, and different levels of spectral features are extracted step by step through a three-level residual network ResidualBlock Each convolution block solves the gradient disappearance problem of the deep network through the skip connection structure in the residual network, improves the depth and accuracy of feature extraction, uses batch normalization to improve the training stability and convergence speed of the model, and then each level is followed by 2x2 maximum pooling for spatial dimension down-sampling, and the residual network is embedded with an activation function ReLU to enhance the expression ability of the model b. The spatial feature extraction branch focuses on extracting spatial structure features from the hyperspectral image, and is composed of three spatial feature convolution blocks in series, wherein the first convolution block extracts spatial local features through two-dimensional convolution, and then stabilizes the training through batch normalization and introduces nonlinearity through a linear rectifier unit; the second and third convolution blocks are consistent with the structure of the spectral feature convolution block, and the depth network gradient disappearance problem is solved through the residual network, and the batch normalization and maximum pooling are combined to further extract deep spatial features, and finally output spatial features at different scales c. The elevation feature extraction branch is used for processing the original image data of the laser radar to extract elevation information features It is composed of three elevation feature convolution blocks in series, and adopts the same residual network structure as the spectral feature extraction branch to ensure feature level compatibility, and uses a three-level residual network to extract multi-scale terrain features, i.e. elevation information features, from the original image data of the LiDAR .

[0024] Step 2, using a feature fusion module to perform attention-guided cross-modal fusion on the three initial features at the same scale to obtain multi-modal fusion features.

[0025] In the embodiment of the application, the feature fusion module is a self-designed module.

[0026] In the embodiment of the application, as Figure 3 ​As shown, step 2 includes: The feature fusion module in the decoupled guided fusion generator is used to fuse three initial features on the same scale Attention-guided cross-modal fusion is performed to obtain multi-modal fusion features That is The feature fusion module adopts a hierarchical attention fusion mechanism to adaptively fuse multi-modal features. At each feature level The feature fusion module performs the following processing flow: d, spectral-spatial feature fusion d1, feature concatenation: concatenate the spectral features and spatial features in the same level in the channel dimension to form joint features ; d2, feature projection: dimension reduction fusion is performed through a 1x1 convolution layer, batch normalization and a linear rectifier unit: ; wherein ReLU represents the ReLU activation function, BN represents batch normalization, and represents the convolution kernel parameter; d3, channel attention weighting: apply the channel attention mechanism to the features to calculate the weight of each channel, identify and highlight the important channel information, and generate a channel attention weight map; ; wherein represents channel-wise multiplication, Sigmoid represents the Sigmoid activation function, and MLP is composed of two fully connected layers; e, LiDAR guided cross-modal enhancement e1, dual-channel feature extraction: extract the elevation information features through average pooling and maximum pooling to extract the multi-scale statistical characteristics of the elevation features: ; e2, spatial attention generation: first, concatenate the average pooling and maximum pooling results to integrate the multi-scale statistical information; then, perform convolution operation on the concatenated features, and convert them into a spatial attention weight map through a mapping function: ; wherein represents a 1x1 convolution kernel, Sigmoid represents the Sigmoid activation function; f, modal interaction enhancement: the spectral-spatial weighted features processed by the upper branch are again multiplied element-wise with the elevation weighted features processed by the lower branch to obtain the final fusion features : ; Final output multi-modal feature , which contains the spectral-spatial feature of HSI and the elevation information feature of LiDAR, is input into a hierarchical weighted classifier for final decision, realizing end-to-end joint optimization.

[0027] Step 3, input the deepest feature in the multi-modal fusion feature into the weight sharing decoder to perform the data reconstruction task and obtain the reconstructed pseudo data.

[0028] In the embodiment of the application, step 3 comprises: A dual-path decoding architecture with shared weights is adopted to realize cross-modal feature collaborative reconstruction; the decoder takes the deepest level fusion feature of the encoder as input , first improves the spatial resolution of the feature map through upsampling operation, then extracts local correlation information through convolution layer to refine feature expression, and finally introduces nonlinearity through leaky linear rectifier unit, and this operation is repeated multiple times to realize reconstruction of the original input data, and the weights of the convolution layers in the two paths are shared, and the shared layers are used to make the model learn the common feature representation of HSI and LiDAR; the reconstructed data output by the decoder forms an adversarial learning loop with the original image data, and provides regularization constraints for the classification task, forming a multi-task collaborative optimization framework.

[0029] Step 4, input the original image real data and the reconstructed pseudo data into the structure consistency perception discriminator for discrimination to distinguish the real data and the pseudo data, and return the discrimination loss.

[0030] In the embodiment of the application, step 4 comprises: Data discrimination is performed through the structure consistency perception discriminator, which adopts a dual discriminator architecture, and independent and identical structure discriminators are designed for HSI and LiDAR modalities respectively; first, the two branches input the original image real data and the reconstructed pseudo data respectively; then, the data is extracted through the convolution layer to extract local features, the batch normalization is performed on the convolution output features to accelerate the model training and improve the stability, then the nonlinearity is introduced through the leaky linear rectifier unit to enhance the expression ability for complex patterns; then the long-distance dependence of the features is captured through the self-attention mechanism to enhance the perception of structural differences; finally, the features are mapped to the discrimination space through the full connection layer, and the probability of real data or pseudo data is output through the S-shaped function, so as to realize data discrimination; the dual discriminator architecture integrates two core technologies to improve the stability and discrimination ability of the adversarial training: h、Spectral Normalization: All convolutional layers and fully connected layers in the discriminator are adopted with spectral normalization, which effectively prevents gradient explosion and mode collapse in the training process by constraining the Lipschitz constant of the discriminator function, making the adversarial training more stable; i、Self-Attention: A self-attention module is embedded after the deep feature map of the discriminator; the self-attention module is used to capture the long-distance dependency relationship between any two positions in the feature map, so that the discriminator can evaluate the authenticity of the local texture and judge the global structural consistency of the generated image, thereby providing more comprehensive and accurate gradient feedback to the generator, prompting the decoupled three-branch encoder in the generator to extract more discriminative features. The expression of the self-attention mechanism feature enhancement is: ; wherein, the query , the key , and the value are feature projections, 、 、 the weights of the query , the key , and the value ; is the dimension of the key K.

[0031] Step 5, return the discriminant loss to the decoupled three-branch encoder for parameter updating, and repeat multiple times to obtain the optimal parameters; Step 6, input the three different scale multi-modal fusion features obtained based on the optimal parameters into the multi-level feature fusion classifier to perform the final classification task.

[0032] In the embodiment of the application, step 6 comprises: In order to utilize the feature information of different scales, the multi-level feature fusion classifier MFFC constructs a hierarchical adaptive decision framework, which improves the classification robustness through multi-scale feature collaborative optimization, and receives the fusion features of different scales from the encoder in parallel ; each feature is input into a dedicated classification path, first passes through a convolutional layer to mine classification key information, then passes through a batch normalization to improve stability, a leaky linear rectifier unit to introduce nonlinearity to refine feature expression, and a superposition combination module to strengthen feature discriminability, and then maps each level of feature to the corresponding class probability through a normalization function, and the expression is: ; wherein, is a softmax activation, is a ReLU nonlinear activation, The r is 1, 2 for learnable parameters; Finally, the decision information of the three category probability maps is fused by weighted summation, the limitations of single-level features are made up, and finally the global optimal classification result considering fine-grained details and global structure consistency is output: ; Wherein, the weights δ1, δ2, δ3 are all learnable parameters, which are automatically optimized in training according to the classification ability of three levels of features, and satisfy δ1+δ2+δ3 tends to 1; The final classification result R i Is determined by the maximum index of the joint classification probability P i , and its expression is: ; Wherein, N is the total number of ground object categories.

[0033] The training target of the model is realized by a multi-task optimization framework composed of adversarial loss and classification loss: (1) Adversarial loss function: the GAN loss based on Wasserstein distance is adopted to improve the training stability. For the discriminator and , the optimization goal is to maximize the difference between the scores of real samples and generated samples. For the generator , the adversarial loss aims to minimize the degree that its generated samples are judged as false data, and its expression is: ; Wherein, n is the batch size; (2) Hierarchical classification loss function: the classification loss Combines cross-entropy loss and mean square error loss. ; Wherein, y is the one-hot encoding of the real label; The mean square error loss is used to constrain the consistency of different levels of classification results: ; Promote the shallow and middle layer network to learn the same effective discriminative features as the deep layer network; The final classification loss is: ; ​(3) Overall loss: Considering the balance between the image reconstruction task and the classification task in the generator, the overall loss function expression of the generator is: ; wherein, is a weight coefficient for balancing the importance of the two tasks, and the entire network is trained by using an alternating optimization strategy.

[0034] In the embodiments of the present application, the three data sets used in the parameter experiment in the present application are the Houston2013 data set, the MUUFL data set and the Augsburg data set. For the three data sets, the batch size, the patch size and the number of training sessions are set to 64, 16x16 and 250 respectively in the experiment process. In addition, there are two important parameters that need to be determined optimally, i.e. the weight coefficient λ and the learning rate LR: the weight coefficient λ is used to balance the adversarial loss and the classification loss in the generative adversarial network (GAN); the learning rate LR is used to control the step size of the optimizer (Adam) and affect the update speed of the model parameters. When selecting the optimal values of λ and LR, the value of λ or LR in the model is changed alone, and the influence of different parameter values on the classification accuracy is observed to determine the optimal parameter value. The experimental results of the two parameters are shown in Figure 4a 、 Figure 4b and Figure 4c and Figure 5a 、 Figure 5b and Figure 5c , the overall classification accuracy (OA) represents the proportion of the number of correctly classified pixels to the total number of pixels. It is the most intuitive and most commonly used indicator; the average classification accuracy (AA) first calculates the classification accuracy of each class of ground objects (i.e. the mapping accuracy or user accuracy), and then averages the accuracy of all classes; the Kappa coefficient (Kappa Statistics) is used to measure the consistency of the classification results compared with random classification, which not only considers the correctly classified pixels, but also considers the consistency caused by random chance. The higher the Kappa coefficient value, the better the effect of the classifier than the random classification.

[0035] In the embodiments of the present application, in order to verify the effectiveness of the method of the present application, three widely used HSI and LiDAR datasets are used in the comparative experiment to evaluate the performance of the model of the present application (such as Proposed in the table), and compared with other advanced methods (Coupled CNN, CALC, M2FNet, DSHFNet, HLMamba, MICFNet, FDNet). In order to objectively evaluate the classification performance, three evaluation indexes are used: overall accuracy (OA), average accuracy (AA) and Kappa coefficient. In order to ensure the fairness and objectivity of the comparison, all experiments are carried out under uniform conditions. The same training samples and test samples are divided for all methods, and the network parameters of the comparative methods all follow the optimal settings in their original papers.

[0036] Tables 1, 2 and 3 respectively show the overall classification accuracy (OA), average classification accuracy (AA) and Kappa coefficient (κ) of all comparative methods on the three real remote sensing datasets of Houston2013, MUUFL and Augsburg. From the comparison results, it can be clearly observed that the method of the present application has achieved significant improvement in all evaluation indexes.

[0037] Table 1 Comparison results of evaluation indexes on Houston2013 dataset ; Table 2 Comparison results of evaluation indexes on MUUFL dataset ; Table 3 Comparison results of evaluation indexes on Augsburg dataset .

[0038] In the embodiments of the present application, in order to verify the effectiveness of the module of the present application, a series of corresponding ablation experiments are constructed on the three datasets. The experimental results are shown in Table 4, which compares the classification performance of the model under different combination configurations (x in the table represents not configured, and represents configured) of the spectral-spatial separation residual module (SSSRM), the spectral-spatial-elevation feature integration module (SSEFI) and the spectral normalization self-attention discriminator (SNSAD), and analyzes the influence of each module on the overall performance of the model one by one, thereby elucidating the core role of the above-mentioned modules in the joint classification task.

[0039] Table 4 Ablation experiment results .

[0040] In the embodiment of the present application, in order to effectively fuse hyperspectral image (HSI) and LiDAR data to improve the accuracy of ground object classification, the present application proposes a deep fusion framework based on generative adversarial network (GAN), which aims to efficiently fuse hyperspectral image (HSI) and LiDAR data to improve the accuracy of ground object classification. The framework systematically solves the key challenges of feature extraction, multi-modal fusion and training stability through three innovative designs: decoupled three-branch encoder, feature fusion module and structure consistency perception discriminator. First, the encoder adopts a decoupled three-branch encoder, which extracts and enhances the spectral and spatial features of hyperspectral HSI and the elevation features of LiDAR data through a three-branch network. Second, the feature fusion module uses attention mechanism at multiple scales to perform hierarchical deep fusion of the spectral-spatial features of HSI and the elevation features of LiDAR, fully exploiting the complementary advantages of the data. Finally, the structure consistency perception discriminator is introduced with spectral normalization and self-attention mechanism. Spectral normalization ensures the stability of GAN training, while self-attention mechanism enables the model to focus on the feature area with the most information, guiding the network to learn more discriminative features; through an end-to-end GAN structure, the challenges of multi-modal data feature extraction, deep fusion and stable training are systematically solved, providing an effective solution for high-precision remote sensing image classification and improving the training stability and classification accuracy of generative adversarial network (GAN).

[0041] In the technical solution provided by the present application, the method includes multi-scale feature extraction of multi-modal data by a decoupled three-branch encoder, followed by attention-guided cross-modal fusion of three initial features at the same scale by a feature fusion module to obtain multi-modal fusion features at three scales; the deepest features in the multi-modal fusion features are input to a weight-shared decoder to perform a data reconstruction task and obtain reconstructed pseudo data; the original image real data and the reconstructed pseudo data are input to a structure consistency perception discriminator for discrimination to distinguish real data and pseudo data and return a discrimination loss; the discrimination loss is input to the decoupled three-branch encoder for parameter updating, repeated multiple times to obtain optimal parameters; and the multi-modal fusion features at three different scales obtained based on the optimal parameters are input to a multi-level feature fusion classifier to perform a final classification task, which improves the classification accuracy of the network and realizes adaptive fusion and enhancement of multi-modal features.

[0042] The various steps of the embodiment of the present application can be performed by an electronic device. The electronic device includes but is not limited to a tablet computer, a portable PC, a desktop computer, etc.

[0043] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium comprises a stored program, wherein when the program is running, the electronic device where the computer readable storage medium is located executes the embodiment of the above method for joint classification of hyperspectral image and laser radar data.

[0044] Figure 6 An electronic device provided by the embodiment of the present application is shown in a schematic diagram as Figure 6 The electronic device 21 comprises a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and capable of running on the processor 211, and the computer program 213 is executed by the processor 211 to implement the method for joint classification of hyperspectral image and laser radar data in the embodiment, and details are not described herein to avoid repetition.

[0045] The electronic device 21 comprises, but is not limited to, the processor 211 and the memory 212. Figure 6 The electronic device 21 is merely an example and does not constitute a limitation on the electronic device 21, and can comprise more or fewer components than the diagram, or combine certain components, or different components, for example, the electronic device can also comprise an input / output device, a network access device, a bus, and the like.

[0046] The processor 211 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0047] The memory 212 can be an internal storage unit of the electronic device 21, for example, a hard disk or a memory of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 21. Further, the memory 212 can also include both the internal storage unit and the external storage device of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by the network device. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0048] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0049] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for joint classification of hyperspectral images and lidar data, characterized in that, The method comprises: Step 1, multi-scale feature extraction of hyperspectral image HSI and raw image data of laser radar LiDAR by decoupling three-branch encoder, to obtain three initial features at different scales; Step 2, attention-guided cross-modal fusion of three initial features at the same scale by using a feature fusion module to obtain multi-modal fusion features; Step 3, input the deepest features in the multi-modal fusion features into a weight-shared decoder to perform a data reconstruction task and obtain reconstructed pseudo data; Step 4, input the original image real data and the reconstructed pseudo data into a structure consistency perception discriminator for discrimination to distinguish real data and pseudo data and return a discrimination loss; Step 5, return the discrimination loss to the decoupling three-branch encoder for parameter updating, and repeat multiple times to obtain optimal parameters; Step 6, input the three multi-modal fusion features at different scales obtained based on the optimal parameters into a multi-level feature fusion classifier to perform a final classification task.

2. The method of claim 1, wherein, The step 1 comprises: The decoupling three-branch encoder in the decoupling guided fusion generator is used for multiscale feature extraction, and the decoupling three-branch encoder adopts a three-branch parallel encoding architecture to extract three initial features at different scales from the original image data of the hyperspectral image and the laser radar , the three initial features are respectively spectral features , spatial features , and elevation features ; the three branches are respectively a spectral feature extraction branch, a spatial feature extraction branch, and an elevation feature extraction branch; each branch adopts a hierarchical residual learning framework for multiscale feature extraction: a、The spectral feature extraction branch is used for processing spectral information in original image data of the hyperspectral image HSI and is composed of three spectral feature convolution blocks in series; the spectral feature extraction branch extracts spectral features at different levels step by step through a three-level residual block Each convolution block solves the gradient disappearance problem of the deep network through a skip connection structure in the residual network, improves the depth and accuracy of feature extraction, and uses batch normalization to improve the training stability and convergence speed of the model, followed by 2*2 max pooling for spatial dimension down-sampling, and a ReLU activation function embedded in the residual network to enhance the expression ability of the model. b. The spatial feature extraction branch focuses on extracting the spatial structure features of the hyperspectral image, which is composed of three spatial feature convolution blocks in series. The first convolution block extracts spatial local features through two-dimensional convolution, and then stabilizes the training through batch normalization and introduces nonlinearity through a linear rectifier unit. The second and third convolution blocks are consistent with the structure of the spectral feature convolution block, which solves the gradient disappearance problem of the deep network through a residual network, further extracts deep spatial features through batch normalization and maximum pooling, and finally outputs spatial features of different scales ; c. The elevation feature extraction branch is used to process the raw image data of the laser radar to extract elevation information features ; it is composed of three elevation feature convolution blocks in series, adopts the same residual network structure as the spectral feature extraction branch to ensure feature level compatibility, and uses a three-level residual network to extract multi-scale terrain features, i.e. elevation information features, from the raw image data of the LiDAR .

3. The method of claim 2, wherein, The step 2 comprises: The feature fusion module in the decoupled guided fusion generator is used for fusing three initial features on the same scale Attention-guided cross-modal fusion is performed to obtain a multi-modal fusion feature That is The feature fusion module adopts a hierarchical attention fusion mechanism to perform adaptive fusion of multi-modal features, and at each feature level The feature fusion module performs the following processing procedure: d, spectral-spatial feature fusion; d1, feature concatenation: concatenating spectral features of the same level with spatial features in the channel dimension, forming joint features ; d2, feature projection: dimensionality reduction fusion through a 1x1 convolution layer, batch normalization and a linear rectifier unit: ； wherein, denotes a ReLU activation function, BN denotes batch normalization, denotes a convolution kernel parameter; d3, channel attention weighting: applying a channel attention mechanism to each channel weight to identify and highlight important channel information and generate a channel attention weight map; ； wherein, denotes a channel-wise multiplication, denotes a Sigmoid activation function, and the MLP consists of two fully connected layers. e, LiDAR-guided cross-modal enhancement; e1, two-way feature extraction: on elevation information features extracting multi-scale statistical properties of elevation features by average pooling and max pooling: ； e2, spatial attention generation: first, splice the results of average pooling and maximum pooling to integrate multi-scale statistical information; then, perform convolution operation on the spliced features, and convert them into a spatial attention weight map through a mapping function: ； wherein, denotes a 1 x 1 convolution kernel, denotes a Sigmoid activation function; f、modal interaction enhancement: the spectral-spatial weighted features processed by the upper branch and the elevation weighted features processed by the lower branch are multiplied again element by element to obtain the final fusion features : ； Final output multimodal features which includes spectral-spatial features of HSI and elevation information features of LiDAR.

4. The method of claim 3, wherein, The step 3 comprises: A dual-path decoding architecture with shared weights is adopted to realize cross-modal feature collaborative reconstruction; the decoder fuses features at the deepest level of the encoder As input, the spatial resolution of the feature map is first improved by upsampling operation, then local correlation information is extracted by convolution layer to refine feature expression, then batch normalization is used to accelerate training and improve stability, finally the nonlinearity is introduced by the leaky linear rectifier unit, and this operation is repeated several times to realize the reconstruction of the original input data, and the weights of the convolution layers in the two paths are shared, and the common feature representation of HSI and LiDAR is learned through the shared layer; the reconstructed data output by the decoder An adversarial learning loop is formed with the original image data, and a regularization constraint is provided for the classification task, forming a multi-task collaborative optimization framework.

5. The method of claim 4, wherein, The step 4 comprises: Data discrimination by a structure consistency perception discriminator, which adopts a double discriminator architecture, and designs independent and identical structure discriminators for HSI and LiDAR modalities; first, the two branches input the original image real data and the pseudo data generated by reconstruction; then, the data is extracted through a convolution layer to obtain local features, and the features output by the convolution are batch normalized to accelerate model training and improve stability, and then a nonlinearity is introduced through a leaky linear rectifier unit to enhance the expression ability for complex patterns; then, the self-attention mechanism is used to capture the long-distance dependence of the features to enhance the perception of structural differences; finally, the features are mapped to the discrimination space through a fully connected layer, and the probability of real data or pseudo data is output through an S-shaped function; the double discriminator architecture integrates two core technologies to improve the stability and discrimination ability of the adversarial training: h, spectral normalization: all convolution layers and fully connected layers in the discriminator adopt spectral normalization, which constrains the Lipschitz constant of the discriminator function; i. Self-attention mechanism: after the deep feature map of the discriminator, a self-attention module is embedded; the self-attention module is used to capture the long-distance dependence between any two positions in the feature map, so that the discriminator evaluates the authenticity of the local texture and judges the global structural consistency of the generated image, thereby providing gradient feedback for the generator, and the expression of the self-attention mechanism feature enhancement is: ； wherein the query , the key , and the value are feature projections, , , are weights of the query , the key , and the value respectively; is the dimension of the key K.

6. The method of claim 5, wherein, The step 6 comprises: To utilize the feature information of different scales, the multi-level feature fusion classifier constructs a hierarchical adaptive decision framework, which improves the classification robustness through multi-scale feature collaborative optimization ; each feature is input into a dedicated classification path, which first mines classification key information through a convolutional layer, then improves stability through batch normalization, introduces nonlinearity through a leaky linear rectifier unit to refine feature expression, and strengthens feature discriminability through a stacking combination module, and then maps each level of feature to the corresponding class probability through a normalization function, which is expressed as: ； wherein, is a softmax activation, is a ReLU nonlinearity activation, are learnable parameters, r takes values 1, 2; Finally, the decision information of the three category probability maps is fused by weighted summation to make up for the limitations of single-level features, and finally the global optimal classification result considering the fine-grained details and the global structural consistency is output: ； Wherein, the weights δ1, δ2, δ3 are all learnable parameters, which are automatically optimized in the training according to the classification ability of the three hierarchical features, and satisfy δ1+δ2+δ3 tends to 1. Final classification result R i is determined by the maximum value index in the joint classification probability P i , whose expression is: ； Wherein, N is the total number of ground object categories.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the joint classification method for hyperspectral image and lidar data according to any one of claims 1-6 when the program is running.

8. An electronic device, comprising: Comprise: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprise instructions, when the instructions are executed by the device, the device executes the joint classification method for hyperspectral image and lidar data according to any one of claims 1-6.

Images