Hyperspectral and ranging image fusion method and device based on enhanced network

Abstract

The invention belongs to the technical field of image fusion, and discloses a hyperspectral and ranging image fusion method and device based on an enhanced network. The method comprises the following steps: constructing a hierarchical space-spectrum enhancement network architecture, wherein the hierarchical space-spectrum enhancement network architecture comprises a multi-scale space enhancement module, a global spectrum enhancement module and a space spectrum enhancement module; constructing a joint loss function including constraint loss and classification loss; and training a hierarchical space-spectrum enhancement network architecture by using a joint loss function, balancing constraint loss and classification loss, and completing collaborative optimization of feature reconstruction and classification tasks. According to the invention, by means of the LiDAR technology, the three-dimensional structure of the ground feature can be clearly presented, and the analysis capability of the ground feature form and spatial distribution is enhanced, so that the classification precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to a method and device for fusing hyperspectral and ranging images based on an enhanced network. Background Art

[0002] Fusion technology for hyperspectral and LiDAR data has become a research hotspot in recent years. Traditional data fusion methods include feature-level fusion and decision-level fusion. Feature-level fusion aims to fuse features from different sources (i.e., hyperspectral and LiDAR data) in the feature space during the model learning process to enhance the model's expressiveness and reasoning capabilities. Decision-level fusion is also a common strategy for multimodal data fusion. It mainly fuses the final decision results after reasoning on each individual model or modality to improve the accuracy and robustness of the overall system. In addition, recent research has also begun to explore multimodal fusion methods based on deep learning, fusing the features of hyperspectral and LiDAR data through methods such as convolutional neural networks and Transformer.

[0003] Research on data fusion methods based on traditional machine learning methods.

[0004] Feature-level fusion strategies typically include early fusion, late fusion, and hybrid fusion. Early fusion directly combines features from different data sources into a multidimensional feature vector. By fusing the spectral features of hyperspectral imagery with the spatial features of lidar, a comprehensive feature vector is generated for use by the classifier. Late fusion performs independent classification on different data sources after feature extraction and then combines the classification results through weighting, voting, and other methods. Hybrid fusion combines the advantages of early and late fusion, performing independent classification after partial fusion and then final fusion through a decision-making mechanism. The choice of each strategy is closely related to the specific application task. For example, in urban classification tasks, the spatial information of lidar data may be more critical, while in ecological monitoring, the spectral information of hyperspectral imagery is more important. For example, Rasti et al. used orthogonal total variation component analysis to fuse two datasets, preserving spatial structure while reducing feature dimensionality. Extinction profiles are used to automatically extract spatial information, while total variation is used as an effective regularization technique to promote spatial correlation and smoothness. However, directly concatenating heterogeneous features can lead to excessively high feature space dimensions, increasing data sparsity and computational complexity. This can also trigger the Hughes phenomenon, causing classifier performance to degrade in these sparse regions, thus impacting collaborative classification performance. Therefore, the model proposed by Hong et al. employs a novel feature fusion strategy that forces the fused features to reconstruct the original multimodal input data, thereby enhancing cross-modal neuronal activation. Decision-level fusion strategies typically fall into two categories: hard fusion and soft fusion. Hard fusion is typically based on a majority voting mechanism, where each classifier votes on each feature class, and the class with the most votes determines the final classification result. Weighted voting, an extension of hard fusion, assigns different weights based on classifier performance or confidence, increasing the influence of higher-performing classifiers in the final decision. Soft fusion, on the other hand, fuses features by processing the confidence or probability outputs of the classifiers, taking a weighted average of the probability values ​​for each class to produce a smooth classification decision.

[0005] Research on data fusion methods based on convolutional neural networks.

[0006] Convolutional neural networks (CNNs), through their hierarchical structure, can learn features ranging from simple to complex. They can also effectively and automatically learn spatial-level features from image data, making them excellent at processing HSI and LiDAR fusion data with rich spatial structure. For example, Guo et al. fused the outputs of two consecutive layers within each large convolutional block as the input to the next layer. They utilized the fusion of multiple spatial-spectral features to improve the accuracy of hyperspectral image classification. By fusing features from different levels and incorporating interactive information, Zhao et al. combined a deep convolutional neural network with a hierarchical random walk algorithm, effectively leveraging the complementary strengths of the two data sources. Hang et al. used a fusion of two coupled convolutional neural networks to learn the spectral-spatial features of hyperspectral data and capture the elevation information of LiDAR data, respectively. Although CNNs have performed well in remote sensing image classification and recognition, achieving particularly remarkable results in processing HSI and LiDAR fusion data, they primarily extract features through a local receptive field and tend to overlook long-range dependencies, limiting their ability to capture complex spatial-spectral correlations in spectral data and insufficiently processing the global sequential nature of spectral features.

[0007] Research on data fusion methods based on Transformer.

[0008] The Transformer has achieved remarkable success in the field of computer vision due to its unique global modeling capabilities, self-attention mechanism, and versatility in processing sequential data. The proposed Vision Transformer (ViT), a first for successfully applying the Transformer model to image processing tasks, provides an efficient and powerful approach for handling complex visual tasks. Furthermore, when processing HSI images, ViT effectively captures global spectral features, flexibly handles different bands and their order, and models global contextual information through a self-attention mechanism. Building on ViT, Hong et al. proposed the SpectralFormer, a model more suitable for hyperspectral image classification tasks that captures more subtle spectral differences. The DBFFT model proposed by Dang et al. uses a dual-branch structure to effectively extract and fuse spectral and spatial features of hyperspectral images, providing new insights and methods for hyperspectral image classification. Furthermore, leveraging the powerful capabilities of Transformer networks based on self-attention mechanisms can be used to handle heterogeneous feature representation and information fusion in multimodal remote sensing data.

[0009] In summary, how to use convolutional networks and Transformer to fully mine the spectral information in hyperspectral images and the spatial information contained in LiDAR images will be the key problem to be overcome in this invention.

[0010] Through the above analysis, the problems and defects of the existing technology are: the existing technology does not fully utilize the spectral and spatial information in the fusion process of hyperspectral image (HSI) and light detection and ranging image (LiDAR), and the image classification accuracy is low. Summary of the Invention

[0011] In order to overcome the problems existing in the related art, the embodiments disclosed in the present invention provide a method and device for the fusion of hyperspectral and ranging images based on an enhancement network, specifically a method and device for the fusion of hyperspectral and LiDAR images based on a layered spatial-spectral enhancement network.

[0012] The technical solution is as follows: A hyperspectral and ranging image fusion method based on an enhanced network includes the following steps:

[0013] S1, constructing a hierarchical spatial-spectral enhancement network architecture, including a multi-scale spatial enhancement module, a global spectral enhancement module, and a spatial-spectral enhancement module;

[0014] S2, construct a joint loss function including constraint loss and classification loss;

[0015] S3 uses a joint loss function to train a hierarchical spatial-spectral enhancement network architecture, balancing constraint loss and classification loss to achieve collaborative optimization of feature reconstruction and classification tasks.

[0016] In step S1, the multi-scale spatial enhancement module is used to extract multi-scale features from hyperspectral and lidar data, including:

[0017] (1) Establish the spatial correlation between HSI and LiDAR data; transform the input features by SE Extract and capture the features that are important for calculating the spatial correlation matrix; this conversion operation is composed of an independent convolutional layer, which includes convolution, batch normalization and Sigmoid activation function to generate low-dimensional and normalized spatial attention weights; The shape of Reshape into in, is the hyperspectral feature set, is the LiDAR image feature set, λ∈{1,2,3} represents spatial feature extraction at three scales, H is the height of the image, W is the width of the image, and C is the number of image bands;

[0018] The calculation form of SE is as follows:

[0019] SE(X λ)=Sigmoid(BN(conv(X λ )))

[0020] In the formula, SE() is the conversion operation, Sigmoid() is the activation function, BN() is the normalization operation, conv() is the convolution operation, X λ are input images of different regimes, where λ∈{1,2,3};

[0021] (2) Yes Perform matrix multiplication on the eigenvector at each position in , where To perform the conversion operation on the hyperspectral features with a scale of λ, To perform conversion operations on LiDAR features with a scale of λ;

[0022] (3) Aggregate features through the Squeeze operation; the Squeeze operation rearranges the dimensions of the matrix obtained by matrix multiplication, and uses a one-dimensional convolutional layer to perform channel compression on the rearranged matrix, integrating the information of local spatial relationships into the channel descriptor, and obtaining two affinity matrices for spatial enhancement. Used to effectively weight and adjust the original input features; the formula is as follows:

[0023]

[0024] Where, is the affinity matrix calculated by different permutations and combinations of input features;

[0025] (4) Weighted to the original features On the other hand, we selectively enhance or suppress different parts of the original features, and the resulting fusion features are The enhancement process is expressed as:

[0026]

[0027] Where α is the adaptive weighting coefficient, and its initial value is set to 0.5; is the spatial fusion feature with scale λ.

[0028] After step (4), you need to do the following:

[0029] First, features from different scales are mapped to the same spatial size H×W×C through a fully connected layer, unifying the size of the feature maps at each scale to the spatial size when λ=1;

[0030] Secondly, the flattening operation is used to flatten the features in the spatial dimension, and the shape of the flattened feature sequence is HW×C;

[0031] Finally, Concat is used to connect the fusion features of the three scales along the channel dimension, and the final multi-scale spatial enhancement feature F spa Calculated using the following expression:

[0032]

[0033] The splicing of multi-scale spatial enhancement features enables the information between different layers to complement and enhance each other and generate feature representations.

[0034] In step S1, the global spectrum enhancement module is improved on the basis of ViT, SpeT is introduced to achieve spectrum enhancement together with ViT, and a dual-branch Transformer structure is designed to achieve global spectrum enhancement; the dual-branch Transformer structure consists of MHSA components and MLP components;

[0035] The calculation process of the MHSA component includes linear transformation, scaled dot product attention calculation and splicing of multiple attention heads; including: the input feature matrix is ​​B l ∈R H×W×D ,l∈{1,2…L}, after linear transformation Represent the query matrix, key matrix and value matrix respectively; among them, is a learnable transformation matrix; in ViT, MHSA divides the input feature vector into different attention heads according to the spectral dimension;

[0036] Q l ,K l ,V l Map to h subspaces and divide them into h groups. The feature dimension in each group is D / h, and independent attention calculation is performed in each group. In SpeT, SMHSA is designed. SMHSA divides the input feature vector into different attention heads according to the spatial dimension, so that each attention head can focus on different spatial regions and capture the spectral changes and relationships in these regions. In grouping in the spatial domain, let M = HW be the size of the feature in the spatial dimension, and the feature dimension of each group is d k =M / h; H is the height of the image, W is the width of the image.

[0037] The calculation process of SMHSA is:

[0038]

[0039] Where, d k is the feature dimension size of each attention head, which is used to scale the dot product result. Softmax() is the Softmax function, which converts the dot product result into a probability distribution, representing the attention weight of each key vector on the query vector.

[0040] The obtained fusion feature F spa Flatten the feature and map it to the specified dimension D through a fully connected layer. Add position encoding to the input embedding to distinguish features at different positions. The feature input to the ViT encoder is B0, so the spectral enhancement process of the ViT encoder is:

[0041] B l =B l-1 +MHSA(LN(B l-1 )),l∈{1,2…L}

[0042] F vit =B L +MLP(LN(B L ))

[0043] Where L is the number of layers of the ViT encoder, LN and MHSA are layer normalization and multi-head attention, respectively; B l-1 is the output feature of the l-th layer encoder, MHSA() is the multi-head attention mechanism, LN() is the normalization layer, F vit is the output feature of vit, B L is the final output of the encoder, MLP( ) is a multi-layer perceptron;

[0044] The MLP component maps linear features to a higher-dimensional nonlinear space, and this process learns the global spectral information to obtain F vit ; SpeT uses SMHSA to collaborate with ViT to learn spectral features more comprehensively; The spectral enhancement process of SpeT encoder is:

[0045] B l =B l-1 +SpeMSA(LN(B l-1) ),l∈{1,2…L}

[0046] F spe =B L +MLP(LN(B L ))

[0047] Where SpeMSA() is spectral multi-head attention;

[0048] The output feature obtained is F spe , F spe With F vit They are obtained by learning and extracting different parts of the input features respectively; combining these two features to achieve more comprehensive spectral enhancement; splicing the tokens obtained from the ViT decoder and the SpeT decoder to obtain F con Used to assist reconstruction tasks.

[0049] In step S1, the spatial spectrum enhancement module combines the F spa ,F vit ,F spe Three features fusion.

[0050] Furthermore, the F obtained by the multi-scale spatial enhancement module and the global spectral enhancement module spa ,F vit ,F spe The three feature fusions include:

[0051] First, a classifier is used to classify the spatial enhancement features F spa The most representative features are extracted through convolution and maximum pooling layers, and these features are mapped to the predefined number of ground object categories C using the fully connected layer. The features are converted into a C-dimensional probability distribution P using the Softmax function. spa ;

[0052] Secondly, F vit and F spe It is the feature corresponding to the CLS token after Transformer processing. Using layer normalization and fully connected layers, the spectral features are mapped to C categories, and the C-dimensional probability distribution P is generated through the Softmax function. vit and P spe ;

[0053] Finally, P is transformed into spa ,P vit ,P spe Fusion: The probability distributions corresponding to the three features are multiplied by the learnable weight coefficients β1, β2, and β3 respectively, and then added together to obtain the final probability distribution P. The initial values ​​of these weight coefficients are all 1 / 3. The fusion formula that represents the fusion result of the probability distribution obtained after different features pass through the softmax classifier is:

[0054] P=β1P spa +β2P vit +β3P spe

[0055] Where P is the probability distribution of the final fusion feature, P spa is the probability distribution of spatial features, P vit Obtain the probability distribution of features for vit, P spe is the probability distribution of spectral features.

[0056] In step S2, a joint loss function including constraint loss and classification loss is constructed, including:

[0057] In the decoder stage, the tokens output from ViT and SpeT are concatenated to form a new token F con ; F con The spatial resolution of the original data is restored by upsampling at different scales; the features of different scales are processed using 3×3 convolution and sigmoid function. and The constraint loss uses mean square error as the loss function, and the formula is as follows:

[0058]

[0059] Where, L con To balance the constraint loss, Represent the reconstructed HSI and LiDAR data respectively, N is the number of samples; these features correspond to the input HSI cube and LiDAR cube L con , the constraint loss is calculated by minimizing the difference between these reconstructed features and the original input data; n is a variable, ranging from 1 to N, are the original HIS data of different scales, The original LiDAR data of different scales;

[0060] The classification loss uses the cross entropy loss function to calculate the difference between the category probability distribution predicted by the model and the actual label. The formula is as follows:

[0061]

[0062] Where, L cls is the classification loss, N is the number of samples in each batch, y n is the true label of the nth sample, p n is the predicted class probability corresponding to the nth sample;

[0063] In order to optimize the reconstruction and classification tasks simultaneously, the constraint loss and classification loss are combined to form a joint loss function; the formula is as follows:

[0064] L=βL con +L cls

[0065] Where β is the weight coefficient used to balance the contribution of reconstruction loss and classification loss; β is set to 1 / 6.

[0066] In step S3, the joint loss function L considers both feature reconstruction and classification tasks, and the loss L is constrained by balancing con and classification loss L cls , achieving collaborative optimization in feature learning and classification performance.

[0067] Another object of the present invention is to provide a hyperspectral and ranging image fusion device based on an enhancement network, which implements the hyperspectral and ranging image fusion method based on an enhancement network, and the device includes:

[0068] The multi-scale spatial enhancement module is used to extract multi-scale features from hyperspectral and lidar data, and the final result is the multi-scale spatial enhancement feature F spa , so that the information between different layers can complement and enhance each other and generate feature representation;

[0069] The global spectral enhancement module is used to achieve global spectral enhancement using a dual-branch Transformer structure, including a Vision Transformer branch and a Spectral Transformer branch; the Vision Transformer branch is used to enhance the spectrum to obtain F vit , the Spectral Transformer branch collaborates with the Vision Transformer branch to learn spectral features and obtain F spe ;

[0070] The spatial spectrum enhancement module is used to transform the F obtained by the multi-scale spatial enhancement module and the global spectrum enhancement module into spa ,F vit ,F spe The three features are fused to obtain the final probability distribution P for classification;

[0071] The hierarchical spatial-spectral enhancement network training module is used to construct a joint loss function composed of the reconstruction loss L con And the classification loss constitutes L cls ; Balance the constraint loss and classification loss to complete the collaborative optimization of feature reconstruction and classification tasks.

[0072] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows:

[0073] This paper aims to address the issue of insufficient utilization of spectral and spatial information during the fusion process of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) imagery. By designing a network device that can perform spatial and spectral enhancement layered at different stages and ultimately achieve fusion, the effective fusion of hyperspectral and LiDAR data is currently a key research hotspot in remote sensing. Hyperspectral imagery, with its broad spectral coverage, can capture fine-grained spectral characteristics of ground objects, greatly improving the ability to accurately identify surface objects. It is widely used in fields such as environmental monitoring, land use change, and agricultural disaster assessment. However, despite the rich spectral information provided by hyperspectral imagery, its processing and classification still face challenges. In particular, the single spectral information often lacks sufficient discriminative power when distinguishing objects with similar spectral responses but different spatial distributions. However, LiDAR technology, which uses laser beams to perform high-precision three-dimensional measurements of the ground, can provide spatial information based on elevation and morphology, providing a powerful complement to the classification and interpretation of hyperspectral remote sensing imagery. Especially in complex terrain and diverse landform environments, LiDAR's point cloud data can accurately distinguish different landforms, such as buildings, roads, trees, etc., which effectively compensates for the shortcomings of traditional hyperspectral images in spatial recognition. With the help of LiDAR technology, the three-dimensional structure of the landform can be clearly presented, which enhances the analysis ability of the landform morphology and spatial distribution, thereby improving the classification accuracy. However, due to the insufficient utilization of complementary information in multi-source data sets, the extraction and integration of spatial and spectral information after fusion are still insufficient. Therefore, the present invention divides feature fusion into three stages. The first stage is the multi-scale spatial enhancement stage, which uses a convolutional neural network (CNN) to extract features of different scales and make full use of the spatial complementary information of HSI and LiDAR data. The second stage is the global spectral enhancement stage, which uses a dual-branch Transformer to better mine the information contained in the spectral dimension. The third stage is to combine the features after spatial enhancement with the features after spectral enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure;

[0075] Figure 1 This is a schematic diagram of a method for fusion of hyperspectral and LiDAR images based on a layered spatial-spectral enhancement network provided by an embodiment of the present invention;

[0076] Figure 2This is a flow chart of a method for fusion of hyperspectral and ranging images based on an enhanced network provided by an embodiment of the present invention;

[0077] Figure 3 is a specific structural diagram of a multi-scale spatial enhancement module provided by an embodiment of the present invention;

[0078] Figure 4 This is a diagram of the dual-branch Transformer structure provided by an embodiment of the present invention;

[0079] Figure 5 These are classification maps of Houston2013 data obtained using different methods, where: (a) is a pseudo-color image of a hyperspectral image, (b) is a digital surface model based on LiDAR, (c) is a ground truth annotation map, (d) is a TBCNN map, (e) is an EndNet map, (f) is an MDL-middle map, (g) is a HRWN map, (h) is a FusAtNet map, (i) is a MAHiDFNet map, and (j) is a rendering of the present invention. DETAILED DESCRIPTION

[0080] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0081] The innovation of this invention lies in its effective combination of multi-scale feature extraction and multi-scale spatial enhancement. The former uses convolution kernels of different sizes to extract features from hyperspectral imagery and LiDAR data in parallel. The latter further processes these features and dynamically adjusts feature weights through spatial compression-expansion technology, thereby highlighting important information, suppressing irrelevant information, and optimizing data representation efficiency.

[0082] In addition, the present invention also introduces a dual-branch Transformer structure and innovatively improves the multi-head self-attention mechanism to identify and strengthen global dependencies in the data, effectively capturing the intrinsic connections in hyperspectral and LiDAR fusion data.

[0083] Example 1: The hyperspectral and ranging image fusion device based on the enhancement network provided by the embodiment of the present invention includes a multi-scale spatial enhancement module, a global spectral enhancement module, a spatial spectral enhancement module and a hierarchical spatial-spectral enhancement network training module.

[0084] For example, the multi-scale spatial enhancement module extracts multi-scale features from hyperspectral and lidar data, and finally obtains the multi-scale spatial enhancement feature F spa , which enables the information between different layers to complement and enhance each other, generating a richer feature representation.

[0085] For example, the global spectrum enhancement module uses a dual-branch Transformer structure to achieve global spectrum enhancement, one branch is composed of Vision Transformer, and the other branch is composed of Spectral Transformer. The Vision Transformer branch is used to enhance the spectrum to obtain f vit , the Spectral Transformer branch can cooperate with the Vision Transformer branch to learn spectral features more comprehensively and obtain F spa .

[0086] For example, the spatial spectrum enhancement module is the F obtained by the first two multi-scale spatial enhancement modules and the global spectrum enhancement module. spa ,F vit ,F spe The three features are fused to obtain the final probability distribution P for classification.

[0087] The hierarchical spatial-spectral enhancement network training module is used to construct a joint loss function composed of the reconstruction loss L con And the classification loss constitutes L cls Balance the constraint loss and classification loss to complete the collaborative optimization of feature reconstruction and classification tasks.

[0088] Example 2, as Figure 1 As shown, a schematic diagram of a hyperspectral and LiDAR image fusion method based on a layered spatial-spectral enhancement network provided by an embodiment of the present invention is shown in FIG. For example, the overall architecture diagram of the layered spatial-spectral enhancement network is shown;

[0089] Specifically, such as Figure 2 As shown, the hyperspectral and ranging image fusion method based on the enhancement network provided by the embodiment of the present invention includes the construction and training of a hierarchical spatial-spectral enhancement network. Figure 1 As shown in the figure, the hierarchical spatial-spectral enhancement network can fully mine the spectral and spatial information contained in hyperspectral images and LiDAR images, which is beneficial to improving the accuracy of the classifier.

[0090] Specifically including: S1, the construction of hierarchical spatial-spectral enhancement network architecture, the hierarchical spatial-spectral enhancement network architecture includes multi-scale spatial enhancement module, global spectral enhancement module, spatial spectral enhancement module, the overall framework is as follows Figure 1 shown.

[0091] For example, the specific structure of the multi-scale spatial enhancement module is as follows: Figure 3 As shown in the figure, the main function of this module is to perform multi-scale feature extraction on hyperspectral and lidar data. The input of this module is cubic data blocks of different scales (3 types), each of which is centered on the same pixel in the HSI and LiDAR datasets, covering multiple spatial scales. These data blocks correspond to different spatial resolutions and are centered on the same pixel position in the image to ensure consistent feature alignment across scales. The multi-scale feature extraction module consists of 2 convolutional layers, each of which contains a convolution operation, as well as a batch normalization and ReLU activation function to enhance the representational ability and nonlinear characteristics of the features. The convolution kernel size of the two convolutional layers is 3x3, the number of kernels is 32 and 64 respectively, the stride is 1, and the padding is 1 to keep the feature map size unchanged. Adding a maximum pooling layer to the second convolution layer can reduce the spatial size by two times. These convolutional layers are used at different scale levels, and the HSI and LiDAR data are organized into a three-dimensional data structure with a certain spatial resolution and spectral dimension. The obtained HSI features are The obtained LiDAR features are Among them, λ∈{1,2,3} indicates that spatial features are extracted at three scales.

[0092] Exemplarily, multi-scale feature extraction of hyperspectral and lidar data includes:

[0093] (1) The key of this module is to establish the spatial correlation between HSI and LiDAR data. Perform feature extraction to capture features that are important for calculating the spatial correlation matrix. This conversion operation consists of an independent convolutional layer, which includes convolution, batch normalization, and Sigmoid activation function to generate low-dimensional and normalized spatial attention weights. This operation will halve the number of feature channels to reduce computational complexity. For subsequent matrix multiplication, The shape of Reshape into in, is the hyperspectral feature set, is the LiDAR image feature set, λ∈{1,2,3} represents spatial feature extraction at three scales, H is the height of the image, W is the width of the image, and C is the number of image bands. The innovative proposal of this invention is that the calculation form of SE is as follows:

[0094] SE(X λ )=Sigmoid(BN(conv(X λ )))1)

[0095] In the formula, SE represents the conversion operation, which is composed of three operations (convolution conv, batch normalization BN, and activation function sigmoid), SE() is the conversion operation, Sigmoid() is the activation function, BN() is the normalization operation, conv() is the convolution operation, X λ are input images of different regimes, where λ∈{1,2,3};

[0096] (2) Yes Perform matrix multiplication on the eigenvector at each position in , where To perform the conversion operation on the hyperspectral features with a scale of λ, To transform the LiDAR features of scale λ; essentially, it is measuring The characteristics of each position in This is very close to the core idea of ​​the attention mechanism, but compared to the attention mechanism that captures long-range dependencies in the input sequence, matrix multiplication focuses more on local spatial feature enhancement.

[0097] (3) Next, the features are aggregated through the Squeeze operation; the Squeeze operation rearranges the dimensions of the matrix obtained by matrix multiplication, and a one-dimensional convolutional layer is used to perform channel compression on the rearranged matrix. This not only reduces the computational complexity, but also retains the aggregated information of the local spatial relationship. Through these compressed features, the present invention can integrate the information of the local spatial relationship into the channel descriptor, and obtain two affinity matrices for spatial enhancement. It is used to effectively weight and adjust the original input features. The formula is as follows:

[0098]

[0099] Where, The affinity matrices are calculated by permuting the input features. They share a transposition relationship, or symmetry. This symmetry emphasizes the interactions between different dimensions of the feature matrix. Using them to enhance the original features can capture more complementary spatial information.

[0100] (4) Weighted to the original features On the other hand, it selectively enhances or suppresses different parts of the original features, helping the model to focus on key features more effectively. The innovative proposal of this invention is that the fusion feature obtained is The enhancement process is expressed as:

[0101]

[0102] Where α is an adaptive weighting coefficient. In order to give equal importance to the two heterogeneous features, the initial value is set to 0.5; is the spatial fusion feature with a scale of λ. α is defined as a trainable parameter, which is automatically updated and adjusted through back propagation and the work of the optimizer. As the training process progresses, in each round of training, the value of α will be dynamically adjusted according to the gradient information, dynamically changing the fusion ratio of two features of the same scale. The present invention simultaneously introduces affinity matrix and adaptive weighting to play their respective roles. The affinity matrix captures spatial correlation by calculating the similarity of local features to achieve preliminary feature weighting; while adaptive weighting provides flexibility on this basis, enabling the model to adaptively adjust the ratio of feature weighting according to data during training.

[0103] For example, in order to achieve effective fusion of multi-scale features, it is first necessary to map features from different scales to the same spatial size H×W×X through a fully connected layer, that is, to unify the size of the feature map of each scale to the spatial size when λ=1. The purpose of this is to ensure that features of different scales have consistent sizes in the spatial dimension, thereby providing a unified basis for subsequent fusion operations. Next, a flattening operation is used to flatten these features in the spatial dimension, and the shape of the flattened feature sequence is HW×C. This flattening process merges the spatial dimensions into a one-dimensional sequence, providing an appropriate input format for the Transformer module, so that it can capture the dependencies between different spatial positions. Finally, Concat is used to connect the fused features of the three scales along the channel dimension, which ensures that the information between different scales can complement and enhance each other, thereby obtaining a richer feature representation. The final multi-scale spatial enhanced feature F spa It can be calculated using the following expression:

[0104]

[0105] The concatenation of multi-scale spatial enhancement features enables the information between different layers to complement and enhance each other, generating a richer feature representation.

[0106] For example, the global spectral enhancement module is very important in the joint classification of hyperspectral and LiDAR. Vision Transformer is a powerful model that uses the self-attention mechanism to capture the global dependencies of the input sequence. Figure 4Middle (a). Based on the self-attention mechanism, it is possible to capture the global relationship between spectral bands. This global perspective can better understand the complex dependencies between different bands. In addition, in order to capture spectral features more comprehensively, the present invention also improves on ViT and introduces SpectralTransformer (SpeT) ( Figure 4 (b)). A dual-branch Transformer structure is designed to achieve global spectral enhancement. Figure 4 As shown in the dual-branch Transformer structure diagram;

[0107] The core architecture of Transformer mainly consists of two components: Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron, which appear alternately in each layer of Transformer. The dual-branch Transformer in this module adopts the classic encoder-decoder structure. Figure 4 As shown, one branch consists of ViT and the other branch consists of SpeT. The first ViT and SpeT are encoders, which can effectively extract and enhance the spectral information in the input data; the second ViT and SpeT are decoders, which rely on the powerful modeling ability of Transformer to restore and reconstruct details, and adapt to image information of different scales through gradual generation. The number of layers L of ViT and SpeT encoders are both set to 5, and the embedding dimension D is set to 64. The decoder has the same number of layers as the encoder, but the embedding dimension is set to 32. Since the feature map input by the present invention is a patch of three scales divided around the same central pixel point for spatial feature extraction, the flattening operation of the input feature in the spatial dimension has been completed in the multi-scale spatial enhancement module, so the module outputs the feature F spa It is organized into a two-dimensional feature sequence of shape HW × C, where HW represents the total number of spatial positions and C represents the number of features in the channel dimension.

[0108] The calculation process of MHSA involves several key steps, including linear transformation, scaled dot product attention calculation and splicing of multiple attention heads.

[0109] B l ∈R H×W×D ,l∈{1,2…L}, after linear transformation Represent the query matrix, key matrix and value matrix respectively; among them, is a learnable transformation matrix; in ViT, MHSA divides the input feature vector into different attention heads according to the spectral dimension; the present invention divides Q l ,K l ,V l Mapped to h subspaces, they are divided into h groups, the feature dimension in each group is D / h, and independent attention calculation is performed in each group; in SpeT, Spectral Multi-Head Self-Attention (SMHSA) is designed. SMHSA divides the input feature vector into different attention heads according to the spatial dimension, so that each attention head can focus on different spatial regions, capture the spectral changes and relationships in these regions, and provide more fine-grained spectral features; in grouping in the spatial domain, let M = HW be the size of the feature in the spatial dimension, and the feature dimension of each group is d k =M / h; H is the height of the image, W is the width of the image.

[0110] The calculation process of SMHSA is:

[0111]

[0112] Where, d k is the feature dimension size of each attention head, which is used to scale the dot product result. Softmax() is the Softmax function, which converts the dot product result into a probability distribution, representing the attention weight of each key vector on the query vector.

[0113] Because Transformer processes one-dimensional sequence data, before spectral feature enhancement, the present invention needs to transform the fusion feature F obtained in the previous stage into spa Flatten, then map the flattened features to the specified dimension D through a fully connected layer. Adding position encoding to the input embedding enables the model to distinguish features at different positions. The innovative proposal of this invention is that if the feature of the input ViT encoder is B0, then the spectral enhancement process of the ViT encoder is:

[0114] B l =B l-1 +MHSA(LN(B l-1 )),l∈{1,2…L}

[0115] F vit =B L +MLP(LN(B L ))

[0116] Where L is the number of layers of the ViT encoder, LN and MHSA are layer normalization and multi-head attention, respectively; B l-1is the output feature of the l-th layer encoder, MHSA() is the multi-head attention mechanism, LN() is the normalization layer, F vit is the output feature of vit, B L is the final output result of the encoder, MLP() is a multi-layer perceptron;

[0117] MLP is a multilayer perceptron that maps linear features to a higher-dimensional nonlinear space, thereby improving the model's ability to express complex patterns. This process learns global spectral information to obtain F vit Unlike the ViT structure, SpeT uses SMHSA to collaborate with ViT to learn spectral features more comprehensively. The present invention innovatively proposes that the spectral enhancement process of the SpeT encoder is:

[0118] B l =B l-1 +SpeMSA(LN(B l-1 )),l∈{1,2…L}

[0119] F spe =B L +MLP(LN(B L ))8)

[0120] Where SpeMSA() is spectral multi-head attention;

[0121] The output feature obtained is F spa , F spe With F vit They are obtained by learning and extracting different parts of the input features respectively; combining these two features to achieve more comprehensive spectral enhancement; splicing the tokens obtained from the ViT decoder and the SpeT decoder to obtain F con Used to assist reconstruction tasks.

[0122] In summary, the global spectral features provided by ViT ensure the integrity and consistency of global spectral information, while the fine-grained spectral features provided by SpeT enhance the feature expression of local regions. This dual-branch Transformer combination strategy not only improves the robustness of the model, but also enhances its generalization ability in complex spectral data processing tasks.

[0123] Exemplary, spatial spectrum enhancement module. The purpose of the spatial spectrum enhancement module is to transform the F spa , F spe With F vit Three feature fusion. First, the present invention needs to use a classifier to classify the spatial enhancement feature F spaThe processing is specifically implemented as follows: the most representative features are extracted through convolution operations and maximum pooling layers, these features are mapped to the predefined number of ground object categories C using the fully connected layer, and the features are converted into a C-dimensional probability distribution P through the Softmax function. spa Secondly, F vit and F spe In fact, it is the features corresponding to the CLS token after Transformer processing. Using layer normalization and fully connected layers, the spectral features are mapped to C categories, and the C-dimensional probability distribution P is generated through the Softmax function. vit and P spe Finally, F is adaptively weighted by linear weighting. spa , P vit With P spe Fusion is specifically implemented by multiplying the probability distributions corresponding to the three features by the learnable weight coefficients β1, β2, and β3 and then adding them together to obtain the final probability distribution P. The initial value of these weight coefficients is 1 / 3 to ensure that each feature has equal initial importance when fused, but they can be optimized and adjusted during the training process. The present invention innovatively proposes that the fusion formula representing the fusion result of the probability distribution obtained after different features pass through the softmax classifier is:

[0124] P=β1P spa +β2P vit +β3P spe (9)

[0125] Where P is the probability distribution of the final fusion feature, P spa is the probability distribution of spatial features, P vit Obtain the probability distribution of features for vit, P spe is the probability distribution of spectral features.

[0126] Through this feature fusion method, the present invention can fully utilize the information of spatial and spectral features, thereby improving the accuracy and robustness of classification.

[0127] S2, construct a joint loss function including constraint loss and classification loss;

[0128] Exemplarily, a joint loss function is used to improve the generalization ability and classification accuracy of the model.

[0129] The constraint loss is also called reconstruction loss, which is to ensure that the data after spatial and spectral enhancement can restore the original input data as much as possible. In the decoder stage, the tokens output from ViT and SpeT are concatenated (and linked) to form a new token F con . F conThe spatial resolution of the original data is restored by upsampling at different scales. Subsequently, 3×3 convolution and sigmoid function are used to process the features at different scales to obtain and The constraint loss calculates the difference between the reconstructed image output by the network and the original input data, using the mean square error as the loss function, and the formula is as follows:

[0130]

[0131] Where, L con To balance the constraint loss, Represent the reconstructed HSI and LiDAR data respectively, N is the number of samples; these features correspond to the input HSI cube and LiDAR cube L con , the constraint loss is calculated by minimizing the difference between these reconstructed features and the original input data; n is a variable, ranging from 1 to N, are the original HIS data of different scales, is the original LiDAR data at different scales; the constraint loss is calculated by minimizing the difference between these reconstructed features and the original input data.

[0132] Classification loss is used to optimize the model's discriminative ability, enabling it to accurately classify different categories. This part of the loss uses the cross-entropy loss function to calculate the difference between the class probability distribution predicted by the model and the actual label. The formula is as follows:

[0133]

[0134] Where, L cls is the classification loss, N is the number of samples in each batch, y n is the true label of the nth sample, p n is the predicted class probability corresponding to the nth sample;

[0135] This loss function can effectively guide model learning so that the predicted probability p n As close as possible to the true label, thereby improving the classification accuracy of the model.

[0136] In order to optimize the reconstruction and classification tasks simultaneously, the present invention combines the constraint loss and the classification loss to form a joint loss function. The formula is as follows:

[0137] L=βL con +L cls (12)

[0138] Where β is a weight coefficient used to balance the contribution of reconstruction loss and classification loss; β is set to 1 / 6 to balance the six reconstruction targets (reconstructing HSI and LiDAR at three scales)

[0139] S3 uses a joint loss function to train a hierarchical spatial-spectral enhancement network architecture, balancing constraint loss and classification loss to achieve collaborative optimization of feature reconstruction and classification tasks.

[0140] For example, the joint loss function L is designed to consider both feature reconstruction and classification tasks, by balancing the constraint loss L con and classification loss L cls , achieving collaborative optimization of the model’s feature learning and classification performance. This ensures that the model captures important data features while improving its classification accuracy, leading to better performance in hyperspectral and LiDAR data fusion classification tasks.

[0141] To further illustrate the effects of the embodiments of the present invention, the following experiment was conducted: Compared with existing methods, the present invention can more comprehensively extract spectral information and spatial information, thereby improving classification accuracy.

[0142] Experimental parameters and environment settings.

[0143] In order to comprehensively measure the performance of HSSE-Net in the joint classification task of hyperspectral and lidar datasets, the present invention also selected OA, AA and Kappa as evaluation indicators, which provide a quantitative evaluation of the model performance. In order to more comprehensively understand the performance of the model, the present invention uses data visualization technology to intuitively present the classification results, further demonstrating the qualitative performance of the model, such as Figure 5 Classification plots of the Houston 2013 data obtained using different methods. Figure 5 Middle (a), pseudo color image of hyperspectral image, Figure 5 (b) The digital surface model based on LiDAR. Figure 5 Middle (c) Figure is the ground truth annotation map, Figure 5 Middle (d) TBCNN (89.56%), Figure 5 Middle (e) EndNet (88.71%), Figure 5 Middle (f) MDL-middle (92.24%), Figure 5 Middle (g) HRWN (90.36%), Figure 5 Middle (h) FusAtNet (88.43%), Figure 5 Middle (i) MAHiDFNet (90.55%), Figure 5Figure (j) in the middle represents the present invention (94.96%). "unclassed" indicates unclassified, "healthy grass" indicates healthy grass, "stressed grass" indicates stressed grass, "synthetic grass" indicates synthetic grass, "trees" indicates trees, "soil" indicates soil, "water" indicates water, "residential" indicates a residential area, "commercial" indicates a commercial district, "road" indicates a road, "highway" indicates a highway, "railway" indicates a railway, "parking lot 1" indicates parking lot 1, "parking lot 2" indicates parking lot 2, "tennis court" indicates a table tennis court, and "running track" indicates a running track.

[0144] This paper uses the deep learning framework PyTorch to implement the HSSE-Net method and conducts detailed experimental validation on the hyperspectral image classification task. Specifically, the Adam optimizer is used to drive model training, with an initial learning rate set to 0.0005. A step-wise learning rate decay mechanism is employed to optimize the learning rate during model training. This mechanism multiplies the learning rate by a decay factor of 0.9 every 20 training rounds, gradually reducing the learning rate. This configuration ensures rapid exploration capabilities in the early stages of model training while also ensuring stability and convergence in later training rounds. To balance computational efficiency and model performance, the present invention selected a training batch size of 64. This choice adheres to the widely held consensus in the field that, when processing large datasets, a moderate batch size effectively promotes stable gradient updates, thereby accelerating model convergence while avoiding the risk of overfitting. Furthermore, after careful monitoring and analysis of the model training process, the present invention identified a training cycle of 200 rounds as the optimal practice, which not only fully stimulates the model's learning potential but also effectively avoids the loss of generalization ability caused by overtraining.

[0145] Table 1 Performance comparison of different methods on the Houston 2013 dataset (bold shows the best results)

[0146]

[0147] The above description is only a preferred specific implementation method of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A hyperspectral and ranging image fusion method based on an enhanced network, characterized in that: The method comprises the following steps: S1, constructing a hierarchical spatial-spectral enhancement network architecture, including a multi-scale spatial enhancement module, a global spectral enhancement module, and a spatial-spectral enhancement module; S2, construct a joint loss function including constraint loss and classification loss; S3 uses a joint loss function to train a hierarchical spatial-spectral enhancement network architecture, balancing constraint loss and classification loss to achieve collaborative optimization of feature reconstruction and classification tasks.

2. The hyperspectral and ranging image fusion method based on enhanced network according to claim 1 is characterized in that: In step S1, the multi-scale spatial enhancement module is used to extract multi-scale features from hyperspectral and lidar data, including: (1) Establish the spatial correlation between HSI and LiDAR data; transform the input features by SE Extract and capture the features that are important for calculating the spatial correlation matrix; this conversion operation is composed of an independent convolutional layer, which includes convolution, batch normalization and Sigmoid activation function to generate low-dimensional and normalized spatial attention weights; The shape of Reshape into in, is the hyperspectral feature set, is the LiDAR image feature set, λ∈{1,2,3} represents spatial feature extraction at three scales, H is the height of the image, W is the width of the image, and C is the number of image bands; The calculation form of SE is as follows: SE(X λ )=Sigmoid(BN(conv(X λ ))) In the formula, SE() is the conversion operation, Sigmoid() is the activation function, BN() is the normalization operation, conv() is the convolution operation, X λ are input images of different regimes, where λ∈{1,2,3}; (2) Yes Perform matrix multiplication on the eigenvector at each position in , where To perform the conversion operation on the hyperspectral features with a scale of λ, To perform conversion operations on LiDAR features with a scale of λ; (3) Aggregate features through the Squeeze operation; the Squeeze operation rearranges the dimensions of the matrix obtained by matrix multiplication, and uses a one-dimensional convolutional layer to perform channel compression on the rearranged matrix, integrating the information of local spatial relationships into the channel descriptor, and obtaining two affinity matrices for spatial enhancement. Used to effectively weight and adjust the original input features; the formula is as follows: Where, is the affinity matrix calculated by different permutations and combinations of input features; (4) Weighted to the original features On the other hand, we selectively enhance or suppress different parts of the original features, and the resulting fusion features are The enhancement process is expressed as: Where α is the adaptive weighting coefficient, and its initial value is set to 0.5; is the spatial fusion feature with scale λ.

3. The hyperspectral and ranging image fusion method based on enhanced network according to claim 2 is characterized in that: After step (4), you need to do the following: First, features from different scales are mapped to the same spatial size H×W×C through a fully connected layer, unifying the size of the feature maps at each scale to the spatial size when λ=1; Secondly, the flattening operation is used to flatten the features in the spatial dimension, and the shape of the flattened feature sequence is HW×C; Finally, Concat is used to connect the fusion features of the three scales along the channel dimension, and the final multi-scale spatial enhancement feature F spa Calculated using the following expression: The splicing of multi-scale spatial enhancement features enables the information between different layers to complement and enhance each other and generate feature representations.

4. The hyperspectral and ranging image fusion method based on enhanced network according to claim 1 is characterized in that: In step S1, the global spectrum enhancement module is improved on the basis of ViT, SpeT is introduced to achieve spectrum enhancement together with ViT, and a dual-branch Transformer structure is designed to achieve global spectrum enhancement; the dual-branch Transformer structure consists of MHSA components and MLP components; The calculation process of the MHSA component includes linear transformation, scaled dot product attention calculation and splicing of multiple attention heads; including: the input feature matrix is After linear transformation Represent the query matrix, key matrix and value matrix respectively; among them, is a learnable transformation matrix; in ViT, MHSA divides the input feature vector into different attention heads according to the spectral dimension; Q l ,K l ,V l Map to h subspaces and divide them into h groups. The feature dimension in each group is D / h, and independent attention calculation is performed in each group. In SpeT, SMHSA is designed. SMHSA divides the input feature vector into different attention heads according to the spatial dimension, so that each attention head can focus on different spatial regions and capture the spectral changes and relationships in these regions. In grouping in the spatial domain, let M = HW be the size of the feature in the spatial dimension, and the feature dimension of each group is d k =M / h; H is the height of the image, W is the width of the image.

5. The hyperspectral and ranging image fusion method based on enhanced network according to claim 4 is characterized in that: The calculation process of SMHSA is: Where, d k is the feature dimension size of each attention head, which is used to scale the dot product result. Softmax() is the Softmax function that converts the dot product result into a probability distribution, representing the attention weight of each key vector on the query vector. The obtained fusion feature F spa Flatten the feature and map it to the specified dimension D through a fully connected layer. Add position encoding to the input embedding to distinguish features at different positions. The feature input to the ViT encoder is B0, so the spectral enhancement process of the ViT encoder is: B l =B l-1 +MHSA(LN(B l-1 )),l∈{1,2…L} F vit =B L +MLP(LN(B L )) Where L is the number of layers of the ViT encoder, LN and MHSA are layer normalization and multi-head attention, respectively; B l-1 is the output feature of the l-th layer encoder, MHSA() is the multi-head attention mechanism, LN() is the normalization layer, F vit is the output feature of vit, B L is the final output result of the encoder, MLP() is a multi-layer perceptron; The MLP component maps linear features to a higher-dimensional nonlinear space, and this process learns the global spectral information to obtain F vit ;SpeT uses SMHSA and collaborates with ViT to learn spectral features more comprehensively; The spectrum enhancement process of the SpeT encoder is: B l =B l-1 +SpeMSA(LN(B l-1 )),l∈{1,2…L} F spe =B L +MLP(LN(B L )) Where SpeMSA() is spectral multi-head attention; The output feature obtained is F spe , F spe With F vit They are obtained by learning and extracting different parts of the input features respectively; combining these two features to achieve more comprehensive spectral enhancement; splicing the tokens obtained from the ViT decoder and the SpeT decoder to obtain F con Used to assist reconstruction tasks.

6. The hyperspectral and ranging image fusion method based on enhanced network according to claim 1, characterized in that: In step S1, the spatial spectrum enhancement module combines the F spa ,F vit ,F spe Three features fusion.

7. The method for fusion of hyperspectral and ranging images based on enhanced network according to claim 6, characterized in that: The F obtained by the multi-scale spatial enhancement module and the global spectral enhancement module spa ,F vit ,F spe The three feature fusions include: First, a classifier is used to classify the spatial enhancement features F spa The most representative features are extracted through convolution and maximum pooling layers, and these features are mapped to the predefined number of ground object categories C using the fully connected layer. The features are converted into a C-dimensional probability distribution P using the Softmax function. spa ; Secondly, F vit and F spe It is the feature corresponding to the CLS token after Transformer processing. Using layer normalization and fully connected layers, the spectral features are mapped to C categories, and the C-dimensional probability distribution P is generated through the Softmax function. vit and P spe ; Finally, P is transformed into spa ,P vit ,P spe Fusion: The probability distributions corresponding to the three features are multiplied by the learnable weight coefficients β1, β2, and β3 respectively, and then added together to obtain the final probability distribution P. The initial values ​​of these weight coefficients are all 1 / 3. The fusion formula that represents the fusion result of the probability distribution obtained after different features pass through the softmax classifier is: P=β1P spa +β2P vit +β3P spe Where P is the probability distribution of the final fusion feature, P spa is the probability distribution of spatial features, P vit Obtain the probability distribution of features for vit, P spe is the probability distribution of spectral features.

8. The hyperspectral and ranging image fusion method based on enhanced network according to claim 1 is characterized in that: In step S2, a joint loss function including constraint loss and classification loss is constructed, including: In the decoder stage, the tokens output from ViT and SpeT are concatenated to form a new token F con ; F con The spatial resolution of the original data is restored by upsampling at different scales; the features of different scales are processed using 3×3 convolution and sigmoid function. and The constraint loss uses mean square error as the loss function, and the formula is as follows: Where, L con To balance the constraint loss, Represent the reconstructed HSI and LiDAR data respectively, N is the number of samples; these features correspond to the input HSI cube and LiDAR cube L con , the constraint loss is calculated by minimizing the difference between these reconstructed features and the original input data; n is a variable, ranging from 1 to N, are the original HIS data of different scales, The original LiDAR data of different scales; The classification loss uses the cross entropy loss function to calculate the difference between the category probability distribution predicted by the model and the actual label. The formula is as follows: Where, L cls is the classification loss, N is the number of samples in each batch, y n is the true label of the nth sample, p n is the predicted class probability corresponding to the nth sample; In order to optimize the reconstruction and classification tasks simultaneously, the constraint loss and classification loss are combined to form a joint loss function; the formula is as follows: L=βL con +L cls Where β is the weight coefficient used to balance the contribution of reconstruction loss and classification loss; β is set to 1 / 6.

9. The method for fusion of hyperspectral and ranging images based on enhanced network according to claim 8, characterized in that: In step S3, the joint loss function L considers both feature reconstruction and classification tasks, and the loss L is constrained by balancing con and classification loss L cls , achieving collaborative optimization in feature learning and classification performance.

10. A hyperspectral and ranging image fusion device based on an enhanced network, characterized in that: The device implements the hyperspectral and ranging image fusion method based on the enhanced network as described in any one of claims 1 to 9, and the device includes: The multi-scale spatial enhancement module is used to extract multi-scale features from hyperspectral and lidar data, and the final result is the multi-scale spatial enhancement feature F spa , so that the information between different layers can complement and enhance each other and generate feature representation; The global spectral enhancement module is used to achieve global spectral enhancement using a dual-branch Transformer structure, including a Vision Transformer branch and a Spectral Transformer branch; the Vision Transformer branch is used to enhance the spectrum to obtain F vit , the Spectral Transformer branch collaborates with the Vision Transformer branch to learn spectral features and obtain F spa ; The spatial spectrum enhancement module is used to transform the F obtained by the multi-scale spatial enhancement module and the global spectrum enhancement module into spa ,F vit ,F spe The three features are fused to obtain the final probability distribution P for classification; The hierarchical spatial-spectral enhancement network training module is used to construct a joint loss function composed of the reconstruction loss L con And the classification loss constitutes L cls ; Balance the constraint loss and classification loss to complete the collaborative optimization of feature reconstruction and classification tasks.

Images