A hyperspectral and LiDAR data joint classification method based on channel modulation moving convolution and space-frequency enhancement

Abstract

A hyperspectral and LiDAR data joint classification method based on channel modulation moving convolution and space-frequency enhancement belongs to the technical field of multi-source remote sensing image classification. The present application aims at the problems of great difference between hyperspectral image and LiDAR data modalities, more redundant responses, insufficient local structure representation and limited fusion discriminability, constructs a joint classification network based on channel modulation moving convolution and space-frequency enhancement, respectively uses a channel modulation moving convolution module to extract spectral-spatial features and elevation structure features, enhances effective channel response and suppresses weak discriminative feature propagation, further introduces a space-frequency feature enhancement module to strengthen channel dependence, local structure and global frequency response from the spatial domain and the frequency domain, then uses a bidirectional cross-modal fusion module to realize complementary interaction of heterogeneous features, relieve the problems of modal difference and inconsistent distribution between heterogeneous data, and improve the mining ability of the model to cross-modal associated features. The method can be applied to the ground feature classification task of multi-source remote sensing data.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image classification, specifically relating to a joint classification method for hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement. Background Technology

[0002] With the development of remote sensing imaging technology, joint classification of hyperspectral images and LiDAR data has become an important direction for multi-source remote sensing ground object identification. Hyperspectral images can provide rich continuous spectral information, which is helpful in distinguishing ground object categories with different material properties, but they are insufficient in expressing elevation structure. LiDAR data can reflect surface elevation and geometric structure features, which is helpful in characterizing targets such as buildings, roads, and vegetation, but its spectral discrimination capability is limited. Therefore, how to fully integrate the spectral-spatial information of hyperspectral images with the elevation structure information of LiDAR data is a key issue in improving the classification accuracy of complex scenes.

[0003] Existing methods often employ convolutional networks, attention mechanisms, or feature concatenation for joint classification, but these methods still suffer from problems such as excessive redundant channel responses, insufficient utilization of local structural information, and inadequate cross-modal interaction, limiting the discriminative power and stability of the fused features. Channel-modulated moving convolution, spatial-frequency feature enhancement, and bidirectional cross-modal fusion can improve feature representation capabilities from aspects such as effective channel selection, spatial-frequency feature enhancement, and heterogeneous information interaction. Therefore, constructing a joint classification method for hyperspectral and LiDAR data that integrates the above mechanisms is of great significance for improving the classification accuracy of multi-source remote sensing ground objects. Summary of the Invention

[0004] The purpose of this invention is to address the problems in existing joint classification methods of hyperspectral and LiDAR data, such as insufficient expression of modal features, excessive redundant channel responses, insufficient utilization of spatial and frequency domain information, and insufficient interaction of heterogeneous features, which lead to limited accuracy and stability in the classification of complex ground features. Therefore, this invention proposes a joint classification method of hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a joint classification method for hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement, the method specifically including the following steps:

[0006] Step 1: Obtain LiDAR data and hyperspectral image data from the dataset;

[0007] Step 2: Perform dimensionality reduction processing on the acquired hyperspectral image data to obtain dimensionality-reduced hyperspectral image data;

[0008] Step 3: Slice the LiDAR data and the dimensionality-reduced hyperspectral image data respectively, and then divide the sliced ​​LiDAR data blocks and hyperspectral image data blocks into two parts: training set and test set.

[0009] Step 4: Construct a joint classification network based on channel modulation moving convolution and spatial frequency enhancement. Train the joint classification network using the training set until the maximum number of training iterations is reached or the joint classification network meets the convergence condition, and then stop training to obtain the trained joint classification network.

[0010] The joint classification network includes a channel modulation moving convolution module, a space-frequency feature enhancement module, a bidirectional cross-modal fusion module, and a classification block. The working process of the joint classification network based on channel modulation moving convolution and space-frequency enhancement is as follows:

[0011] The dimensionality-reduced hyperspectral image data blocks are used as inputs for hyperspectral feature extraction, and the LiDAR data blocks are used as inputs for LiDAR feature extraction.

[0012] The dimensionality-reduced hyperspectral image data blocks are input into the channel modulation moving convolution module. Through local spatial response modeling, channel projection, and channel modulation, the spectral-spatial features of the hyperspectral image are extracted.

[0013] The LiDAR data block is input into the channel modulation moving convolution module, and the elevation structure features of the LiDAR data are extracted through local elevation structure modeling, channel projection and channel modulation.

[0014] The extracted hyperspectral spectral-spatial features and LiDAR elevation structure features are input into the space-frequency feature enhancement module, respectively. Channel description features are constructed using global channel statistics, and the two modal features are adaptively enhanced by combining spatial domain channel response and frequency domain channel response to obtain enhanced hyperspectral features and enhanced LiDAR features.

[0015] The enhanced hyperspectral features and enhanced LiDAR features are input into the bidirectional cross-modal fusion module. Through bidirectional information interaction between the hyperspectral features and LiDAR features, the fused multimodal features are obtained.

[0016] The fused multimodal features are fed into the classification block to obtain the final classification result;

[0017] Step 5: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified to obtain the classification result.

[0018] The beneficial effects of this invention are:

[0019] This invention captures joint spectral-spatial features from hyperspectral image data and extracts elevation structure features from LiDAR data. Through a channel modulation moving convolution module, it performs local context modeling, channel projection, channel-by-channel spatial response modeling, and residual channel modulation, enhancing key channel responses, suppressing the propagation of redundant and weakly discriminative features, and improving the expressive power of both modalities. Furthermore, this invention introduces a space-frequency feature enhancement module, utilizing global pooling to construct channel description features and combining spatial domain channel dependence and frequency domain channel response for adaptive recalibration of features, enhancing the expressive power of local structure, salient channels, and frequency domain distribution information. Based on this, a bidirectional cross-modal fusion module enables complementary interaction between hyperspectral spectral-spatial information and LiDAR elevation structure information, avoiding insufficient information utilization caused by simple stitching or one-way fusion. Finally, the fused multimodal features are processed by a classification block to output land cover category results, thereby improving classification accuracy and stability in complex remote sensing scenarios. Attached Figure Description

[0020] Figure 1 This is a structural diagram of a joint classification method for hyperspectral and LiDAR data based on channel-modulated moving convolution and spatial frequency enhancement.

[0021] Figure 2 This is a training flowchart for a joint classification method of hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement.

[0022] Figure 3 This is a structural diagram of the channel modulation moving convolution module;

[0023] Figure 4 This is a structural diagram of the space-frequency feature enhancement module;

[0024] Figure 5 This is a structural diagram of a bidirectional cross-modal fusion module;

[0025] Figure 6 It is a classification block structure diagram;

[0026] Figure 7 This is a table showing the land cover categories and sample size statistics for the Trento dataset;

[0027] Figure 8 This is a statistical table of land cover categories and sample sizes for the Augsburg dataset;

[0028] Figure 9 This is a statistical table of land cover categories and sample size for the Houston2013 dataset;

[0029] Figure 10 It is the Trento dataset;

[0030] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the ground truth image;

[0031] Figure 11 It is the Augsburg dataset;

[0032] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the ground truth image;

[0033] Figure 12 It is the Houston 2013 dataset;

[0034] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the ground truth image;

[0035] Figure 13 The results are subjective classifications of different datasets based on a joint classification method of hyperspectral and LiDAR data using channel-modulated moving convolution and spatial frequency enhancement.

[0036] In the image, (a) is Trento, (b) is Augsburg, and (c) is Houston 2013. Detailed Implementation

[0037] Specific implementation method one: Combining Figure 1 and Figure 2 This embodiment describes a joint classification method for hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement. The method specifically includes the following steps:

[0038] Step 1: Obtain LiDAR data and hyperspectral image data from the dataset;

[0039] For each hyperspectral image obtained from the dataset, LiDAR data of the corresponding region of the hyperspectral image was also obtained simultaneously.

[0040] Step 2: Perform dimensionality reduction processing on the acquired hyperspectral image data to obtain dimensionality-reduced hyperspectral image data;

[0041] Step 3: Slice the LiDAR data and the dimensionality-reduced hyperspectral image data respectively, and then divide the sliced ​​LiDAR data blocks and hyperspectral image data blocks into two parts: training set and test set.

[0042] When partitioning, the LiDAR data corresponding to the same region and the dimensionality-reduced hyperspectral image data need to be partitioned into the training set or test set at the same time, so that when inputting into the model later, the LiDAR data corresponding to the same region and the dimensionality-reduced hyperspectral image data can be input into the model at the same time.

[0043] Step 4: Construct a joint classification network based on channel modulation moving convolution and spatial frequency enhancement. Train the constructed joint classification network using the training set until the maximum number of training iterations is reached or the classification accuracy of the network on the test set no longer improves. Then stop training to obtain the trained joint classification network.

[0044] The joint classification network includes a channel modulation moving convolution module, a space-frequency feature enhancement module, a bidirectional cross-modal fusion module, and a classification block. The working process of the joint classification network based on channel modulation moving convolution and space-frequency enhancement is as follows:

[0045] The dimensionality-reduced hyperspectral image data blocks are used as inputs for hyperspectral feature extraction, and the LiDAR data blocks are used as inputs for LiDAR feature extraction.

[0046] The dimensionality-reduced hyperspectral image data blocks are input into the channel modulation moving convolution module. Through local spatial response modeling, channel projection, and channel modulation, the spectral-spatial features of the hyperspectral image are extracted.

[0047] The LiDAR data block is input into the channel modulation moving convolution module, and the elevation structure features of the LiDAR data are extracted through local elevation structure modeling, channel projection and channel modulation.

[0048] The extracted hyperspectral spectral-spatial features and LiDAR elevation structure features are input into the space-frequency feature enhancement module, respectively. Channel description features are constructed using global channel statistics, and the two modal features are adaptively enhanced by combining spatial domain channel response and frequency domain channel response to obtain enhanced hyperspectral features and enhanced LiDAR features.

[0049] The enhanced hyperspectral features and enhanced LiDAR features are input into the bidirectional cross-modal fusion module. Through bidirectional information interaction between the hyperspectral features and LiDAR features, the fused multimodal features are obtained.

[0050] The fused multimodal features are fed into the classification block to obtain the final classification result;

[0051] Step 5: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified to obtain the classification result.

[0052] Before being input into the classification network, both the hyperspectral image and LiDAR data of the region to be classified need to be preprocessed. Specifically, the hyperspectral image needs to be reduced in dimension and sliced, and the LiDAR data needs to be sliced.

[0053] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the dimensionality reduction processing of the acquired hyperspectral image data is performed using principal component analysis.

[0054] The other steps and parameters are the same as in Specific Implementation Method 1.

[0055] Specific implementation method three: Combining Figure 3 This embodiment is described below. The difference between this embodiment and specific embodiments one or two is that the operation of the channel modulation moving convolution module is as follows:

[0056] The dimensionality-reduced hyperspectral image data blocks and LiDAR data blocks are respectively input into the channel modulation moving convolution module. The channel modulation moving convolution module first performs neighborhood context modeling through a first local convolutional unit composed of a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a Hard-Sigmoid activation function layer to obtain initial local response features; then, it performs channel projection and inter-channel information integration through a first pointwise convolutional unit composed of a 1×1 pointwise convolutional layer, a batch normalization layer, and a Hard-Sigmoid activation function layer to obtain channel projection features.

[0057] The channel projection features are input into a deep convolutional unit consisting of a 3×3 deep convolutional layer, a batch normalization layer, and a Hard-Sigmoid activation function layer. Local spatial response modeling is performed within each channel to obtain deep spatial features. Subsequently, global average pooling and global max pooling are performed on the deep spatial features, and channel modulation weights are generated through a shared channel mapping structure.

[0058] Based on the channel modulation weights and learnable scaling parameters, the depth spatial features are recalibrated in residual form to enhance the effective channel response and suppress the propagation of redundant features. The recalibrated features are then input into a second pointwise convolutional unit consisting of a 1×1 pointwise convolutional layer and a batch normalization layer for channel integration, and residuals are added to the channel projection features to obtain residual fused features.

[0059] The residual fusion features are sequentially input into the ReLU activation function layer and the second local convolutional unit for feature refinement. The second local convolutional unit includes a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer, ultimately obtaining the output features of the channel modulation moving convolution module. Specifically, when the input is a hyperspectral image data block, the output is a hyperspectral spectral-spatial feature, and when the input is a LiDAR data block, the output is a LiDAR elevation structure feature.

[0060] Other steps and parameters are the same as in specific implementation method one or two.

[0061] The channel modulation moving convolution module, through the synergistic effect of local convolution, pointwise convolution, depthwise convolution, and residual channel modulation, can fully extract local structural information and key channel features from hyperspectral and LiDAR data, and enhance effective response and suppress redundant information.

[0062] Specific implementation method four: Combination Figure 4 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to three is that the working process of the space-frequency feature enhancement module is as follows:

[0063] Use the features output by the channel modulation moving convolution module as input features;

[0064] Global average pooling and global max pooling are performed on the input features to obtain average pooling features and max pooling features; wherein, the global average pooling is used to obtain the overall response statistics of each channel, and the global max pooling is used to obtain the significant response statistics of each channel;

[0065] The average pooling feature and the max pooling feature are added element by element to obtain the channel description feature that includes both the overall response statistics and the significant response statistics.

[0066] Remove the spatial dimension from the channel description features and transpose the dimensions to obtain one-dimensional channel sequence features;

[0067] One-dimensional channel sequence features are input into a one-dimensional convolutional layer in the spatial domain. Local dependencies between adjacent channels are established along the channel sequence direction to obtain the spatial domain channel response. The one-dimensional convolutional layer in the spatial domain has 1 input channel and 1 output channel, and its kernel size is adaptively determined according to the number of channels of the input features.

[0068] The one-dimensional channel sequence features are subjected to real-valued fast Fourier transform along the channel sequence direction to obtain the frequency domain channel features;

[0069] Amplitudes are extracted from the frequency domain channel features to obtain the frequency domain amplitude response used to characterize the frequency distribution of the channel sequence;

[0070] The frequency domain amplitude response is input into a one-dimensional frequency domain convolutional layer, and local frequency domain dependency modeling is performed along the frequency response dimension to obtain frequency domain modeling features; wherein, the one-dimensional frequency domain convolutional layer has 1 input channel, 1 output channel, 3 convolutional kernels, and 1 padding size;

[0071] The frequency domain modeling features are subjected to inverse real-valued fast Fourier transform to restore the frequency domain modeling features to the channel sequence space, thus obtaining the frequency domain channel response;

[0072] The spatial domain channel response and the frequency domain channel response are weighted and fused according to a learnable fusion coefficient to obtain the dual-domain fused channel response; wherein, the learnable fusion coefficient is used to adaptively adjust the contribution ratio of the spatial domain channel response and the frequency domain channel response in the dual-domain fused channel response;

[0073] The dual-domain fusion channel response is input into the Sigmoid activation function to obtain the normalized channel attention weights;

[0074] The normalized channel attention weights are dimension-restored to correspond to the channel dimensions of the input features;

[0075] The channel attention weights after dimension restoration are multiplied channel by channel with the input features to obtain the output features of the spatial frequency feature enhancement module.

[0076] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0077] The spatial frequency feature enhancement module, through the synergistic effect of global channel statistics, spatial domain channel dependency modeling, and frequency domain channel response modeling, can fully enhance the key channel responses in hyperspectral and LiDAR features, and improve the model's ability to express local structure, salient features, and frequency distribution information.

[0078] Specific Implementation Method Five: Combining Figure 5 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to four is that the working process of the bidirectional cross-modal fusion module is as follows:

[0079] The hyperspectral features and LiDAR features output by the spatial frequency feature enhancement module are reconstructed in dimension, and the two-dimensional spatial features are converted into sequence features and then subjected to layer normalization.

[0080] In the hyperspectral feature update direction, the normalized LiDAR features are mapped to query features, and the normalized hyperspectral features are mapped to key features and value features. The cross-modal attention response of LiDAR features to hyperspectral features is obtained through matrix multiplication, scale normalization, and the Softmax function. Then, the response is multiplied with the hyperspectral value features to obtain the hyperspectral interactive features that absorb LiDAR elevation structure information. The hyperspectral interactive features are added to the original hyperspectral features by residual addition, and then refined by layer normalization and multilayer perceptron to obtain the updated hyperspectral features.

[0081] In the LiDAR feature update direction, normalized hyperspectral features are mapped to query features, and normalized LiDAR features are mapped to key features and value features. The cross-modal attention response of hyperspectral features to LiDAR features is obtained through matrix multiplication, scale normalization, and the Softmax function. Then, the response is multiplied with the LiDAR value features to obtain the LiDAR interactive features that absorb hyperspectral spectral-spatial information. The residuals of the LiDAR interactive features and the original LiDAR features are added together, and then refined by layer normalization and multilayer perceptron to obtain the updated LiDAR features.

[0082] The updated hyperspectral and LiDAR features are averaged along the sequence dimension to obtain the hyperspectral global descriptive features and the LiDAR global descriptive features, and then the two are concatenated to obtain the fused multimodal features.

[0083] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0084] The bidirectional cross-modal fusion module can fully explore the complementary correlation between spectral-spatial information and elevation structure information through bidirectional attention interaction between hyperspectral features and LiDAR features, thereby enhancing the discriminativeness and expression stability of the fused features.

[0085] Specific Implementation Method Six: Combination Figure 6 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to five is that the working process of the classification block is as follows:

[0086] The multimodal features output by the bidirectional cross-modal fusion module are input into the classification block. The classification block sequentially passes through a layer normalization layer, a fully connected layer, a Gaussian error linear unit (GELU) nonlinear activation layer, a random deactivation layer, and a final fully connected layer to perform distribution normalization, high-dimensional mapping, nonlinear transformation, and class space mapping on the fused features. Then, the Softmax function outputs the normalized class probability distribution, and the pixel-level land cover classification label is determined according to the probability maximization criterion.

[0087] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0088] The classification block can convert the fusion representation formed by hyperspectral spectral-spatial information and LiDAR elevation structure information into a probability distribution of land cover categories, thereby achieving pixel-level land cover classification.

[0089] This invention proposes a joint classification method for hyperspectral and LiDAR data based on channel-modulated moving convolution and spatial frequency enhancement. The implementation process of the method is shown in Table 1:

[0090] Table 1. Algorithm flow of the joint classification network structure based on channel modulation moving convolution and spatial frequency enhancement

[0091] The specific implementation steps are as follows:

[0092] Step 1: Acquire hyperspectral image data and LiDAR data. The data used is a public dataset, and the two types of data are guaranteed to correspond to each other in spatial location.

[0093] Step 2: Use PCA (Principal Component Analysis) to reduce the dimensionality of the obtained hyperspectral image data, thereby reducing the number of bands in the original hyperspectral data, reducing data redundancy, and improving computational efficiency.

[0094] Step 3: Preprocess the LiDAR data and the dimensionality-reduced hyperspectral image data. Divide the data into training and test sets based on the number of labeled pixel samples, and slice the LiDAR data and the dimensionality-reduced hyperspectral image data respectively.

[0095] Step 4: Train and classify the joint classification network based on channel modulation moving convolution and spatial frequency enhancement.

[0096] Step 4.1: Construct a joint classification network based on channel modulation moving convolution and spatial frequency enhancement. The training and test set images used in this invention both have an 11×11 pixel resolution. For example... Figure 1 As shown, M×N represents the spatial dimension, and L represents the number of bands in the original hyperspectral image. After PCA dimensionality reduction and slicing, a series of s×s×d hyperspectral image blocks and s×s LiDAR data blocks are obtained, where d represents the number of bands after dimensionality reduction.

[0097] Step 4.2: Extract hyperspectral image features and LiDAR features using the channel modulation moving convolution module. This module first performs local context modeling through 3×3 2D convolution, batch normalization, and Hard-Sigmoid activation, then completes channel projection through 1×1 pointwise convolution, and extracts channel-wise spatial responses through 3×3 depth convolution. Subsequently, channel modulation weights are generated using global average pooling and global max pooling to perform residual channel recalibration on the depth spatial features to enhance effective responses and suppress redundant features. Finally, the hyperspectral spectral-spatial features and LiDAR elevation structure features are obtained through 1×1 pointwise convolution, residual connection, ReLU activation, and 3×3 2D convolution for refinement.

[0098] Step 4.3: Enhance the hyperspectral and LiDAR features using the spatial-frequency feature enhancement module. This module constructs channel description features through global average pooling and global max pooling, and converts them into a one-dimensional channel sequence. On the one hand, it models spatial domain channel dependencies through one-dimensional convolution; on the other hand, it obtains frequency domain channel responses through real-valued fast Fourier transform, frequency domain amplitude modeling, and inverse real-valued fast Fourier transform. Finally, it performs learnable weighted fusion of spatial and frequency domain responses, generates channel attention weights through the Sigmoid function, and recalibrates the input features channel by channel to obtain the enhanced hyperspectral and LiDAR features.

[0099] Step 4.4: Utilize the bidirectional cross-modal fusion module to complete the interactive fusion of hyperspectral and LiDAR features. The enhanced hyperspectral and LiDAR features are reconstructed into sequence features and subjected to layer normalization. In the LiDAR feature update direction, using hyperspectral features as the query end and LiDAR features as the key end, LiDAR interactive features absorbing hyperspectral spectral-spatial information are obtained. In the hyperspectral feature update direction, using LiDAR features as the query end and hyperspectral features as the key end, hyperspectral interactive features absorbing LiDAR elevation structure information are obtained. Both types of interactive features are refined through residual connections, layer normalization, and multilayer perceptrons, followed by average pooling. The resulting global descriptive features are then concatenated to form the fused multimodal features.

[0100] Step 4.5: Input the fused multimodal features into the classification block for land cover category determination. The classification block performs distribution standardization, high-dimensional mapping, nonlinear transformation, and category space mapping on the fused features through a layer normalization layer, a fully connected layer, a GELU nonlinear activation layer, a random deactivation layer, and a final fully connected layer. The Softmax function outputs the normalized category probability distribution, and the land cover category label of the pixel to be classified is determined according to the maximization probability criterion. During the training phase, cross-entropy loss is used to optimize the joint classification network, and during the testing phase, pixel-level classification results are directly output.

[0101] Step 5: Classification Result Output and Analysis. The trained joint classification network is used to jointly classify the hyperspectral image and LiDAR data of the region to be classified, obtaining pixel-level land cover category labels. A remote sensing classification map is then reconstructed based on the pixel spatial location. Simultaneously, the overall classification accuracy, average classification accuracy, and Kappa coefficient are used to evaluate the classification results.

[0102] The network model training and classification result verification experiments of this invention were completed on the following platform:

[0103] The hardware configuration includes an Intel Core i7-13700HX processor, 16GB DDR5 5600MHz dual-channel memory, an NVIDIA GeForce RTX 4070 laptop GPU, and a storage system consisting of a 1TB PCIe 4.0×4 NVMe solid-state drive (SSD) and an optional 2TB hard disk drive (HDD). The operating system is Windows 11 Home Chinese Edition with integrated Legion ColdFront 5.0 cooling. The experiments used three well-known multimodal remote sensing datasets: the Trento dataset, the Augsburg dataset, and the Houston 2013 dataset. The land cover categories and sample sizes for each dataset are as follows: Figure 7 , Figure 8 and Figure 9 As shown, different colors represent different land cover categories. Detailed statistical information and feature descriptions for each dataset are as follows: Figure 10 , Figure 11 and Figure 12 As shown.

[0104] 1. The Trento dataset, collected in a typical agroforestry ecoregion in the province of Trento, Italy (46°04'N, 11°07'E), consists of data from simultaneous observations by an airborne hyperspectral imaging system and lidar. It has become a standard test set for multimodal remote sensing classification research. The hyperspectral imagery in this dataset contains 600×166 pixels in spatial dimension and 63 continuous spectral bands, covering the visible-near-infrared region from 420.89 to 989.09 nm (average spectral resolution 9.2 nm), with a spatial resolution of 1 meter. It features accurate radiometric and geometric correction characteristics. The accompanying LiDAR elevation data generates a digital surface model (DSM) using 1550 nm pulsed laser ranging. Its single-channel grayscale values ​​represent surface elevation information and maintain strict spatial registration with the hyperspectral image data. Ground real-world annotations cover six typical land features: apple trees, buildings, ground, trees, vineyards, and roads.

[0105] 2. The Augsburg dataset, collected over Augsburg, Germany, aims to provide benchmark data for multimodal remote sensing data analysis. The hyperspectral imagery (HSI) data was acquired by the advanced DAS-EOC HySpex sensor, with a spectral resolution covering 180 bands and wavelengths ranging from 0.4 μm to 2.5 μm, providing rich spectral information for material identification and classification. The LiDAR digital surface model (LiDAR-DSM) data was collected by the DLR-3K system, renowned for its high spatial resolution. To achieve spatial consistency between HSI and LiDAR data in multimodal fusion, the spatial resolution of both datasets was uniformly downsampled to 30 meters to ensure efficient and accurate data fusion. The dataset has a spatial dimension of 332 × 485 pixels and covers seven different land cover categories.

[0106] 3. The Houston2013 dataset, released in 2013 as part of the IEEE GRSS Data Fusion Competition, covers the University of Houston campus and surrounding urban areas. This dataset contains spatially registered hyperspectral images and LiDAR data. The hyperspectral images contain 144 spectral bands ranging from 0.38 to 1.05 μm, while the LiDAR data consists of single-band elevation structure data for the same region. Both datasets have a spatial size of 349 × 1905 pixels and a spatial resolution of 2.5 m. The dataset includes 15 categories of urban land features, such as grassland, trees, soil, water bodies, residential areas, commercial areas, roads, highways, railways, parking lots, tennis courts, and running tracks. Due to the spectral similarities, complex spatial structures, and overlapping category boundaries among different land features, this dataset effectively validates the classification performance of the joint hyperspectral and LiDAR classification method in complex urban remote sensing scenarios.

[0107] In evaluating model performance, this embodiment employs three widely recognized and commonly used objective evaluation metrics in the industry: Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (K). These metrics are all calculated based on the Confusion Matrix, which can comprehensively and objectively reflect the accuracy and reliability of the model in classification tasks.

[0108] The following sections will introduce these evaluation indicators in detail:

[0109] (1) Confusion Matrix: The confusion matrix is ​​an n x n error matrix used to systematically evaluate the performance of classification models. The detailed composition of this matrix can be found in Table 2. It presents the correspondence between the model's predictions and the actual labels in matrix form, and is a key tool for calculating classification accuracy. Each column of the confusion matrix represents the model's predicted category, and each row corresponds to the actual true category. This matrix structure can intuitively reveal the confusion between different categories, i.e., whether a certain category is incorrectly predicted as another category. Its name, "confusion matrix," aptly reflects this function; through its visualization, researchers can clearly identify the model's strengths and weaknesses in classification tasks.

[0110] Table 2. Composition of the confusion matrix

[0111]

[0112] (2) Overall Classification Accuracy (OA): OA is a fundamental and crucial metric in evaluating classification models, used to measure the proportion of correctly classified samples out of the total number of samples. It reflects the accuracy of the model in classifying all categories globally, and is specifically calculated as the ratio of correctly classified samples to the total number of samples. Overall classification accuracy can intuitively reflect the overall performance of the classification algorithm and is an important reference for evaluating the model's classification effect.

[0113]

[0114] Where: TP (True Positive): Number of samples correctly predicted as positive. TN (True Negative): Number of samples correctly predicted as negative. FN (False Negative): Number of samples incorrectly predicted as negative. FP (False Positive): Number of samples incorrectly predicted as positive.

[0115] (3) Average Classification Accuracy (AA): AA is one of the key metrics used to evaluate the performance of classification algorithms. It measures the model's ability to classify data across different categories by calculating the average classification accuracy for each category. Specifically, AA first calculates the classification accuracy for each category, which is the ratio of the number of correctly classified samples in each category to the total number of samples in that category. Then, it averages these classification accuracy values. Average classification accuracy differs from overall classification accuracy and focuses on evaluating the algorithm's classification accuracy for each type of land cover.

[0116]

[0117] Where: n represents the number of rows and columns of the classification matrix, This represents the number of samples correctly classified into the i-th category. This represents the number of samples that were misclassified in the i-th category.

[0118] (4) Kappa coefficient (K): The Kappa coefficient is an important indicator for measuring classification accuracy, used to evaluate the degree of improvement of the classification results compared to completely random classification. It reflects the actual accuracy of the classification model by quantifying the difference between the actual classification results and the random classification results. As a supplementary indicator to the confusion matrix, the Kappa coefficient can more intuitively reveal the quality of classification accuracy:

[0119]

[0120] Where: OA (Overall Accuracy): Overall classification accuracy; PE (Probability of Random Agreement): Expected accuracy of random classification, calculated using the following formula:

[0121]

[0122] Where: N is the total number of samples. This represents the number of samples correctly classified into the i-th category. This represents the number of samples that were incorrectly predicted as positive for the i-th category. This represents the number of samples that were incorrectly predicted as negative for the i-th category.

[0123] The objective classification results of the network model used in this invention on three datasets are shown in Table 3. From left to right, the first column is the dataset; the second column is the OA data of the classification results; the third column is the AA data of the classification results; and the fourth column is the K × 100 data of the classification results. The subjective classification results are as follows... Figure 13 As shown, (a) represents the classification effect of the Trento dataset, (b) represents the classification effect of the Augsburg dataset, and (c) represents the classification effect of the Houston2013 dataset. The higher the accuracy, the less salt-and-pepper noise in the image.

[0124] Table 3

[0125]

[0126] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A hyperspectral and LiDAR data joint classification method based on channel modulation moving convolution and space-frequency enhancement, characterized in that, The method specifically includes the following steps: Step 1: Obtain LiDAR data and hyperspectral image data from the dataset; Step 2: Perform dimensionality reduction processing on the acquired hyperspectral image data to obtain dimensionality-reduced hyperspectral image data; Step 3: Slice the LiDAR data and the dimensionality-reduced hyperspectral image data respectively, and then divide the sliced ​​LiDAR data blocks and hyperspectral image data blocks into two parts: training set and test set. Step 4: Construct a joint classification network based on channel modulation moving convolution and spatial frequency enhancement. Train the joint classification network using the training set until the maximum number of training iterations is reached or the joint classification network meets the convergence condition, and then stop training to obtain the trained joint classification network. The joint classification network includes a channel-modulated moving convolutional module, a space-frequency feature enhancement module, a bidirectional cross-modal fusion module, and a classification block. The working process based on the joint classification network is as follows: The dimensionality-reduced hyperspectral image data blocks are used as inputs for hyperspectral feature extraction, and the LiDAR data blocks are used as inputs for LiDAR feature extraction. The dimensionality-reduced hyperspectral image data blocks are input into the channel modulation moving convolution module. Through local spatial response modeling, channel projection, and channel modulation, the spectral-spatial features of the hyperspectral image are extracted. The LiDAR data block is input into the channel modulation moving convolution module, and the elevation structure features of the LiDAR data are extracted through local elevation structure modeling, channel projection and channel modulation. The extracted hyperspectral spectral-spatial features and LiDAR elevation structure features are input into the space-frequency feature enhancement module, respectively. Channel description features are constructed using global channel statistics, and the two modal features are adaptively enhanced by combining spatial domain channel response and frequency domain channel response to obtain enhanced hyperspectral features and enhanced LiDAR features. The enhanced hyperspectral features and enhanced LiDAR features are input into the bidirectional cross-modal fusion module. Through bidirectional information interaction between the hyperspectral features and LiDAR features, the fused multimodal features are obtained. The fused multimodal features are fed into the classification block to obtain the final classification result; Step 5: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified to obtain the classification result.

2. The method for joint classification of hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement according to claim 1, characterized in that, The dimensionality reduction processing of the acquired hyperspectral image data is performed using principal component analysis.

3. The method for joint classification of hyperspectral and LiDAR data based on channel modulation moving convolution and spatial frequency enhancement according to claim 2, characterized in that, The working process of the channel modulation moving convolution module is as follows: The dimensionality-reduced hyperspectral image data block and LiDAR data block are respectively input into the first local convolutional unit. The first local convolutional unit includes a two-dimensional convolutional layer with a kernel size of 3×3, a batch normalization layer, and a Hard-Sigmoid activation function layer. The input features are modeled in the neighborhood context through the first local convolutional unit to obtain the initial local response features. The initial local response features are input into the first pointwise convolutional unit, which includes a pointwise convolutional layer with a kernel size of 1×1, a batch normalization layer, and a Hard-Sigmoid activation function layer. The initial local response features are then processed by the first pointwise convolutional unit to perform channel projection and inter-channel information integration, resulting in channel projection features. The channel projection features are input into a deep convolutional unit, which includes a 3×3 kernel deep convolutional layer, a batch normalization layer, and a Hard-Sigmoid activation function layer; the deep convolutional unit independently models the local spatial response within each channel to obtain the deep spatial features. Global average pooling and global max pooling are performed on the depth spatial features to obtain average pooling channel descriptors and max pooling channel descriptors. The average pooling channel descriptor is used to characterize the overall response intensity of each channel, and the max pooling channel descriptor is used to characterize the local salient response of each channel. The average pooling channel descriptor and the max pooling channel descriptor are respectively input into a shared channel mapping structure, which includes a first 1×1 convolutional layer, a ReLU activation function layer, and a second 1×1 convolutional layer; the channel statistics are nonlinearly mapped through the shared channel mapping structure to obtain the average pooling mapping result and the max pooling mapping result; The average pooling mapping result and the max pooling mapping result are fused together, and channel modulation weights are generated by using the Sigmoid activation function; Based on the channel modulation weights and learnable scaling parameters, the depth space features are recalibrated in residual form to obtain recalibrated features. The residual form of channel recalibration is used to enhance the effective channel response while maintaining the original depth space response and to suppress the propagation of redundant channels or weak discriminative channels. The channel recalibrated features are input into the second pointwise convolutional unit, which includes a pointwise convolutional layer with a kernel size of 1×1 and a batch normalization layer; the channel recalibrated features are integrated through the second pointwise convolutional unit to obtain the channel integrated features. The residuals of the channel integration features and the channel projection features are added together to obtain the residual fusion features; The residual fusion features are input into the ReLU activation function layer for non-linear activation, and then input into the second local convolution unit for feature refinement. The second local convolution unit includes a two-dimensional convolution layer with a kernel size of 3×3, a batch normalization layer, and a ReLU activation function layer. The features output by the second local convolutional unit are used as the output features of the channel modulation moving convolutional module. Specifically, when the input is a hyperspectral image data block, the output feature is a hyperspectral spectral-spatial feature; when the input is a LiDAR data block, the output feature is a LiDAR elevation structure feature.

4. The hyperspectral and LiDAR data joint classification method based on channel modulation moving convolution and space-frequency enhancement according to claim 3, characterized in that, The working process of the space frequency feature enhancement module is as follows: Use the features output by the channel modulation moving convolution module as input features; Global average pooling and global max pooling are performed on the input features to obtain average pooling features and max pooling features; wherein, the global average pooling is used to obtain the overall response statistics of each channel, and the global max pooling is used to obtain the significant response statistics of each channel; The average pooling feature and the max pooling feature are added element by element to obtain the channel description feature that includes both the overall response statistics and the significant response statistics. Remove the spatial dimension from the channel description features and transpose the dimensions to obtain one-dimensional channel sequence features; One-dimensional channel sequence features are input into a one-dimensional convolutional layer in the spatial domain. Local dependencies between adjacent channels are established along the channel sequence direction to obtain the spatial domain channel response. The one-dimensional convolutional layer in the spatial domain has 1 input channel and 1 output channel, and its kernel size is adaptively determined according to the number of channels of the input features. The one-dimensional channel sequence features are subjected to real-valued fast Fourier transform along the channel sequence direction to obtain the frequency domain channel features; Amplitudes are extracted from the frequency domain channel features to obtain the frequency domain amplitude response used to characterize the frequency distribution of the channel sequence; The frequency domain amplitude response is input into a one-dimensional frequency domain convolutional layer, and local frequency domain dependency modeling is performed along the frequency response dimension to obtain frequency domain modeling features; wherein, the one-dimensional frequency domain convolutional layer has 1 input channel, 1 output channel, 3 convolutional kernels, and 1 padding size; The frequency domain modeling features are subjected to inverse real-valued fast Fourier transform to restore the frequency domain modeling features to the channel sequence space, thus obtaining the frequency domain channel response; The spatial domain channel response and the frequency domain channel response are weighted and fused according to a learnable fusion coefficient to obtain the dual-domain fused channel response; wherein, the learnable fusion coefficient is used to adaptively adjust the contribution ratio of the spatial domain channel response and the frequency domain channel response in the dual-domain fused channel response; The dual-domain fusion channel response is input into the Sigmoid activation function to obtain the normalized channel attention weights; The normalized channel attention weights are dimension-restored to correspond to the channel dimensions of the input features; The channel attention weights after dimension restoration are multiplied channel by channel with the input features to obtain the output features of the spatial frequency feature enhancement module.

5. The hyperspectral and LiDAR data joint classification method based on channel modulation moving convolution and space-frequency enhancement according to claim 4, characterized in that, The working process of the bidirectional cross-modal fusion module is as follows: The hyperspectral and LiDAR features output by the spatial frequency feature enhancement module are reconstructed in dimension, and the two-dimensional spatial features are converted into sequence features, where each sequence unit corresponds to a local spatial location. Layer normalization was performed on the hyperspectral features and LiDAR features respectively to obtain the normalized hyperspectral features and normalized LiDAR features. In the hyperspectral feature update direction, the normalized LiDAR features are input into the LiDAR query mapping layer to obtain the LiDAR query features; the normalized hyperspectral features are input into the key value mapping layer to obtain the hyperspectral key features and hyperspectral value features. The LiDAR query features and hyperspectral bond features are multiplied by a matrix, and scaled according to the feature dimension. The cross-modal attention response of the LiDAR features to the hyperspectral features is then obtained through the Softmax function. The cross-modal attention response and the hyperspectral value feature are multiplied by a matrix to obtain the hyperspectral interactive feature that absorbs LiDAR elevation structure information. The hyperspectral interactive features are added to the original hyperspectral features by residual addition to obtain the first updated hyperspectral features; The hyperspectral features after the first update are processed by the input layer normalization layer and multilayer perceptron, and then the residuals are added to the hyperspectral features after the first update to obtain the updated hyperspectral features. In the LiDAR feature update direction, the normalized hyperspectral features are input into the hyperspectral query mapping layer to obtain the hyperspectral query features; the normalized LiDAR features are input into the key-value mapping layer to obtain the LiDAR bond features and LiDAR value features. The hyperspectral query features and LiDAR key features are multiplied by a matrix, and scaled according to the feature dimension. The cross-modal attention response of the hyperspectral features to the LiDAR features is then obtained through the Softmax function. The cross-modal attention response and the LiDAR value feature are multiplied by a matrix to obtain the LiDAR interaction feature that absorbs hyperspectral spectral-spatial information. The residuals of the LiDAR interactive features and the original LiDAR features are added together to obtain the LiDAR features after the first update. The normalization layer and multilayer perceptron of the input layer of the first updated LiDAR feature are refined nonlinearly, and the residuals are added to the LiDAR feature of the first updated feature to obtain the updated LiDAR feature. The updated hyperspectral features and updated LiDAR features are averaged along the sequence dimension to obtain the hyperspectral global descriptive features and the LiDAR global descriptive features, respectively. The hyperspectral global descriptive features and the LiDAR global descriptive features are concatenated to obtain the fused multimodal features.

6. The hyperspectral and LiDAR data joint classification method based on channel modulation moving convolution and space-frequency enhancement according to claim 5, characterized in that, The working process of the classification block is as follows: The multimodal features fused by the bidirectional cross-modal fusion module are input into the classification block. The classification block performs high-dimensional mapping and nonlinear transformation on the fused features through a layer normalization layer, a fully connected layer, a Gaussian error linear unit (GELU) nonlinear activation layer, a random deactivation layer, and a final fully connected layer. The normalized class probability distribution is output by the Softmax function, and the pixel-level land cover classification label is determined according to the maximization probability criterion.

Images