Directional perceptual wavelet-based hyperspectral and lidar data classification method

Abstract

This invention relates to the fields of artificial intelligence and remote sensing image processing technology, and in particular provides a method for classifying hyperspectral and lidar data based on direction-aware wavelets. The method includes constructing a multi-scale input setting and dual-domain feature extraction architecture to capture spatial local structure and multi-resolution frequency details; designing a dual-coordinate attention gating module to locate discriminative regions on the structure; designing a frequency-space fusion enhancement module to jointly calibrate the initial fused features in terms of spatial context and frequency domain response; using a cross-domain feature aggregation mechanism to flatten, linearly project, and element-wise add the dual-scale cross-domain fused features to achieve unified feature embedding of multi-scale information; and constructing a directional cross-subband attention module, which adaptively enhances structural features in horizontal, vertical, and diagonal directions through directional subband gating and cross-subband interaction mechanisms. This method alleviates cross-modal semantic differences and enhances the ability to distinguish ground features in complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and remote sensing image processing technology, and in particular to a method for classifying hyperspectral and lidar data based on direction-aware wavelets. Background Technology

[0002] With the rapid development of Earth observation technology, the fusion of multi-source data from hyperspectral imaging (HSI) and lidar (LiDAR) has become a core direction for accurate land cover classification. HSI can provide continuous narrow-band spectral responses and has rich material properties and spectral discrimination information; however, its imaging quality is easily affected by changes in illumination, atmospheric conditions, and shadowing, and it often exhibits strong spectral confusion phenomena such as heterospectral representation of the same object and homospectral representation of different objects in complex urban environments. LiDAR, as an active sensing technology, can provide accurate elevation contours, morphological boundaries, and three-dimensional geometry, effectively compensating for the shortcomings of HSI in height perception and geometric representation.

[0003] Despite the promising future of HSI and LiDAR data fusion, existing methods still face the following technical challenges in achieving efficient fusion: First, direct interaction between heterogeneous modalities in the spatial domain easily introduces cross-modal semantic noise. Due to the significant differences in imaging mechanisms and semantic representations between HSI and LiDAR, direct alignment in the spatial domain leads to inconsistent responses in the fused features and degrades classification performance. Second, existing frequency domain learning methods (such as Fourier transform-based methods) excel at capturing global frequency responses but are relatively lacking in spatial localization capabilities. When processing remote sensing images with complex edges and fine-grained contours, it is difficult to accurately locate the local structural details of HSI and LiDAR. Finally, existing wavelet transform-based fusion frameworks typically treat high-frequency subbands (such as horizontal, vertical, and diagonal subbands) in an isotropic manner. This approach fails to fully utilize the strong directional prior differences of urban features (such as building boundaries and road orientations), limiting the model's accuracy in representing orientation-sensitive structures. Summary of the Invention

[0004] In view of this, the present invention provides a hyperspectral and lidar data classification method based on direction-aware wavelets to effectively alleviate cross-modal semantic differences and enhance the ability to identify ground features in complex scenarios.

[0005] In a first aspect, the present invention provides a method for classifying hyperspectral and lidar data based on direction-aware wavelets, the method comprising:

[0006] Step 1: Construct a multi-scale input setting and dual-domain feature extraction architecture, and capture spatial local structure and multi-resolution frequency details through parallel spatial convolutional neural network (CNN) branches and wavelet branches; Step 2: Based on Step 1, design a dual-coordinate attention gate DCAG module, using parallel paths of mean pooling and max pooling to locate the discriminative region on the structure; Step 3: Based on Step 2, design the frequency-space fusion enhancement module FEB, which combines multi-scale dilated convolution and amplitude modulation based on Fast Fourier Transform (FFT) to jointly calibrate the spatial context and frequency domain response of the preliminary fusion features. Step 4: Based on step 3, the cross-domain feature aggregation mechanism is used to flatten, linearly project, and add the cross-domain fusion features of the two scales to achieve unified feature embedding of multi-scale information. Step 5: Based on step 4, construct a directional cross-subband attention DCSA module, which adaptively enhances the structural features in the horizontal, vertical and diagonal directions through directional subband gating and cross-subband interaction mechanisms.

[0007] Optionally, step 1 includes: Step 11, Data Acquisition and Alignment: Acquire HSI data cubes from the same geographic region that have undergone spatial geometric registration. and LiDAR data ; Step 12, Multi-scale Local Block Construction: For each target center pixel in the image, from... and Two sets of bimodal local image patches with different spatial scales are cropped from the image, and the scale set is set as S = {8}. 8,16 16}; Step 13, Dual-Domain Parallel Branch Construction: At each scale and modality, construct the spatial convolutional neural network (CNN) branch and wavelet branch in parallel to achieve a dual-domain joint representation of spatial and frequency characteristics. The spatial convolutional neural network (CNN) branch extracts basic spatial local structure and neighborhood correlation information from input image patches through concatenated convolutional layers, batch normalization layers, and nonlinear activation layers, obtaining four sets of preliminary spatial features. The wavelet domain branch uses the two-dimensional discrete wavelet transform (DWT) to decompose the input two-dimensional spatial features X into four mutually orthogonal sub-band components, the expression of which is: ; Wherein, LL is the low-frequency approximation component; LH, HL and HH are the high-frequency detail components in the vertical, horizontal and diagonal directions, respectively; then, each component is encoded by an independent convolution Conv and input into the frequency domain feature alignment FDFA module for preliminary cross-frequency band interactive alignment.

[0008] Optionally, step 2 includes: Step 21: Bidirectional pooling feature extraction. The four wavelet domain features, after initial alignment, are extracted along the coordinate attention paths of the mean and the maximum value. Pooling is performed in both the horizontal and vertical directions to obtain horizontal and vertical pooling vectors; mean pooling is used to capture smooth global background features, and max pooling is used to capture high-frequency local salient edges. Step 22: Coordinate attention generation. Within the coordinate attention paths of the mean and the maximum, the horizontal and vertical pooling vectors are concatenated and sequentially passed through a shared-parameter convolution Conv1. 1. Batch Normalization (BN) and ReLU activation are applied; the interacted features are then split again and fed into independent convolutional layers (Conv) and activation functions (Sigmoid) to generate mean-coordinated attention maps. Attention map with maximum coordinates ; Step 23: Adaptive gated path fusion, introducing learnable lightweight gated branches; the lightweight gated branches are applied to wavelet domain features. Global average pooling (GAP) is performed, and mean path weights are adaptively generated using a multilayer perceptron. Path weight with maximum value Attention features of fused spatial coordinates Its expression is: ; Step 24, Channel Calibration and Feature Output: The fused spatial coordinate attention features are then processed. With wavelet domain features Element-by-element multiplication yields preliminary weighted features Subsequently, channel attention based on an activation mechanism is used to calibrate the initial weighted features, and residual connections are made with the original input to output the final refined wavelet domain features. It includes four refined wavelet domain features. , and .

[0009] Optionally, step 3 includes mid-layer fusion refinement and frequency-space collaborative denoising: Step 31: Preliminary multimodal fusion, combining the four sets of preliminary spatial features with the final refined wavelet domain features. The input is processed by the multimodal fusion module (MFM), which uses hyperspectral features as a spatial guide to perform preliminary cross-interactions on the bimodal features, resulting in four sets of fused features, including 8×8 scale spatial domain fusion features. 8×8 scale wavelet domain fusion features Spatial domain fusion features at the 16×16 scale 16×16 scale wavelet domain fusion features ; Step 32: The frequency-space fusion enhancement module (FEB) inputs the four sets of fusion features into the frequency-space fusion enhancement module (FEB) to obtain four sets of enhancement features, including 8×8 scale spatial domain enhancement features. 8×8 scale wavelet domain enhancement features 16×16 scale spatial domain enhancement features 16×16 scale wavelet domain enhancement features ; To eliminate cross-modal semantic conflicts and high-frequency noise in the imaging mechanism of heterogeneous modalities, four sets of fused features are input into the frequency-space fusion enhancement module (FEB) for dual-domain joint calibration. The process is as follows: Step 321, Spatial Enhancement Branch: The four sets of fused features are processed... After dimensionality increase, three parallel expansion coefficients d are fed in. The depthwise separable convolution DWConv of {1, 4, 9} is performed, and then the three results are directly added together to obtain the multi-scale sum. The multi-scale sum is fed into the lightweight neural network operator SG, passes through channel attention CA, and finally undergoes convolution dimensionality reduction to obtain the spatially enhanced feature Z. i Utilizing learnable parameters The spatial calibration feature Y is obtained by adding the residuals. i Where i∈[1,4] Step 322: Frequency domain enhancement branch. Using Fast Fourier Transform (FFT), the spatial calibration feature Y is... i Transformed to the frequency domain, decoupled into an amplitude matrix. and phase matrix : ; FreMLP (Frequency Domain Multilayer Perceptron) is used to analyze the amplitude matrix. Perform global amplitude modulation while preserving the phase matrix. Subsequently, the data is mapped back to the spatial domain using Inverse Fast Fourier Transform (IFFT); the amplitude matrix is ​​then processed separately using a frequency domain multilayer perceptron (FreMLP). Perform a nonlinear mapping, then with the phase matrix Reconstruction is performed by using the Inverse Fast Fourier Transform (IFFT) to return to the spatial domain and obtain the frequency domain modulation features. ; Step 323, Gated Fusion Output: Introduce channel-level learnable frequency domain residual scaling parameters. The spatial calibration feature Y is obtained through a frequency domain gating mechanism. i Frequency domain modulation characteristics Dynamically weighted residual fusion is performed to output the final frequency-space enhancement features. .

[0010] Optionally, step 4 includes cross-domain feature aggregation and multi-scale latent space unification: Step 41: Cross-domain information interaction. The spatial domain enhancement features refined by the frequency-space fusion enhancement module FEB and the corresponding wavelet domain enhancement features are sent to the cross-domain fusion module CDFM to perform multi-dimensional semantic-level deep interaction between spatial topology and multi-resolution frequency domain information. 8×8 scale spatial domain enhancement features and 8×8 scale wavelet domain enhancement features The fusion yields cross-domain fusion features at an 8×8 scale. Enhance the spatial domain features at the 16×16 scale. and wavelet domain enhancement features at the 16×16 scale The fusion yielded cross-domain fusion features at a scale of 16×16. ; Step 42: Latent space mapping. Perform spatial dimension flattening and linear projection operations on the 8×8 and 16×16 dual-scale cross-domain fusion features respectively, mapping them into a latent space with a unified feature dimension to obtain multi-scale feature embedding vectors. and ; Step 43, Topological Reshaping and Aggregation: In the unified feature latent space, the two sets of multi-scale embedded features are structurally aggregated through element-wise addition, as expressed in the following expression: ; In order to leverage the topological prior of spatial neighborhood in the subsequent backend refining stage, the one-dimensional aggregation sequence Reshape and reconstruct into a three-dimensional feature tensor .

[0011] Optionally, step 5 includes back-end refinement of the directional cross-subband attention DCSA module: Before entering the final classifier, the three-dimensional feature tensor Input Directional Cross Subband Attention (DCSA) module to enhance the anisotropic directional geometric priors of linear roads and rectangular building edges; Step 51, Anisotropic Wavelet Decomposition: Using two-dimensional wavelet transform to decompose the three-dimensional feature tensor It is further decomposed into a low-frequency substrate LL and three high-frequency sub-bands in three directions, namely the horizontal direction HL, the vertical direction LH, and the diagonal direction HH. Step 52: Adaptive soft thresholding denoising for each high-frequency sub-band. Introducing learnable threshold parameters and average amplitude response The adaptive soft thresholding operation is used for shrinkage and denoising, and its expression is: ; Step 53: Cross-subband interaction and directional gating. Calculate the average amplitude response of each denoised high-frequency subband along the channel dimension to generate a channel descriptor reflecting the energy in the current direction. Utilize cross-subband Softmax normalization to generate cross-subband attention weights. Simultaneously, assign independent learnable directional gating parameters to subbands in each direction, adaptively emphasizing or suppressing responses in specific directions. These are then weighted and fused into a highly comprehensive directional hybrid high-frequency representation. ; Step 54: Subband Dynamic Reassignment and Reconstruction. A learnable subband allocation matrix is ​​constructed, and redistribution weights for the three directions are generated through channel-level mapping. The mixed high-frequency directional information is then inversely mapped back to the corresponding wavelet subband space to generate new anisotropic subband features. , and Finally, the newly generated anisotropic high-frequency subband is combined with the low-frequency basis LL, and the reconstructed features with anisotropic directional contour characteristics are obtained through inverse discrete wavelet transform (IDWT). ; Step 55: Dual-weighted residual fusion, utilizing channel allocation weights generated by global average pooling and enhanced weights generated by channel attention mechanism, outputs fully refined final decision features through learnable residual coefficients; 3D feature tensor Weights are generated using Global Average Pooling (GAP) and Softmax activation functions. Reconstructing features Generate through channel attention The weighted sum of the two outputs the final decision feature. .

[0012] Optionally, step 5 further includes lightweight classification head decision and end-to-end optimization: step 56, probability mapping, which maps the final decision features. A lightweight classification head consisting of convolutional layers, batch normalization layers, ReLU activation layers, adaptive average pooling layers, and fully connected layers is fed in, mapping the target pixel to the predicted probability distribution of each land cover category. ; Step 57: End-to-end training. During the model training phase, the real ground object labels y and the predicted probability distribution are used. The standard cross-entropy loss, used as a global gradient monitoring signal, is expressed as follows: ; The entire network implements end-to-end gradient backpropagation and network parameter updates based on the Adam optimizer.

[0013] The technical solution provided by this invention includes a method that constructs a multi-scale input setting and dual-domain feature extraction architecture, capturing spatial local structure and multi-resolution frequency details through parallel spatial convolutional neural network (CNN) branches and wavelet branches; designs a dual-coordinate attention gate (DCAG) module, using parallel paths of mean pooling and max pooling to locate discriminative regions on the structure; designs a frequency-space fusion enhancement module (FEB), combining multi-scale dilated convolution and amplitude modulation based on Fast Fourier Transform (FFT) to jointly calibrate the spatial context and frequency domain response of the preliminary fused features; uses a cross-domain feature aggregation mechanism to flatten, linearly project, and element-wise add the dual-scale cross-domain fused features to achieve unified feature embedding of multi-scale information; and constructs a directional cross-subband attention (DCSA) module, which adaptively enhances structural features in horizontal, vertical, and diagonal directions through directional subband gating and cross-subband interaction mechanisms. This method effectively alleviates cross-modal semantic differences and enhances the ability to distinguish ground features in complex scenes. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart of a hyperspectral and lidar data classification method based on direction-aware wavelets provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the dual-coordinate attention gate DCAG module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the frequency-space fusion enhancement module FEB provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the directional cross subband attention DCSA module provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0018] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0019] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0020] This invention provides a method for classifying hyperspectral and lidar data based on direction-aware wavelets, such as... Figures 1 to 4 As shown, the method includes: Step 1: Construct a multi-scale input setting and dual-domain feature extraction architecture, and capture spatial local structure and multi-resolution frequency details through parallel spatial convolutional neural network (CNN) branches and wavelet branches; In this embodiment of the invention, step 1 includes: Step 11, Data Acquisition and Alignment: Acquire HSI data cubes from the same geographic region that have undergone spatial geometric registration. and LiDAR data ; Step 12, Multi-scale Local Block Construction: For each target center pixel in the image, from... and Two sets of bimodal local image patches with different spatial scales are cropped from the image, and the scale set is set as S = {8}. 8,16 16}; Step 13, Dual-Domain Parallel Branch Construction: At each scale and modality, construct the spatial convolutional neural network (CNN) branch and wavelet branch in parallel to achieve a dual-domain joint representation of spatial and frequency characteristics. The spatial convolutional neural network (CNN) branch takes an input image patch and extracts basic spatial local structure and neighborhood correlation information through a series of convolutional layers, batch normalization layers, and nonlinear activation layers, obtaining four sets of preliminary spatial features (the four branches are two 8×8 and two 16×16). The wavelet domain branch uses the two-dimensional discrete wavelet transform (DWT) to decompose the input two-dimensional spatial features X into four mutually orthogonal sub-band components, the expression of which is: ; Wherein, LL is the low-frequency approximation component; LH, HL and HH are the high-frequency detail components in the vertical, horizontal and diagonal directions, respectively; then, each component is encoded by an independent convolution Conv and input into the frequency domain feature alignment FDFA module for preliminary cross-frequency band interactive alignment.

[0021] Step 2: Based on Step 1, design a dual-coordinate attention gate DCAG module. Through the parallel paths of mean pooling and max pooling, enhance the spatial location perception capability of wavelet features to locate discriminative regions on the structure. The four wavelet domain features after preliminary alignment The inputs were fed into the dual-coordinate attention-gated DCAG module to obtain four refined wavelet domain features. , and This allows wavelet features to not only have frequency information, but also a strong sense of absolute spatial location, accurately anchoring the long-distance coordinates of urban features (such as roads and buildings).

[0022] In this embodiment of the invention, step 2 includes: Step 21: Bidirectional pooling feature extraction. The four wavelet domain features, after initial alignment, are extracted along the coordinate attention paths of the mean and the maximum value. Pooling is performed in both the horizontal and vertical directions to obtain horizontal and vertical pooling vectors; mean pooling is used to capture smooth global background features, and max pooling is used to capture high-frequency local salient edges. Step 22: Coordinate attention generation. Within the coordinate attention paths of the mean and the maximum, the horizontal and vertical pooling vectors are concatenated and sequentially passed through a shared-parameter convolution Conv1. 1. Batch Normalization (BN) and ReLU activation are used to significantly reduce computation and allow cross-axis interaction between X and Y axis information. The interacted features are then split again and fed into independent Convolutional (Conv) and Sigmoid activation functions to generate mean-coordinated attention maps. Attention map with maximum coordinates ; Step 23: Adaptive gated path fusion, introducing learnable lightweight gated branches; the lightweight gated branches are applied to wavelet domain features. A global average pooling GAP is performed, and mean path weights are adaptively generated using a multilayer perceptron (Conv-ReLU-Conv). Path weight with maximum value Attention features of fused spatial coordinates Its expression is: ; Step 24, Channel Calibration and Feature Output: The fused spatial coordinate attention features are then processed. With wavelet domain features Element-by-element multiplication yields preliminary weighted features Subsequently, channel attention based on an activation mechanism is used to calibrate the initial weighted features, and residual connections are made with the original input to output the final refined wavelet domain features. It includes four refined wavelet domain features. , and .

[0023] Step 3: Based on Step 2, design the frequency-space fusion enhancement module FEB, which combines multi-scale dilated convolution and amplitude modulation based on Fast Fourier Transform (FFT) to jointly calibrate the spatial context and frequency domain response of the preliminary fusion features, so as to effectively suppress heterogeneous semantic noise. In this embodiment of the invention, step 3 includes mid-layer fusion refinement and frequency-space collaborative denoising: Step 31: Preliminary multimodal fusion, combining the four sets of preliminary spatial features with the final refined wavelet domain features. The input is processed by the multimodal fusion module (MFM), which uses hyperspectral features as a spatial guide to perform preliminary cross-interactions on the bimodal features, resulting in four sets of fused features, including 8×8 scale spatial domain fusion features. 8×8 scale wavelet domain fusion features Spatial domain fusion features at the 16×16 scale 16×16 scale wavelet domain fusion features ; Step 32: The frequency-space fusion enhancement module (FEB) inputs the four sets of fusion features into the frequency-space fusion enhancement module (FEB) to obtain four sets of enhancement features, including 8×8 scale spatial domain enhancement features. 8×8 scale wavelet domain enhancement features 16×16 scale spatial domain enhancement features 16×16 scale wavelet domain enhancement features ; To eliminate cross-modal semantic conflicts and high-frequency noise in the imaging mechanism of heterogeneous modalities, four sets of fused features are input into the frequency-space fusion enhancement module (FEB) for dual-domain joint calibration. The process is as follows: Step 321, Spatial Enhancement Branch: The four sets of fused features are processed... After dimensionality increase, three parallel expansion coefficients d are fed in. A depthwise separable convolution DWConv is performed on {1, 4, 9}, and the three results are then directly summed to obtain a multi-scale sum. This multi-scale sum is then fed into a lightweight neural network operator (SimpleGate, SG), undergoes channel attention CA, and finally undergoes convolutional dimensionality reduction to obtain the spatially enhanced feature Z. i Utilizing learnable parameters The spatial calibration feature Y is obtained by adding the residuals. i Where i∈[1,4] Step 322: Frequency domain enhancement branch. Using Fast Fourier Transform (FFT), the spatial calibration feature Y is... i Transformed to the frequency domain, decoupled into an amplitude matrix. and phase matrix : ; FreMLP (Frequency Domain Multilayer Perceptron) is used to analyze the amplitude matrix, which is susceptible to noise interference. Perform global amplitude modulation while preserving the phase matrix. Subsequently, the data is mapped back to the spatial domain using Inverse Fast Fourier Transform (IFFT); the amplitude matrix is ​​then processed separately using a frequency domain multilayer perceptron (FreMLP). Perform a nonlinear mapping, then with the phase matrix Reconstruction is performed by using the Inverse Fast Fourier Transform (IFFT) to return to the spatial domain and obtain the frequency domain modulation features. ; Step 323, Gated Fusion Output: Introduce channel-level learnable frequency domain residual scaling parameters. The spatial calibration feature Y is obtained through a frequency domain gating mechanism. i Frequency domain modulation characteristics Dynamically weighted residual fusion is performed to output the final frequency-space enhancement features. .

[0024] Step 4: Based on step 3, the cross-domain feature aggregation mechanism is used to flatten, linearly project, and add the cross-domain fusion features of the two scales to achieve unified feature embedding of multi-scale information. In this embodiment of the invention, step 4 includes cross-domain feature aggregation and multi-scale latent space unification: Step 41: Cross-domain information interaction. The spatial domain enhancement features refined by the frequency-space fusion enhancement module FEB and the corresponding wavelet domain enhancement features are sent to the cross-domain fusion module CDFM to perform multi-dimensional semantic-level deep interaction between spatial topology and multi-resolution frequency domain information. 8×8 scale spatial domain enhancement features and 8×8 scale wavelet domain enhancement features The fusion yields cross-domain fusion features at an 8×8 scale. Enhance the spatial domain features at the 16×16 scale. and wavelet domain enhancement features at the 16×16 scale The fusion yielded cross-domain fusion features at a scale of 16×16. ; Step 42: Latent space mapping. Perform spatial dimension flattening and linear projection operations on the 8×8 and 16×16 dual-scale cross-domain fusion features respectively, mapping them into a latent space with a unified feature dimension to obtain multi-scale feature embedding vectors. and ; Step 43, Topological Reshaping and Aggregation: In the unified feature latent space, the two sets of multi-scale embedded features are structurally aggregated through element-wise addition, as expressed in the following expression: ; In order to leverage the topological prior of spatial neighborhood in the subsequent backend refining stage, the one-dimensional aggregation sequence Reshape and reconstruct into a three-dimensional feature tensor .

[0025] Step 5: Based on step 4, construct a directional cross-subband attention DCSA module. Through directional subband gating and cross-subband interaction mechanism, adaptively enhance the structural features in the horizontal, vertical and diagonal directions to overcome the limitations of isotropic processing.

[0026] In this embodiment of the invention, step 5 includes backend refinement of the directional cross-subband attention DCSA module: Before entering the final classifier, the three-dimensional feature tensor Input Directional Cross Subband Attention (DCSA) module to enhance the anisotropic directional geometric priors of linear roads and rectangular building edges; Step 51, Anisotropic Wavelet Decomposition: Using two-dimensional wavelet transform to decompose the three-dimensional feature tensor It is further decomposed into a low-frequency substrate LL and three high-frequency sub-bands in three directions, namely the horizontal direction HL, the vertical direction LH, and the diagonal direction HH. Step 52: Adaptive soft thresholding denoising. Addressing the vulnerability of high-frequency subbands to modal heterogeneous noise, this step involves denoising each high-frequency subband... Introducing a learnable threshold parameter and average amplitude response The adaptive soft thresholding operation is used for shrinkage and denoising, and its expression is: ; Step 53: Cross-subband interaction and directional gating. Calculate the average amplitude response of each denoised high-frequency subband along the channel dimension to generate a channel descriptor reflecting the energy in the current direction. Utilize cross-subband Softmax normalization to generate cross-subband attention weights. Simultaneously, assign independent learnable directional gating parameters to subbands in each direction, adaptively emphasizing or suppressing responses in specific directions. These are then weighted and fused into a highly comprehensive directional hybrid high-frequency representation. ; Step 54: Subband Dynamic Reassignment and Reconstruction. A learnable subband allocation matrix is ​​constructed, and redistribution weights for the three directions are generated through channel-level mapping. The mixed high-frequency directional information is then inversely mapped back to the corresponding wavelet subband space to generate new anisotropic subband features. , and Finally, the newly generated anisotropic high-frequency subband is combined with the low-frequency basis LL, and the reconstructed features with anisotropic directional contour characteristics are obtained through inverse discrete wavelet transform (IDWT). ; Step 55: Dual-weighted residual fusion, utilizing channel allocation weights generated by global average pooling and enhanced weights generated by channel attention mechanism, outputs fully refined final decision features through learnable residual coefficients; 3D feature tensor Weights are generated using Global Average Pooling (GAP) and Softmax activation functions. Reconstructing features Generate through channel attention The weighted sum of the two outputs the final decision feature. .

[0027] In this embodiment of the invention, step 5 further includes lightweight classification head decision and end-to-end optimization: Step 56: Probability mapping, transferring the final decision features A lightweight classification head consisting of convolutional layers, batch normalization layers, ReLU activation layers, adaptive average pooling layers, and fully connected layers is fed in, mapping the target pixel to the predicted probability distribution of each land cover category. ; Step 57: End-to-end training. During the model training phase, the real ground object labels y and the predicted probability distribution are used. The standard cross-entropy loss, used as a global gradient monitoring signal, is expressed as follows: ; The entire network implements end-to-end gradient backpropagation and network parameter updates based on the Adam optimizer.

[0028] In this embodiment of the invention, as shown in Table 1, the model performance verification and experimental results analysis are as follows: The performance of the method of this invention was verified and demonstrated through rigorous experiments. All experiments were conducted in the same hardware and software environment and were tested on the Houston2013 dataset.

[0029] 1. Evaluation metrics are used to comprehensively assess model performance. Four evaluation metrics widely recognized in the field of remote sensing image classification are adopted: Overall accuracy (OA): the proportion of all correctly classified pixels out of the total number of pixels; Average accuracy (AA): the arithmetic mean of the classification accuracy of all categories, which can effectively reflect the model's ability to identify a few categories; Kappa coefficient: used to measure the consistency between the classification results and the actual situation, and can better eliminate the influence of random classification than OA; Category accuracy: the individual classification accuracy of each land cover category.

[0030] 2. Performance Comparison Experiment: To comprehensively and objectively evaluate the classification performance of the proposed method, a detailed comparison was conducted with eight of the most representative multimodal remote sensing image joint classification methods. The baseline for comparison consisted of advanced networks based on Transformer or multi-scale feature interaction that have demonstrated excellent performance in the field of multimodal fusion in recent years, specifically including HCT, FDNet, CALC, GLT, M2FNet, MICF_Net, DSHFNet, and MCFNet. This fully demonstrates their superior capabilities in handling heterogeneous data fusion and complex spatial structure analysis.

[0031] Table 1. Classification performance of different classification methods on the Houston 2013 dataset. .

[0032] Compared with the prior art, the present invention has the following advantages: Frequency-space co-calibration: By introducing the frequency-space fusion enhancement module (FEB), the spatial context modeling of multi-scale dilated convolution and frequency domain amplitude modulation based on FFT are cleverly combined, which effectively filters redundant frequency domain response and high-frequency artifacts in the heterogeneous mode fusion process, while enhancing the local structure representation.

[0033] Spatial coordinate perception capability is greatly improved: The dual coordinate attention gate (DCAG) enhances the wavelet feature perception capability of long-range spatial coordinates through parallel pooling mechanism and adaptive gating, enabling the model to accurately locate target areas with structural discriminative characteristics in complex urban layouts.

[0034] Breaking through the bottleneck of directional structure representation: The Directional Cross Subband Attention (DCSA) module overcomes the shortcomings of traditional wavelet fusion methods in isotropic processing of high-frequency subbands. Through subband redistribution and directional gating mechanisms, it adaptively enhances anisotropic details in horizontal, vertical, and diagonal directions, significantly improving the ability to identify directional prior urban features such as linear roads and building edges.

[0035] The technical solution provided by this invention includes a method that constructs a multi-scale input setting and dual-domain feature extraction architecture, capturing spatial local structure and multi-resolution frequency details through parallel spatial convolutional neural network (CNN) branches and wavelet branches; designs a dual-coordinate attention gate (DCAG) module, using parallel paths of mean pooling and max pooling to locate discriminative regions on the structure; designs a frequency-space fusion enhancement module (FEB), combining multi-scale dilated convolution and amplitude modulation based on Fast Fourier Transform (FFT) to jointly calibrate the spatial context and frequency domain response of the preliminary fused features; uses a cross-domain feature aggregation mechanism to flatten, linearly project, and element-wise add the dual-scale cross-domain fused features to achieve unified feature embedding of multi-scale information; and constructs a directional cross-subband attention (DCSA) module, which adaptively enhances structural features in horizontal, vertical, and diagonal directions through directional subband gating and cross-subband interaction mechanisms. This method effectively alleviates cross-modal semantic differences and enhances the ability to distinguish ground features in complex scenes.

[0036] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for classifying hyperspectral and lidar data based on direction-aware wavelets, characterized in that, The method includes: Step 1: Construct a multi-scale input setting and dual-domain feature extraction architecture, and capture spatial local structure and multi-resolution frequency details through parallel spatial convolutional neural network (CNN) branches and wavelet branches; Step 2: Based on Step 1, design a dual-coordinate attention gate DCAG module, using parallel paths of mean pooling and max pooling to locate the discriminative region on the structure; Step 3: Based on Step 2, design the frequency-space fusion enhancement module FEB, which combines multi-scale dilated convolution and amplitude modulation based on Fast Fourier Transform (FFT) to jointly calibrate the spatial context and frequency domain response of the preliminary fusion features. Step 4: Based on step 3, the cross-domain feature aggregation mechanism is used to flatten, linearly project, and add the cross-domain fusion features of the two scales to achieve unified feature embedding of multi-scale information. Step 5: Based on step 4, construct a directional cross-subband attention DCSA module, which adaptively enhances the structural features in the horizontal, vertical and diagonal directions through directional subband gating and cross-subband interaction mechanisms.

2. The method according to claim 1, characterized in that, Step 1 includes: Step 11, Data Acquisition and Alignment: Acquire HSI data cubes from the same geographic region that have undergone spatial geometric registration. and LiDAR data ; Step 12, Multi-scale Local Block Construction: For each target center pixel in the image, from... and Two sets of bimodal local image patches with different spatial scales are cropped from the image, and the scale set is set as S = {8}. 8, 16 16}; Step 13, Dual-Domain Parallel Branch Construction: At each scale and modality, construct the spatial convolutional neural network (CNN) branch and wavelet branch in parallel to achieve a dual-domain joint representation of spatial and frequency characteristics. The spatial convolutional neural network (CNN) branch extracts basic spatial local structure and neighborhood correlation information from input image patches through concatenated convolutional layers, batch normalization layers, and nonlinear activation layers, obtaining four sets of preliminary spatial features. The wavelet domain branch uses the two-dimensional discrete wavelet transform (DWT) to decompose the input two-dimensional spatial features X into four mutually orthogonal sub-band components, the expression of which is: ； Wherein, LL is the low-frequency approximation component; LH, HL and HH are the high-frequency detail components in the vertical, horizontal and diagonal directions, respectively; then, each component is encoded by an independent convolution Conv and input into the frequency domain feature alignment FDFA module for preliminary cross-frequency band interactive alignment.

3. The method according to claim 2, characterized in that, Step 2 includes: Step 21: Bidirectional pooling feature extraction. The four wavelet domain features, after initial alignment, are extracted along the coordinate attention paths of the mean and the maximum value. Pooling is performed in both the horizontal and vertical directions to obtain horizontal and vertical pooling vectors; mean pooling is used to capture smooth global background features, and max pooling is used to capture high-frequency local salient edges. Step 22: Coordinate attention generation. Within the coordinate attention paths of the mean and the maximum, the horizontal and vertical pooling vectors are concatenated and sequentially passed through a shared-parameter convolution Conv1.

1. Batch Normalization (BN) and ReLU activation are applied; the interacted features are then split again and fed into independent convolutional layers (Conv) and activation functions (Sigmoid) to generate mean-coordinated attention maps. Attention map with maximum coordinates ; Step 23: Adaptive gated path fusion, introducing learnable lightweight gated branches; the lightweight gated branches are applied to wavelet domain features. Global average pooling (GAP) is performed, and mean path weights are adaptively generated using a multilayer perceptron. Path weight with maximum value Attention features of fused spatial coordinates Its expression is: ； Step 24, Channel Calibration and Feature Output: The fused spatial coordinate attention features are then processed. With wavelet domain features Element-by-element multiplication yields preliminary weighted features Subsequently, channel attention based on an activation mechanism is used to calibrate the initial weighted features, and residual connections are made with the original input to output the final refined wavelet domain features. It includes four refined wavelet domain features. , and .

4. The method according to claim 3, characterized in that, Step 3 includes mid-layer fusion refinement and frequency-space collaborative denoising: Step 31: Preliminary multimodal fusion, combining the four sets of preliminary spatial features with the final refined wavelet domain features. The input is processed by the multimodal fusion module (MFM), which uses hyperspectral features as a spatial guide to perform preliminary cross-interactions on the bimodal features, resulting in four sets of fused features, including 8×8 scale spatial domain fusion features. 8×8 scale wavelet domain fusion features Spatial domain fusion features at the 16×16 scale 16×16 scale wavelet domain fusion features ; Step 32: The frequency-space fusion enhancement module (FEB) inputs the four sets of fusion features into the frequency-space fusion enhancement module (FEB) to obtain four sets of enhancement features, including 8×8 scale spatial domain enhancement features. 8×8 scale wavelet domain enhancement features 16×16 scale spatial domain enhancement features 16×16 scale wavelet domain enhancement features ; To eliminate cross-modal semantic conflicts and high-frequency noise in the imaging mechanism of heterogeneous modalities, four sets of fused features are input into the frequency-space fusion enhancement module (FEB) for dual-domain joint calibration. The process is as follows: Step 321, Spatial Enhancement Branch: The four sets of fused features are processed... After dimensionality increase, three parallel expansion coefficients d are fed in. The depthwise separable convolution DWConv of {1, 4, 9} is performed, and then the three results are directly added together to obtain the multi-scale sum. The multi-scale sum is fed into the lightweight neural network operator SG, passes through channel attention CA, and finally undergoes convolution dimensionality reduction to obtain the spatially enhanced feature Z. i Utilizing learnable parameters The spatial calibration feature Y is obtained by adding the residuals. i Where i∈[1,4] Step 322: Frequency domain enhancement branch. Using Fast Fourier Transform (FFT), the spatial calibration feature Y is... i Transformed to the frequency domain, decoupled into an amplitude matrix. and phase matrix : ； FreMLP (Frequency Domain Multilayer Perceptron) is used to analyze the amplitude matrix. Perform global amplitude modulation while preserving the phase matrix. Subsequently, the data is mapped back to the spatial domain using Inverse Fast Fourier Transform (IFFT); the amplitude matrix is ​​then processed separately using a frequency domain multilayer perceptron (FreMLP). Perform a nonlinear mapping, then with the phase matrix Reconstruction is performed by using the Inverse Fast Fourier Transform (IFFT) to return to the spatial domain and obtain the frequency domain modulation features. ; Step 323, Gated Fusion Output: Introduce channel-level learnable frequency domain residual scaling parameters. The spatial calibration feature Y is obtained through a frequency domain gating mechanism. i Frequency domain modulation characteristics Dynamically weighted residual fusion is performed to output the final frequency-space enhancement features. .

5. The method according to claim 4, characterized in that, Step 4 includes cross-domain feature aggregation and multi-scale latent space unification: Step 41: Cross-domain information interaction. The spatial domain enhancement features refined by the frequency-space fusion enhancement module FEB and the corresponding wavelet domain enhancement features are sent to the cross-domain fusion module CDFM to perform multi-dimensional semantic-level deep interaction between spatial topology and multi-resolution frequency domain information. 8×8 scale spatial domain enhancement features and 8×8 scale wavelet domain enhancement features The fusion yields cross-domain fusion features at an 8×8 scale. Enhance the spatial domain features at the 16×16 scale. and wavelet domain enhancement features at the 16×16 scale The fusion yielded cross-domain fusion features at a scale of 16×16. ; Step 42: Latent space mapping. Perform spatial dimension flattening and linear projection operations on the 8×8 and 16×16 dual-scale cross-domain fusion features respectively, mapping them into a latent space with a unified feature dimension to obtain multi-scale feature embedding vectors. and ; Step 43, Topological Reshaping and Aggregation: In the unified feature latent space, the two sets of multi-scale embedded features are structurally aggregated through element-wise addition, as expressed in the following expression: ； In order to leverage the topological prior of spatial neighborhood in the subsequent backend refining stage, the one-dimensional aggregation sequence Reshape and reconstruct into a three-dimensional feature tensor .

6. The method according to claim 5, characterized in that, Step 5 includes backend refinement of the directional cross-subband attention DCSA module: Before entering the final classifier, the three-dimensional feature tensor Input Directional Cross Subband Attention (DCSA) module to enhance the anisotropic directional geometric priors of linear roads and rectangular building edges; Step 51, Anisotropic Wavelet Decomposition: Using two-dimensional wavelet transform to decompose the three-dimensional feature tensor It is further decomposed into a low-frequency substrate LL and three high-frequency sub-bands in three directions, namely the horizontal direction HL, the vertical direction LH, and the diagonal direction HH. Step 52: Adaptive soft thresholding denoising for each high-frequency sub-band. Introducing learnable threshold parameters and average amplitude response The adaptive soft thresholding operation is used for shrinkage and denoising, and its expression is: ； Step 53: Cross-subband interaction and directional gating. Calculate the average amplitude response of each denoised high-frequency subband along the channel dimension to generate a channel descriptor reflecting the energy in the current direction. Utilize cross-subband Softmax normalization to generate cross-subband attention weights. Simultaneously, assign independent learnable directional gating parameters to subbands in each direction, adaptively emphasizing or suppressing responses in specific directions. These are then weighted and fused into a highly comprehensive directional hybrid high-frequency representation. ; Step 54: Subband Dynamic Reassignment and Reconstruction. A learnable subband allocation matrix is ​​constructed, and redistribution weights for the three directions are generated through channel-level mapping. The mixed high-frequency directional information is then inversely mapped back to the corresponding wavelet subband space to generate new anisotropic subband features. , and Finally, the newly generated anisotropic high-frequency subband is combined with the low-frequency basis LL, and the reconstructed features with anisotropic directional contour characteristics are obtained through inverse discrete wavelet transform (IDWT). ; Step 55: Dual-weight residual fusion, using channel allocation weights generated by global average pooling and enhanced weights generated by channel attention mechanism, outputs fully refined final decision features through learnable residual coefficients; 3D feature tensor Weights are generated using Global Average Pooling (GAP) and Softmax activation functions. Reconstructing features Generate through channel attention The weighted sum of the two outputs the final decision feature. .

7. The method according to claim 6, characterized in that, Step 5 also includes lightweight classification head decision and end-to-end optimization: Step 56: Probability mapping, transferring the final decision features A lightweight classification head consisting of convolutional layers, batch normalization layers, ReLU activation layers, adaptive average pooling layers, and fully connected layers is fed in, mapping the target pixel to the predicted probability distribution of each land cover category. ; Step 57: End-to-end training. During the model training phase, the real ground object labels y and the predicted probability distribution are used. The standard cross-entropy loss, used as a global gradient monitoring signal, is expressed as follows: ； The entire network implements end-to-end gradient backpropagation and network parameter updates based on the Adam optimizer.

Images