Ground feature classification method based on frequency-space collaborative learning hierarchical fusion network

Abstract

The invention discloses a ground feature classification method based on a frequency-space collaborative learning hierarchical fusion network. The method comprises the following steps: step 1, data preparation; step 2, the HMFE module extracts frequency domain features; step 3, extracting spatial domain features by an MLSA module; 4, the AWF module dynamically integrates the features; 5, executing a classification task; the MLSA module strengthens cross-scale interaction through a tree fusion structure and a cross attention mechanism, realizes efficient fusion of HSI and LiDAR data in a spatial domain, improves classification consistency of complex scenes, and reduces boundary blur phenomena; the HMFE module introduces a learnable frequency coding mechanism into the Mmba module so as to enhance spectrum-frequency components with strong discrimination; the AWF module realizes dynamic integration of spatial domain and frequency domain features through adaptive weighted fusion, fully mines deep complementarity of the spatial domain and the frequency domain, and improves the utilization efficiency of the model for heterogeneous data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ground cover classification, in particular to a ground object classification method based on a frequency-space collaborative learning hierarchical fusion network. BACKGROUND

[0002] With the continuous development of remote sensing and deep learning technology, land cover classification (LCC) has become a research hotspot in this field. LCC plays an important role in key applications such as mineral exploration, scene understanding, target detection and change monitoring. Because spectral information can be used as a unique identifier of ground object categories, similar to human fingerprints, it can achieve accurate material identification. Hyperspectral image (HSI) has hundreds of continuous spectral bands due to its unique imaging mechanism, providing great potential for LCC tasks. However, early methods relying on HSI data are subject to inherent limitations, such as susceptibility to atmospheric interference and spectral similarity between different ground objects. Therefore, it is difficult to achieve robust ground object classification relying solely on HSI data. To alleviate the challenges brought by the inherent limitations of HSI, LiDAR data, as a complementary information to HSI, is widely used to improve the performance of LCC. LiDAR can generate a digital surface model, providing height information of the target, which helps classification and reduces spectral confusion. Fusing HSI and LiDAR data in spatially registered regions can take advantage of their complementary discriminative features, thereby effectively improving classification accuracy.

[0003] However, the existing ground object classification method based on fusion of HSI and LiDAR data still has the following shortcomings: 1. Although the dynamic scale mechanism is introduced in the existing spatial domain heterogeneous data fusion method to improve the flexibility of scale selection, the interaction between cross-scale features is only realized by simple splicing or superposition, and the effective association between shallow details and deep semantics cannot be established, resulting in difficulty in capturing both local fine information and global context, and the classification result is prone to boundary blur or semantic inconsistency; 2. In terms of frequency domain feature enhancement, the existing method relies on traditional fixed frequency decomposition techniques, such as pre-set filters, which have problems such as decoupling of spectral and spatial features, lack of flexibility, and difficulty in adapting to the frequency characteristics of diverse topographic features. In addition, relying only on single-scale frequency information ignores the cross-layer interaction, resulting in the inability to strengthen discriminative frequency bands, limiting the utilization efficiency of spectral information; 3. The existing technology for fusion of frequency domain and spatial domain features mostly stays at the stage of shallow feature splicing or weighting, and does not deeply explore the deep complementarity of the two, resulting in insufficient extraction and integration of deep features, and the collaborative advantages of heterogeneous data cannot be fully utilized, making it difficult to break through the bottleneck of classification accuracy. SUMMARY

[0004] The present application aims to provide a ground object classification method based on a frequency-space collaborative learning hierarchical fusion network to solve the problems raised in the background technology.

[0005] To achieve the above object, the present application provides the following technical scheme: a ground object classification method based on a frequency-space collaborative learning hierarchical fusion network, comprising the following steps: step one, data preparation; step two, extracting frequency domain features by an HMFE module; step three, extracting spatial domain features by an MLS²A module; step four, dynamically integrating features by an AWF module; and step five, performing a classification task.

[0006] In the above step one, the HSI and LiDAR data are preprocessed, and the preprocessed data are used as model inputs.

[0007] In the above step two, the frequency domain features are extracted from the preprocessed HSI data by the HMFE module.

[0008] In the above step three, the heterogeneous features of the HSI and LiDAR data are fused in the spatial domain by the MLS²A module, and then the spatial domain features are obtained through multi-scale down-sampling and tree fusion structure.

[0009] In the above step four, the frequency domain features obtained in step two and the spatial domain features obtained in step three are dynamically integrated by the AWF module to obtain the final joint features.

[0010] In the above step five, the joint features are mapped to the ground object categories by a classifier, and the model is optimized by focal loss.

[0011] Preferably, in the above step one, the following steps are included:

[0012] 1.1 Data definition: let the HIS data be , and the LiDAR data be , wherein represents the number of bands of the hyperspectral image, , represents the spatial size of the image.

[0013] 1.2 Patch extraction: a center pixel patch extraction strategy is adopted to generate HIS patches and LiDAR patches , wherein is the patch size.

[0014] 1.3 Spectral normalization: the HSI patches are projected to a 64-dimensional feature space by a 1×1 two-dimensional convolution layer to obtain , and as model inputs.

[0015] Preferably, in the above step two, the following steps are included:

[0016] 2.1 Obtain multi-scale frequency information: process the input After the application layer normalization, the multi-scale frequency information is captured through the high-frequency branch, the medium-frequency branch and the low-frequency branch processing;

[0017] 2.2 Adaptive fusion: process the outputs of the high-frequency branch, the medium-frequency branch and the low-frequency branch 、 、 Adaptive weighted fusion is performed to obtain , and finally a 3x3 convolution layer is refined to generate the final frequency domain feature ; the adaptive weighted fusion can be expressed by the following formula:

[0018]

[0019] wherein, is the output of the high-frequency branch, is the weight thereof, is the output of the medium-frequency branch, is the weight thereof, is the output of the low-frequency branch, is the weight thereof, is a learnable weight, and the initial value is all set to 0.5, which can be optimized and adjusted in training.

[0020] Preferably, in the step 2.1, the following steps are specifically included:

[0021] 2.1.1 High-frequency branch processing: focusing on edge and texture information, process the input Apply a 5x5 depth separable convolution to generate a coarse-grained spatial feature , which is converted to a frequency domain feature by a fast Fourier transform, and the converted feature map is denoted as , which is then filtered by a learnable frequency domain filter to retain high-frequency components to obtain , and , and then process the high-frequency spatial domain feature map by a Mamba module to obtain enhanced features , and finally combine the residual connection and point-by-point convolution to output the spatial domain feature map under the guidance of high-frequency ; wherein the learnable frequency domain filter is composed of a group of trainable Gaussian kernel functions , and represent the learnable center frequency and bandwidth, respectively;

[0022] 2.1.2 Medium-frequency branch processing: focusing on structural information, process the input Apply a 3x3 depth separable convolution to generate a medium-scale spatial feature , combine the high-frequency texture feature obtained by the high-frequency branch and Fusion of mid-frequency guided spatial domain feature maps , After the mid-frequency components are reserved by the learnable frequency domain filter, the mid-frequency guided spatial domain feature maps are obtained by inverse fast Fourier transform , and the coarse-grained spatial features are processed and fused by the Mamba module , to obtain mid-frequency enhanced features , and finally the mid-frequency enhanced spatial domain feature maps are obtained by point-wise convolution ;

[0023] 2.1.3 Low-frequency branch processing: focusing on global semantics, generating low-frequency spatial features by 1x1 depth separable convolution , and fusing the mid-frequency enhanced features After the low-frequency components are reserved by the learnable frequency domain filter, the Mamba module is processed to generate , and the mid-frequency spatial features are cross-layer fused, and the low-frequency guided spatial feature maps are obtained by point-wise convolution .

[0024] Preferably, the formula of the fast Fourier transform is as follows:

[0025]

[0026] wherein, is the converted frequency domain feature, denotes the imaginary unit, denotes the inner product operation, is the input spatial domain feature map, and the final converted feature map is denoted as ;

[0027] The formula of the inverse fast Fourier transform is as follows:

[0028] ;

[0029] The process of processing the high-frequency spatial domain feature map by the Mamba module is as follows: the Mamba module includes two parallel branches, the first branch starts with linear projection, and then extracts spatial patterns by depth separable convolution, the core module of the first branch is module, which is used to model the long-distance dependence relationship in the spectral dimension, and the second branch first performs linear transformation, and then introduces non-linear characteristics and gating mechanism by linear unit activation function with Sigmoid weight, the outputs of the two branches are fused by element-wise multiplication, and the fusion result is then restored to the original feature dimension by inverse linear projection, the whole process can be represented by the following formula:

[0030]

[0031]

[0032] in, Represents high-frequency texture features, For modeling long-range dependencies Module, the symbol ⊙ represents element-wise product. This represents depthwise separable convolution. Indicates linear projection. Indicates inverse linear projection. This represents the linear unit activation function, and the final output is... It integrates high-frequency spatial information with global contextual semantics;

[0033] Combining residual connections and pointwise convolution to output high-frequency guided spatial domain feature maps The process specifically involves: converting coarse-scale spatial features With enhanced features The information is added together and then fused through pointwise convolution to finally output the result. The calculation formula is as follows:

[0034] in, This represents pointwise convolution.

[0035] Preferably, the high-frequency branching is used to obtain high-frequency texture features. and The fusion yields a spatial domain feature map guided by mid-frequency. The process can be represented by the following formula:

[0036]

[0037] The process of preserving the intermediate frequency component using a learnable frequency domain filter can be represented by the following formula:

[0038]

[0039] in, Represents the Fast Fourier Transform. This indicates a learnable frequency domain filter. Indicates the inverse fast Fourier transform;

[0040] Coarse-grained spatial features are processed and fused using the Mamba module. The process can be represented by the following formula:

[0041]

[0042] The spatial domain feature map with enhanced mid-frequency is obtained through pointwise convolution. The process can be expressed by the following formula:

[0043]

[0044] wherein, represents point-wise convolution.

[0045] Preferably, in step three, it specifically comprises the following steps:

[0046] 3.1 Initial fusion: splice and to obtain the initial fusion feature map ;

[0047] 3.2 Multi-scale down-sampling: apply two consecutive down-sampling stages to and to generate multi-scale features and ; wherein each down-sampling stage contains a 3x3 depth separable convolution, followed by a BN and a ReLU activation function, followed by a 1x1 point-wise convolution, followed by a BN and a ReLU activation function;

[0048] 3.3 Tree fusion: fuse multi-resolution features using a recursive structure, and apply spatial attention and channel attention to large-scale input and small-scale input in parallel, and generate small-scale intermediate feature and large-scale intermediate feature , through bilinear interpolation up-sampling to the same spatial dimension as , to obtain , and through cross-attention fusion, to obtain , add the residual connection to , to obtain , and finally obtain the spatial domain feature through 1x1 convolution.

[0049] The calculation formula is as follows:

[0050]

[0051] wherein, represents the dimension of the key vector, the query and the key are extracted by , and the value comes from .

[0052] Preferably, in step four, specifically: let the feature maps of the frequency domain and spatial domain branches be respectively... and The fusion process is then defined as:

[0053]

[0054] in, Indicates channel-dimensional splicing. For linear projection layers, It is the Sigmoid activation function. This indicates element-wise multiplication. These are learnable parameters. .

[0055] Preferably, in step five, the classifier consists of three fully connected layers. The first two layers contain linear transformation, ReLU activation function and Dropout operation, and the third layer outputs class logits.

[0056] Preferably, in step five, a modulating factor is introduced to adjust the focus loss. and category weights Together, they reduce the relative loss contribution to correctly classified samples and highlight the importance of hard-to-classify samples. The definition of focus loss is as follows:

[0057]

[0058] in, For batch size, For the number of categories, Indicates sample Category The predicted probability, For this category Real labels, parameters Control the focus intensity, The balancing weights are adjusted based on category frequency.

[0059] Compared with the prior art, the MLS²A module of the application strengthens cross-scale interaction through a tree-shaped fusion structure and a cross-attention mechanism, realizes efficient fusion of HSI and LiDAR data in the spatial domain, improves the classification consistency of complex scenes, and reduces the boundary blur phenomenon; the HMFE module introduces a learnable frequency coding mechanism in the Mamba module to enhance the spectral-frequency components with strong discriminability, and improves the feature discriminability through multi-scale frequency interaction, so that the model can stably extract effective information in different landform scenes; and the AWF module realizes dynamic integration of spatial domain and frequency domain features through adaptive weighted fusion, fully excavates the deep complementarity of the two, and improves the utilization efficiency of the model for heterogeneous data. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 Figure 1 is a schematic diagram of the HFS-CL network architecture of the application;

[0061] Figure 2 Figure 2 is a schematic diagram of the tree-shaped fusion strategy of the application;

[0062] Figure 3 Figure 3 is a result diagram for effectiveness verification of the AWF module;

[0063] Figure 4 Figure 4 is a full classification diagram of the Houston2013 dataset;

[0064] Figure 5 Figure 5 is a full classification diagram of the MUUFL dataset;

[0065] Figure 6 Figure 6 is a full classification diagram of the Trento dataset;

[0066] Figure 7 Figure 7 is a flowchart of the method of the application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0068] Please refer to the accompanying Figure 1 - the accompanying Figure 7 An embodiment provided by the application is a feature classification method based on a frequency-space collaborative learning hierarchical fusion network, which comprises the following steps: step one, data preparation; step two, the HMFE module extracts frequency domain features; step three, the MLS²A module extracts spatial domain features; step four, the AWF module dynamically integrates features; and step five, a classification task is performed.

[0069] wherein in the above step one, the HSI and LiDAR data are pre-processed, and the pre-processed data are taken as the model input, specifically comprising the following steps:

[0070] 1.1 Data definition: let the HIS data be and the LiDAR data be wherein represents the number of bands of the hyperspectral image, , represents the spatial size of the image;

[0071] 1.2 Patch extraction: a center pixel patch extraction strategy is adopted to generate the HIS patch and the LiDAR patch wherein is the patch size;

[0072] 1.3 Spectral normalization: the HSI patch is projected to a 64-dimensional feature space through a 1x1 two-dimensional convolution layer to obtain , and as the model input;

[0073] wherein in the above step two, the frequency domain features are extracted from the pre-processed HSI data through the HMFE module, specifically comprising the following steps:

[0074] 2.1 Obtain multi-scale frequency information: after the input is applied with layer normalization, it is processed through a high-frequency branch, a medium-frequency branch and a low-frequency branch to capture multi-scale frequency information;

[0075] 2.1.1 High-frequency branch processing: focusing on edge and texture information, the input is applied with a 5x5 depth separable convolution to generate coarse-grained spatial features , which are converted to frequency domain features through a fast Fourier transform, and the converted feature map is denoted as , which is then filtered to retain high-frequency components through a learnable frequency domain filter to obtain , and is applied with a Mamba module to process the high-frequency spatial domain feature map to obtain enhanced features , and finally the spatial domain feature map under the guidance of high frequency is output through residual connection and point-wise convolution ; wherein the learnable frequency domain filter is composed of a group of trainable Gaussian kernel functions , and represent the learnable center frequency and bandwidth, respectively; the formula of the fast Fourier transform is as follows:

[0076]

[0077] where, is the converted frequency domain feature, denotes the imaginary unit, denotes the inner product operation, is the input spatial domain feature map, and the final converted feature map is denoted as ;

[0078] The inverse fast Fourier transform is shown as follows:

[0079] ;

[0080] The process of processing the high-frequency spatial domain feature map by the Mamba module is as follows: the Mamba module includes two parallel branches, the first branch starts with a linear projection, and then extracts spatial patterns through a depth separable convolution, the core module of the first branch is the module, which is used to model the long-distance dependence relationship in the spectral dimension, and the second branch first performs a linear transformation, and then introduces a nonlinear property and a gating mechanism through a linear unit activation function with a Sigmoid weight, the outputs of the two branches are fused through element-wise multiplication, and the fusion result is then restored to the original feature dimension through an inverse linear projection, the whole process can be represented by the following formula:

[0081]

[0082]

[0083] where, denotes the high-frequency texture feature, is the module for modeling long-distance dependence, the symbol denotes a depth separable convolution, denotes a linear projection, denotes an inverse linear projection, denotes a linear unit activation function, and the final output integrates high-frequency spatial information and global context semantics;

[0084] The process of outputting the spatial domain feature map under the guidance of high-frequency by combining residual connection and point-wise convolution is as follows: the coarse-scale spatial feature is added to the enhanced feature , and the channel information is fused through point-wise convolution, and finally is output, and the calculation formula is as follows:

[0085] wherein, denotes point-wise convolution;

[0086] 2.1.2 Mid-frequency branch processing: focusing on structural information, input Applying 3x3 depth separable convolution to generate mid-scale spatial features , high-frequency texture features are obtained by the high-frequency branch and fusion to obtain mid-frequency guided spatial domain feature map , After retaining the mid-frequency components by the learnable frequency domain filter, the mid-frequency guided spatial domain feature map is obtained by inverse fast Fourier transform , and the coarse-grained spatial features are processed and fused by the Mamba module to obtain mid-frequency enhanced features , and finally the mid-frequency enhanced spatial domain feature map is obtained by point-wise convolution ; high-frequency texture features are obtained by the high-frequency branch and fusion to obtain mid-frequency guided spatial domain feature map The process can be represented by the following formula:

[0087]

[0088] The process of retaining the mid-frequency components by the learnable frequency domain filter can be represented by the following formula:

[0089]

[0090] wherein, denotes fast Fourier transform, denotes a learnable frequency domain filter, denotes inverse fast Fourier transform;

[0091] The process of processing and fusing coarse-grained spatial features by the Mamba module can be represented by the following formula:

[0092]

[0093] The process of obtaining mid-frequency enhanced spatial domain feature map by point-wise convolution can be represented by the following formula:

[0094]

[0095] wherein, denotes point-wise convolution;

[0096] 2.1.3 Low-frequency branch processing: focusing on global semantics, generating low-frequency spatial features through 1x1 depth separable convolution , fusing mid-frequency enhanced features , and then retaining low-frequency components through a learnable frequency domain filter, processing through a Mamba module to generate , and then fusing with mid-frequency spatial features across layers, and obtaining low-frequency guided spatial feature maps through point-wise convolution ;

[0097] 2.2 Adaptive fusion: adaptively weighting and fusing the outputs of the high-frequency branch, the mid-frequency branch, and the low-frequency branch , , to obtain , and finally refining through a 3x3 convolution layer to generate the final frequency domain feature ; the adaptive weighting and fusion can be represented by the following formula:

[0098]

[0099] wherein is the output of the high-frequency branch, is the weight thereof, is the output of the mid-frequency branch, is the weight thereof, is the output of the low-frequency branch, is the weight thereof, is a learnable weight, and the initial value is set to 0.5, which can be optimized and adjusted in training;

[0100] wherein in the above step three, the heterogeneous features of HSI and LiDAR data are fused in the spatial domain through the MLS²A module, and then multi-scale down-sampling and tree fusion structure are used to obtain spatial domain features, including the following steps:

[0101] 3.1 Initial fusion: splicing and to obtain initial fusion feature map ;

[0102] 3.2 Multi-scale down-sampling: applying two consecutive down-sampling stages to and to generate multi-scale features and ; wherein each down-sampling stage contains a 3x3 depth separable convolution, followed by a BN and a ReLU activation function, followed by a 1x1 point-wise convolution, followed by a BN and a ReLU activation function;

[0103] 3.3 Tree fusion: recursively fusing multi-resolution features, respectively applying spatial attention and channel attention on large-scale input and small-scale input , the resulting outputs are respectively added element-wise to generate small-scale intermediate feature and large-scale intermediate feature , , , , and , , , , , ;

[0104] The calculation formula is as follows:

[0105]

[0106] wherein, denotes the dimension of the key vector, the query item and the key item are extracted by , and the value comes from ;

[0107] wherein in the above step four, the frequency domain feature obtained in step two and the spatial domain feature obtained in step three are dynamically integrated by the AWF module to obtain the final joint feature, specifically: let the feature maps of the frequency domain and the spatial domain branch be and , then the fusion process is defined as:

[0108]

[0109] wherein, denotes channel dimension concatenation, is a linear projection layer, is a Sigmoid activation function, denotes element-wise multiplication, is a learnable parameter, ;

[0110] In step five above, the joint features are mapped to land cover categories using a classifier, and the focus loss optimization model is employed. The classifier consists of three fully connected layers: the first two layers contain linear transformations, ReLU activation functions, and Dropout operations; the third layer outputs category logits. A modulating factor is introduced into the focus loss. and category weights Together, they reduce the relative loss contribution to correctly classified samples and highlight the importance of hard-to-classify samples. The definition of focus loss is as follows:

[0111]

[0112] in, For batch size, For the number of categories, Indicates sample Category The predicted probability, For this category Real labels, parameters Control the focus intensity, The balancing weights are adjusted based on category frequency.

[0113] To verify the effectiveness of this invention, experiments 1 through 5 were conducted. All experiments were implemented using the PyTorch framework, running on a server equipped with an AMD Ryzen 7 6800H processor (with Radeon integrated graphics, 3.20GHz), 24GB of RAM, and an NVIDIA GeForce RTX 4060 Laptop GPU (6GB VRAM). HFS-CL used the Adam optimizer, with 300 training epochs and a mini-batch size of 64. In the S6 module, the state space dimension was empirically set to 16 based on performance considerations. To ensure the objectivity of performance evaluation, three standard evaluation metrics were used: Overall Accuracy (OA), Average Accuracy (AA), and Kappa coefficient. In addition, to enhance the reliability of the results, all experimental results were the average of 10 independent experiments. The datasets used are shown in Tables 1 to 3, namely the Houston2013 dataset, the MUUFL dataset, and the Trento dataset.

[0114] Experimental Example 1:

[0115] Ablation experiments of HMFE and MLS²A module: To evaluate the contribution of each of the two feature extraction modules (HMFE and MLS²A) proposed in this application, ablation experiments were conducted on three benchmark datasets, as shown in Table 4, and the results of OA, AA and , ensuring the robustness and persuasiveness of experimental comparison, in order to ensure the fairness of the evaluation, the AWF module is not introduced here, from the experimental results, it can be seen that the classification performance of the model is the lowest when only using the HMFE module, however, after introducing the MLS²A module, the accuracy is significantly improved, especially on the Houston2013 dataset, which brings a 5.70% improvement, which fully illustrates the important role of spatial domain heterogeneous feature fusion in improving the classification effect; based on the use of only the MLS²A module, when the HMFE module is further added, the performance is improved again, which brings a 2.98% gain on the Houston2013 dataset, verifying the ability of the HMFE module in extracting effective frequency domain features, in addition, on the three datasets, the use of the two modules simultaneously achieves the optimal classification result, indicating that the frequency domain features and the spatial domain features have high complementarity, and further verifying the significant advantage of the joint design in the hyperspectral and LiDAR joint classification task.

[0116] Experimental Example 2:

[0117] Verification of the effectiveness of the AWF module: To verify the effectiveness of the AWF module proposed in this application, comparative experiments were conducted between it and two traditional feature fusion strategies: element-by-element addition (ADD) and feature concatenation (CAT), comprehensive evaluation was conducted on three benchmark datasets, and OA was used as the comprehensive performance indicator to ensure the fairness of the comparison, as shown in Figure 3 , the experimental results show that the AWF strategy outperforms the other two methods in both cascaded fusion and element-level fusion scenarios, and this result highlights the superior ability of the AWF module in integrating frequency domain guided spectral features and spatial domain features from heterogeneous data sources.

[0118] Experimental Example 3:

[0119] Verification of the effectiveness of the LFDF: To evaluate the effectiveness of the LFDF module proposed in this application, quantitative comparative experiments were conducted with the traditional TFDF module on three benchmark datasets, in order to ensure the fairness of the experiment, all other components except the filter module were kept at the optimal configuration, as shown in Table 5, the proposed LFDF module brings a performance improvement of 3.32%, 1.97% and 1.04% on the three datasets respectively, these results not only verify the contribution of LFDF to the overall model performance, but also show that it has stronger ability than the fixed filter in extracting discriminative frequency domain features.

[0120] Experimental Example 4:

[0121] To evaluate the individual and joint contribution of different frequency branches, a comprehensive ablation experiment was conducted, and other network structures and hyperparameters remained unchanged. The results are summarized in Table 6, showing that the cross-frequency interaction is always superior to the single branch configuration, highlighting the importance of multi-frequency collaboration. The sensitivity of different data sets to frequency components differs, for example, the combination of high-frequency and medium-frequency branches performs more competitively on the Houston2013 data set, but performs poorly on the MUUFL data set. These findings emphasize the need to coordinate the integration of all frequency components to improve model generalization. The superior performance of the complete model further validates this conclusion.

[0122] Experimental Example 5:

[0123] To evaluate the performance of the proposed HFS-CL, it was compared with various existing classification methods, which can be broadly divided into spatial domain methods and frequency domain methods. The spatial domain methods include six representative algorithms: CoupleCNN and HCT (based on CNN), ExViT and CrossHL (based on Vision Transformer), and HLMamba and S2CM (based on Mamba architecture). The frequency domain methods include two: FDNet and GFSFN. Quantitative analysis: Tables 7.1 to 9.3 show the OA, AA, and class accuracy of the proposed method and all comparison methods on three data sets. The best results are highlighted in bold font, as shown in the table. HFS-CL significantly outperforms other comparison methods in all evaluation indicators, demonstrating excellent classification performance. Specifically, on the Houston2013 data set, HFS-CL improves the OA by 2.49% compared to the best spatial domain method CoupleCNN, and by 6.54% compared to the best frequency domain method FDNet. Compared to HLMamba, HFS-CL performs better in OA, AA, and class accuracy, further verifying its effectiveness. HFS-CL achieves the highest class accuracy in 8 classes, while HLMamba ranks first in only 4 classes. This performance improvement is attributed to the effective cross-frequency interaction mechanism designed in HFS-CL, which enhances the extraction of fine-grained information in high-frequency components and effectively alleviates the class boundary ambiguity problem. On the MUUFL data set, HFS-CL also exhibits robust performance, achieving the best results in classes C07 (building shadow), C08 (building), and C09 (sidewalk), and achieving the best overall OA, AA, The indicators are ranked first, however, for C10 (yellow curb) and C11 (cloth marker) two categories, the classification performance is slightly insufficient, because the adopted sampling strategy leads to limited number of training samples; on the Trento dataset, HFS-CL exceeds all other methods in three categories, and in the C3 (bare ground) category, it reaches 100% classification accuracy, which again verifies the ability of HFS-CL in integrating spatial and frequency domain features, enabling the model to perform complementary feature extraction, at the same time, HFS-CL is always leading in OA, AA and The indicators are ranked first, and its excellent robustness and generalization ability are fully reflected; visual analysis: Figure 4 、 Figure 5 and Figure 6 show the classification maps generated by all methods on the three datasets, Figure 4 (a)-(j) are the results of 10 experiments, Figure 5 (a)-(j) are the results of 10 experiments, Figure 6 (a)-(j) are the results of 10 experiments; Specifically, spatial domain guided methods such as ExViT and CrossHL have obvious salt and pepper noise, which is mainly because although the ViT-based architecture can model long-distance spectral dependence, its ability to capture fine spatial details is weak, similarly, frequency domain guided classification methods also show obvious salt and pepper noise, which is mainly due to their insufficient consideration of the complementary nature of spatial and frequency domains, and the lack of cross-frequency information interaction modeling, in addition, spatial domain guided methods such as CoupleCNN and S2CM show fuzzy at the class boundary, which is mainly due to their limited ability to extract deep semantic information, and insufficient discrimination of fine-grained features, in contrast, the present application combines the complementary advantages of spatial and frequency domains through the hierarchical feature extraction framework, thereby improving the classification accuracy, and the output classification map is more clear in visual effect.

[0124] Table 1 Training samples, validation samples and test samples of Houston2013 dataset

[0125]

[0126] Table 2 Training samples, validation samples and test samples of MUUFL dataset

[0127]

[0128] Table 3 Training samples, validation samples and test samples of Trento dataset

[0129]

[0130] Table 4 Ablation experiments of HMFE with MLS2A module

[0131]

[0132] Table 5 Validation of effectiveness of LFDF

[0133]

[0134] Table 6 Sensitivity analysis of different frequency domain branches (best results are shown in bold)

[0135]

[0136] Table 7.1 Classification accuracy obtained by different methods on Houston2013 dataset (I) (best results are shown in bold)

[0137]

[0138] Table 7.2 Classification accuracy obtained by different methods on Houston2013 dataset (II) (best results are shown in bold)

[0139]

[0140] Table 7.3 Classification accuracy obtained by different methods on Houston2013 dataset (III) (best results are shown in bold)

[0141]

[0142] Table 8.1 Classification accuracy obtained by different methods on MUUFL dataset (I) (best results are shown in bold)

[0143]

[0144] Table 8.2 Classification accuracy obtained by different methods on MUUFL dataset (II) (best results are shown in bold)

[0145]

[0146] Table 8.3 Classification accuracy obtained by different methods on MUUFL dataset (III) (best results are shown in bold)

[0147]

[0148] Table 9.1 Classification accuracy obtained by different methods on Trento dataset (I) (best results are shown in bold)

[0149]

[0150] Table 9.2 Classification accuracy (two) obtained by different methods on the Trento dataset (best results are shown in bold)

[0151]

[0152] Table 9.3 Classification accuracy (three) obtained by different methods on the Trento dataset (best results are shown in bold)

[0153]

[0154] Based on the above, the advantages of the present application are that when the application is used, the Hierarchical Fusion Network based on Frequency-Spatial Collaborative Learning (HFS-CL) is used to realize the joint classification of HSI and LiDAR, the HFS-CL uses a Hierarchical Mamba Frequency Enhancement (HMFE) module to extract frequency domain features from hyperspectral data, the HMFE module combines the Mamba structure with the hierarchical frequency domain feature extraction mechanism, effectively enhancing the frequency domain representation ability, introduces the LFDF to replace the traditional fixed filter, dynamically adjusts the center frequency and bandwidth through the trainable Gaussian kernel, and adaptively adapts the frequency characteristics of diversified topography, at the same time, through the parallel processing and cross-layer fusion of the high-frequency, medium-frequency and low-frequency three branches, the inter-frequency cooperation is strengthened; a Multi-Level Spatial and Spectral Aggregation (MLS²A) module is used to effectively fuse HSI and LiDAR data in the spatial domain, the tree-shaped fusion structure and cross attention mechanism are used to strengthen the cross-scale interaction, the spatial attention and channel attention are applied to the multi-resolution features respectively, the dynamic correlation is realized through the cross attention, and the key information is reserved through the residual connection; an Adaptive Weight Fusion (AWF) module is used to dynamically integrate the frequency domain and spatial domain features, the AWF module dynamically integrates the cross-domain features in an adaptive weighting manner, and promotes the collaborative enhancement of frequency and spatial information.

[0155] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. A land cover classification method based on a frequency-space collaborative learning hierarchical fusion network, comprising the following steps: Step 1: Data preparation; Step 2: HMFE module extracts frequency domain features; Step 3: MLS²A module extracts spatial domain features; Step 4: AWF module dynamically integrates features; Step 5: Execute the classification task; Its key feature is: In step one above, the HSI and LiDAR data are preprocessed, and the preprocessed data is used as the model input. In step two above, frequency domain features are extracted from the preprocessed HSI data using the HMFE module; In step three above, the heterogeneous features of HSI and LiDAR data are fused in the spatial domain using the MLS²A module, and then spatial domain features are obtained through multi-scale downsampling and a tree-like fusion structure. In step four above, the frequency domain features obtained in step two and the spatial domain features obtained in step three are dynamically integrated by the AWF module to obtain the final joint features. In step five above, the joint features are mapped to land cover categories using a classifier, and the focus loss optimization model is employed.

2. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 1, characterized in that: Step one specifically includes the following steps: 1.1 Data definition: Let the HIS data be... LiDAR data is ,in Indicates the number of bands in a hyperspectral image. , Indicates the spatial dimensions of the image; 1.2 Patch Extraction: A center pixel patch extraction strategy is used to generate HIS patches. and LiDAR patch ,in Patch size; 1.3 Spectral normalization: The HSI patch is projected onto the 64-dimensional feature space through a 1×1 two-dimensional convolutional layer, resulting in... , and As input to the model.

3. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 1, characterized in that: Step two specifically includes the following steps: 2.1 Obtaining multi-scale frequency information: for the input After application layer normalization, multi-scale frequency information is captured through high-frequency, mid-frequency, and low-frequency branching; 2.2 Adaptive fusion: the output of the high-frequency, mid-frequency, and low-frequency branching processes... , , Perform adaptive weighted fusion to obtain Finally, the final frequency domain features are refined through a 3×3 convolutional layer. Adaptive weighted fusion can be expressed by the following formula: in, For high-frequency branch output, For its weight, For intermediate frequency branch output, For its weight, For low-frequency branch output, For its weight, As learnable weights, all are initially set to 0.5 and can be optimized and adjusted during training.

4. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 3, characterized in that: Step 2.1 specifically includes the following steps: 2.1.1 High-frequency branching processing: focusing on edge and texture information, processing the input... Applying 5×5 depthwise separable convolution to generate coarse-grained spatial features The features are transformed into the frequency domain using a Fast Fourier Transform, and the transformed feature map is denoted as... Then, after filtering by a learnable frequency domain filter, high-frequency components are preserved to obtain... ,right The high-frequency spatial domain feature map is then processed using the Mamba module. Enhanced features Finally, by combining residual connections and pointwise convolution, a high-frequency guided spatial domain feature map is output. The learnable frequency domain filter consists of a set of trainable Gaussian kernel functions. constitute, and These represent the learnable center frequency and bandwidth, respectively; 2.1.2 Intermediate Frequency Branch Processing: Focusing on structural information, processing the input... Applying 3×3 depthwise separable convolutions to generate mesoscale spatial features High-frequency branches yield high-frequency texture features. and The fusion yields a spatial domain feature map guided by mid-frequency. , After preserving the intermediate frequency (IF) component through a learnable frequency domain filter, the spatial domain feature map guided by the IF is obtained through an inverse fast Fourier transform. Then, the coarse-grained spatial features are processed and fused using the Mamba module. Mid-frequency enhancement features were obtained. Finally, a spatial domain feature map with enhanced mid-frequency is obtained through pointwise convolution. 2.1.3 Low-frequency branch processing: Focusing on global semantics, low-frequency spatial features are generated through 1×1 depthwise separable convolution. , Integrating mid-frequency enhancement features Then, the low-frequency components are preserved by a learnable frequency domain filter, and the process is generated by the Mamba module. And then with mid-frequency spatial characteristics Cross-layer fusion, followed by pointwise convolution, yields a low-frequency guided spatial feature map. .

5. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 4, characterized in that: The formula for the Fast Fourier Transform is as follows: in, The transformed frequency domain features, Represents the imaginary unit. Indicates inner product operation, It is the input spatial domain feature map, and the final transformed feature map is denoted as . ; The formula for the inverse fast Fourier transform is as follows: ； Processing high-frequency spatial domain feature maps using the Mamba module The process is as follows: The Mamba module contains two parallel branches. The first branch begins with a linear projection, followed by extraction of spatial patterns through depthwise separable convolution. The core module of the first branch is... The first branch is used to model long-range dependencies in the spectral dimension. The second branch first performs a linear transformation, and then introduces nonlinearity and gating mechanisms through a linear unit activation function with sigmoid weights. The outputs of the two branches are fused by element-wise multiplication, and the fused result is then restored to the original feature dimension through inverse linear projection. The entire process can be represented by the following formula: in, Represents high-frequency texture features, For modeling long-range dependencies Module, the symbol ⊙ represents element-wise product. This represents depthwise separable convolution. Indicates linear projection. Indicates inverse linear projection. This represents the linear unit activation function, and the final output is... It integrates high-frequency spatial information with global contextual semantics; Combining residual connections and pointwise convolution to output high-frequency guided spatial domain feature maps The process specifically involves: converting coarse-scale spatial features With enhanced features The information is added together and then fused through pointwise convolution to finally output the result. The calculation formula is as follows: in, This represents pointwise convolution.

6. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 4, characterized in that: The high-frequency branching is used to obtain high-frequency texture features. and The fusion yields a spatial domain feature map guided by mid-frequency. The process can be represented by the following formula: The process of preserving the intermediate frequency component using a learnable frequency domain filter can be represented by the following formula: in, Represents the Fast Fourier Transform. This indicates a learnable frequency domain filter. Indicates the inverse fast Fourier transform; Coarse-grained spatial features are processed and fused using the Mamba module. The process can be represented by the following formula: The spatial domain feature map with enhanced mid-frequency is obtained through pointwise convolution. The process can be represented by the following formula: in, This represents pointwise convolution.

7. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 1, characterized in that: Step three specifically includes the following steps: 3.1 Initial fusion: ... and The features are then stitched together to obtain the initial fused feature map. 3.2 Multi-scale downsampling: For and Multi-scale features are generated by applying two consecutive downsampling stages. and Each downsampling stage contains a 3×3 depthwise separable convolution, followed by BN and ReLU activation functions, then a 1×1 pointwise convolution, and finally BN and ReLU activation functions; 3.3 Tree-like fusion: A recursive structure is used to fuse multi-resolution features, respectively processing large-scale inputs. and small-scale input Parallel application of spatial attention and channel attention, with the outputs used to generate small-scale intermediate features through element-wise addition. and large-scale intermediate features , Upsampled to the same level using bilinear interpolation. With the same spatial dimensions, we obtain , and Through cross-attention fusion, we obtain , via residual connection and Add them together to get Finally, spatial domain features are obtained through 1×1 convolution. ; The calculation formula is as follows: in, The dimension of the key vector, query item and key items All by Extract, and value Then it comes from .

8. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 1, characterized in that: In step four, specifically: let the feature maps of the frequency domain and spatial domain branches be respectively... and The fusion process is then defined as: in, Indicates channel-dimensional splicing. For linear projection layers, It is the Sigmoid activation function. This indicates element-wise multiplication. These are learnable parameters. .

9. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 1, characterized in that: In step five, the classifier consists of three fully connected layers. The first two layers contain linear transformation, ReLU activation function and Dropout operation, and the third layer outputs class logits.

10. The land cover classification method based on a frequency-space collaborative learning hierarchical fusion network according to claim 1, characterized in that: In step five, a modulating factor is introduced to adjust the focus loss. and category weights Together, they reduce the relative loss contribution to correctly classified samples and highlight the importance of hard-to-classify samples. The definition of focus loss is as follows: in, For batch size, For the number of categories, Indicates sample Category The predicted probability, For this category Real labels, parameters Control the focus intensity, The balancing weights are adjusted based on category frequency.

Images