Land coverage classification method based on hyperspectral and laser radar layered frequency domain fusion

Abstract

The invention discloses a land coverage classification method based on hierarchical frequency domain fusion of a hyperspectrum and a laser radar, and the method is characterized in that firstly, a small sample learning framework based on hierarchical frequency domain fusion is provided, and high-frequency difference information is captured by extracting global and local shallow layer features; the deep interaction of low-frequency background information and high-frequency detail information is realized, the complementarity characteristics of the hyperspectrum and the LiDAR in the frequency domain are fully mined, and the classification precision in a complex scene is remarkably improved; secondly, a cross-modal small sample migration strategy is provided for the challenge of scarcity of labeled samples in a target domain, high-frequency prior knowledge of LiDAR is mapped to high-frequency features of hyperspectrum of the target domain through joint learning of source domain multi-modal data (hyperspectrum and LiDAR), and the problem that features of single-source data are incomplete is solved;

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a land cover classification method that integrates hyperspectral and laser radar layered frequency domains. Background Art

[0002] Hyperspectral imagery can distinguish objects with significantly different spectral characteristics through continuous spectral information, but it is easily limited by "different objects with the same spectrum" and insufficient spatial resolution in complex scenes. LiDAR data can distinguish targets with similar spectra but different altitudes through elevation information, but lacks spectral detail. The collaborative observation of hyperspectral imagery and laser radar (LiDAR), as a non-contact, long-distance detection technology, provides a richer feature dimension for surface cover classification. Traditional fusion methods can combine the advantages of both types of data, but they suffer from problems such as insufficient feature interaction and limited classification accuracy. For example, the sparse low-rank decomposition method proposed by Rasti et al. alleviates the Hughes phenomenon but is still constrained by the curse of hyperspectral dimensionality. The high-order tensor decomposition method proposed by Xue et al. improves feature extraction capabilities but still does not fully utilize deep spatial correlations.

[0003] In recent years, although deep learning-based multi-branch networks (CNNs) have improved classification performance through modal feature extraction, the correlation of multimodal spatial information has not been fully modeled. In addition, existing methods mostly adopt a front-end single-stage feature fusion strategy, ignoring the complementarity of frequency domain information, resulting in insufficient cross-modal interaction capabilities. Transformer captures global contextual information through a self-attention mechanism, significantly improving classification accuracy. However, Transformer is weaker than CNN in its ability to perceive local details and requires a large number of labeled samples for training, which limits its application in scenarios with few samples. For example, the hybrid self-attention and CNN framework proposed by Li et al. reduces computational overhead, but the mining of frequency domain features is still insufficient; although the cross-modal Transformer of Roy et al. fuses HSI and LiDAR data through cross-attention, it does not solve the problem of multi-scale feature extraction.

[0004] Few-shot learning (FSL) offers a new approach to addressing the high cost of remote sensing data annotation by learning transferable features from a small number of labeled samples. Existing methods mitigate the challenge of insufficient annotation through metric learning but ignore the impact of cross-domain data distribution differences. While Li et al.'s cross-domain FSL method (DCFSL) improves generalization through a domain alignment strategy, it fails to fully leverage the complementary information of multimodal data. In recent years, the graph-based information aggregation-based Gia-CFSL framework and the adaptive weighted contrastive prototype network (WCPN-FSL) have made progress in single-modal HSI classification but face the bottleneck of insufficient cross-modal knowledge transfer. For example, existing methods often rely on a single data source (e.g., HSI alone) and fail to effectively leverage the low annotation cost of auxiliary modalities such as LiDAR. Furthermore, traditional frequency-domain fusion methods fail to integrate high- and low-frequency features, resulting in limited classification accuracy. Summary of the Invention

[0005] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides a land cover classification method by fusion of hyperspectral and lidar layered frequency domain.

[0006] The technical solution of the present invention is: a land cover classification method based on layered frequency domain fusion of hyperspectral and lidar, which is carried out in the following steps:

[0007] Step 1. Create and initialize the network N HiF , the network N HiF Contains subnetwork N for extracting shallow features MSCB , sub-network N that extracts deep low-frequency features LF , sub-network N that extracts deep high-frequency features HF , fusion sub-network N FUSE and classification subnetwork N D ;

[0008] Step 2. Perform data preprocessing on the source domain hyperspectral image training set H, the source domain LiDAR image training set L, the target domain hyperspectral image training set T, the target domain hyperspectral image training set H2, and the manually annotated source domain and target domain pixel point coordinate sets and label sets to obtain the source domain support set and source domain query set, the target domain support set and query set, and the target domain hyperspectral feature training set;

[0009] Step 3. Use the source domain support set, source domain query set and target domain hyperspectral feature training set to train the network N HiF Perform source domain training; use the target domain support set, target domain query set and target domain hyperspectral feature training set to train the network N HiF Perform target domain training; the source domain training and target domain training are performed alternately to obtain a trained network N HiF ;

[0010] Step 4. Use the trained sub-network N HiF Treat the test set Perform feature extraction and classification through classifier.

[0011] The step 1 is specifically as follows:

[0012] Step 1.1 Establish and initialize the sub-network N for extracting shallow features MSCB , the sub-network N that extracts shallow features MSCB Contains two sets of modules L arranged alternately Attention and L FeedForward ;

[0013] The module L Attention By custom module L PC and L MA constitute;

[0014] The custom module L PC Contains Conv 11 、Conv 12 、Conv 21 、Conv 22 , 5 groups of convolutional layers of TConv, 1 layer of activation operation, and the nonlinear activation function ReLU is selected for operation;

[0015] The Conv 11 Contains one layer of convolution operation, containing 32 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0016] The Conv 12 Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0017] The Conv 21 Contains one layer of convolution operation, containing 32 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0018] The Conv 22 Contains one layer of convolution operation, containing 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0019] The TConv consists of one layer of transposed convolution operation, containing 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 2 pixels;

[0020] The custom module L PC The operation is as follows:

[0021] The input four-dimensional tensor X∈R bs×c×p×p At the same time, it is fed into Conv 11 and Conv 21 Calculate X1∈R bs×32×p×p and X3∈R bs×32×p×p , where bs represents the batch size, c represents the number of channels, and p×p represents the block size;

[0022] Add X1 and X3 to get F1, and send F1 and X1 to Conv 12 and Conv 22 Calculate F2∈R bs×64×p×p , X2∈R bs×64×p×p ;

[0023] Add X2 and F2 to get F3, send F3 to TConv and ReLU for calculation, and then get F4 after one layer of activation operation. Then calculate the local attention matrix L according to formula (1):

[0024]

[0025] Perform matrix multiplication on the local attention matrix L and the four-dimensional tensor X to obtain the local attention feature X PC ∈R bs ×c×p×p ;

[0026] The custom module L MA There are three groups of convolutional layers Conv_Q1, Conv_K1, and Conv_V1. Each of the three convolutional layers contains two layers of convolution operations. The first layer contains 192 convolution kernels of size 1×1, and the second layer is depth-wise separable convolution, which contains 192 convolution kernels of size 3×3. Both convolution operations perform convolution operations with a step size of 1 pixel.

[0027] The custom module L MA The operation is as follows:

[0028] (a) Input four-dimensional tensor X∈R bs×c×p×p Send it to the LayerNorm layer for normalization, and then send it to the convolutional layers Conv_Q1, Conv_K1, and Conv_V1 respectively to obtain the feature Q = {Q i |i=1,...,c}∈R c×p×p , K={K i |i=1,...,c}∈R c×p×p 、V={V i |i=1,...,c}∈R c×p×p ;

[0029] (b) Use the reshape operation to map Q and K to For the i-th channel, i=1,...,c,Q i The jth position in K i The attention weight G between the l-th position in i,j,l Calculated by formula (2), j=1,...,N1, l=1,...,N1;

[0030]

[0031] Where N1 represents the total number of features p×p, represents the transpose of the lth element of the i-th channel of K, represents the transpose of the kth element of the ith channel of K, and then performs matrix multiplication on G and V according to formula (3) to obtain the preliminary global attention feature F MA :

[0032]

[0033] Where V i,l Represents the lth position element of the i-th channel;

[0034] (c) Use the reshape operation to transform F MA Mapping to R bs×c×p×p , and after one layer of two-dimensional convolution operation, 64 convolution kernels of size 1×1, the global attention feature X is obtained MA ∈R bs×c×p×p ;

[0035] (d) Add the global attention feature and the local attention feature to obtain the multi-scale feature X PM , the multi-scale feature X PM Send L FeedForward module;

[0036] The L FeedForward The module operates as follows:

[0037] (a) Multi-scale feature X PM After layer normalization, it undergoes three layers of two-dimensional convolution operations, each layer includes 64 convolution kernels of size 1×1, and operates with a step size of 1 pixel to obtain the feature

[0038] (b) Yes Perform channel maximization operation and GeLU activation function to obtain channel attention feature F ch ∈R bs×1×p×p ,right Perform size maximization operation and GeLU activation function to obtain size attention feature F hw ∈Rbs×4c×1×1 ,right After one layer of depth-wise separable convolution operation, including 256 convolution kernels of size 3×3 and a step size of 1 pixel, the feature F is obtained. dconv ∈R bs×4c×p×p ;

[0039] (c) Feature F dconv , channel attention feature F ch and size attention feature F hw According to formula (4), the shallow feature Y∈R bs×c×p×p :

[0040] Y = Conv 1_1 (F ch ·F dconv +F hw ·F dconv ) (4)

[0041] Among them, Conv 1_1 Represents a convolution operation with 64 1×1 kernels and a step size of 1 pixel.

[0042] The shallow feature extraction sub-network N MSCB The input four-dimensional tensor X is calculated according to formula (5) to obtain the final shallow feature X MSCB ;

[0043]

[0044] Step 1.2 Establish and initialize the sub-network N for extracting deep low-frequency features LF , the sub-network N that extracts deep low-frequency features LF Contains three groups of convolutional layers Conv_Q2, Conv_K2, Conv_V2 and three groups of custom convolutional layers

[0045] The three groups of convolutional layers Conv_Q2, Conv_K2, and Conv_V2 each contain two layers of convolution operations. The first layer contains 192 convolution kernels of size 1×1, and the second layer is a depth-wise separable convolution, containing 192 convolution kernels of size 3×3. Both layers of convolution operations perform convolution operations with a step size of 1 pixel.

[0046] described Contains 1 layer of convolution operation, containing 128 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0047] described Contains 1 layer of depth-wise separable convolution operation, containing 128 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0048] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0049] The sub-network N that extracts deep low-frequency features LF The specific operations are as follows:

[0050] (a) Input four-dimensional tensor X MSCB Send it to the LayerNorm layer for normalization, and then send it to the convolutional layers Conv_Q2, Conv_K2, and Conv_V2 respectively, to obtain the feature Q2 = {Q2 i |i=1,...,c}∈R c×p×p 、K2={K2 i |i=1,...,c}∈R c×p×p 、V2={V2 i |i=1,...,c}∈R c×p×p ;

[0051] (b) Use the reshape operation to map Q2 and K2 to And calculate the low-frequency attention matrix G according to formula (6) L :

[0052]

[0053] in Indicates that on the i-th channel, Q2 i The j-th position element and K2 i The attention weight between the l-th position elements, N1 represents the total number of features p×p, represents the transpose of the lth element of the i-th channel of K2, represents the transpose of the kth element of the ith channel of K2, and then according to formula (7) G L Perform matrix multiplication with V2 to obtain preliminary low-frequency attention features :

[0054]

[0055] Among them V2 i,l Represents the lth position element of the i-th channel;

[0056] (c) With X MSCB Add up to get the initial deep low-frequency feature F L, normalized by the LayerNorm layer, and then sent to Get features Send it in again And through the GeLU activation function to get go through Get features and with Add up to get the final deep low-frequency feature X B ;

[0057] Step 1.3 Establish and initialize the sub-network N for extracting deep high-frequency features HF , the sub-network N that extracts deep high-frequency features HF Contains three identical custom modules N FFIM ;

[0058] The custom N FFIM The module contains 4 sets of custom convolutional layers

[0059] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel size performs convolution operation with a step size of 1 pixel;

[0060] described Each layer contains three convolution operations. The first layer of the three convolution operations contains 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operations with a step size of 1 pixel; the second layer of depthwise convolution operations contains 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operations with a step size of 1 pixel; the third layer of convolution operations contains 32 convolution kernels of size 1×1, and each convolution kernel performs convolution operations with a step size of 1 pixel;

[0061] The custom N FFIM The module operation steps are as follows:

[0062] (a) Input four-dimensional tensor X MSCB Send in Calculate the feature f FFIM , and along the dimension channel f FFIM Mapped into two tensors

[0063] (b)f 1,1 and f 2,1 According to formula (8), the shallow high-frequency feature f is calculated H :

[0064]

[0065] Where CAT(,) represents the splicing operation;

[0066] (c) Shallow high-frequency features f H Loop through the third N FFIM The module obtains deep high-frequency features X D ;

[0067] Step 1.5: Create and initialize the fusion sub-network N FUSE , contains six fully connected layers and two convolutional layers The six fully connected layers are Linear1, Linear2, Linear3, Linear4, Linear5, and Linear6;

[0068] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0069] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0070] The fusion sub-network N FUSE The specific operations are as follows:

[0071] (a) The deep low-frequency feature X B Input to Linear1 and Linear2 to get features The shallow feature X MSCB Input to Linear3 and Linear4 to get features The deep high-frequency feature X D Input to Linear5 and Linear6 to get features

[0072] (b) Use the reshape operation to transform Q 1 , K 1 , K 2 , Q 2 Map to And calculate the low-frequency attention matrix AM1 and the high-frequency attention matrix AM2 according to formula (9);

[0073]

[0074] in Indicates that on the i-th channel, The j-th position element of The attention weight between the l-th position elements, N1 represents the total number of features p×p, Indicates that on the i-th channel, The j-th position element of The attention weight between the l-th position elements, K 1 The transpose of the lth element of the ith channel of K 2 The transpose of the lth channel of the i-th element of K 1 The transpose of the kth element of the ith channel of K 2 The transpose of the k-th channel of the i-th element of ;

[0075] (c) Use the reshape operation to transform V 1 、V 2 Mapping to R c×(p×p) , and then according to formula (10) AM1 and V 1 , AM2 and V 2 Perform matrix multiplication to obtain

[0076]

[0077] in Indicates V 1 The vector composed of the lth position element of the ith channel, Indicates V 2 The vector composed of the lth position element of the ith channel;

[0078] (d) Calculate the fusion feature X according to formula (11) Fuse ;

[0079]

[0080] in and Represents a convolution operation, which contains 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0081] Step 1.6: Create and initialize the classification subnetwork N D , contains a linear layer Linear F , Linear F Contains 1 output unit and 4 groups of fully connected layers FC1, FC2, FC3, FC4;

[0082] FC1, FC2, FC3, and FC4 each contain a linear layer, a ReLU activation function, and a Dropout operation;

[0083] X Fuse Enter FC1, FC2, FC3, FC4 in sequence, and then enter Linear F Get the domain classification feature X domain .

[0084] The step 2 is specifically as follows:

[0085] Step 2.1: Based on the manually labeled source domain pixel point coordinate set and label set, perform data preprocessing on the source domain hyperspectral image training set H and the source domain LiDAR image training set L;

[0086] Step 2.1.1 Based on the manually labeled source domain pixel point coordinate set, extract 1 and 19 labeled pixel points of each class from the source domain hyperspectral image training set H as the source domain hyperspectral support set. and queryset And extract 1 and 19 labeled pixel points of each class from the training set L of the source domain LiDAR image as the source domain LiDAR support set and queryset in express The i-th pixel point in express The i-th pixel point in express The i-th pixel point in express The i-th pixel point in , N represents the total number of pixel points in the source domain support set, and M represents the total number of pixel points in the source domain query set;

[0087] Step 2.1.2. According to the definition of formula (12), After standardization, we can obtain and

[0088]

[0089] Step 2.1.3 is as follows: Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks, which are

[0090] Step 2.1.4 and Integrate into triples

[0091] in the form of They represent the pixel pairs composed of the source domain hyperspectral image and the LiDAR image, and the spatial coordinates of the pixel pairs are the same. and Indicates the corresponding true category label;

[0092] Step 2.2: Preprocess the target domain hyperspectral image training set T using the manually labeled target domain pixel point coordinate set and label set;

[0093] Step 2.2.1 Based on the pixel coordinate set of the manually labeled target domain, extract 1 and 19 labeled pixel points of each class from the training set T of the target domain hyperspectral image as the target domain hyperspectral support set. and queryset in express The i-th pixel point in express The i-th pixel point in , B represents the total number of pixel points in the target domain support set, and C represents the total number of pixel points in the target domain query set;

[0094] Step 2.2.2 According to the definition of formula (13), After standardization, we get

[0095]

[0096] Step 2.2.3 is as follows: Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks.

[0097] Step 2.2.4 and Gaussian noise is added to each pixel block in the target domain to obtain the hyperspectral training set and

[0098] Step 2.2.5 and Integrate into two groups in the form of Represents a pixel pair consisting of a target domain hyperspectral image and a pixel label, and the spatial coordinates of the pixel pair are the same. and Indicates the corresponding true category label;

[0099] Step 2.3: Preprocess the target domain hyperspectral image training set H2 using the manually labeled target domain pixel point coordinate set and label set;

[0100] Step 2.3.1 Based on the pixel coordinate set of the manually labeled target domain, extract 5 samples of each category from the training set H2 of the target domain hyperspectral image to obtain the target domain hyperspectral training set in express The i-th pixel point in , where B2 represents the total number of pixel points in the target domain training set;

[0101] Step 2.3.2 According to the definition of formula (14), After standardization, we get

[0102]

[0103] Step 2.3.3 Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks as the center.

[0104] Step 2.3.4 Gaussian noise is added to each pixel block in the target domain to obtain the hyperspectral training set

[0105] The step 3 is specifically as follows:

[0106] Step 3.1: As the source domain support set, As the source domain query set, the source domain support set, the source domain query set and the target domain hyperspectral training set Network N HiF Conduct source domain training;

[0107] Step 3.1.1 Use the sub-network N that extracts shallow features MSCB Support and query sets for source domain hyperspectral and LiDAR and And the target domain training set Perform feature extraction to obtain the corresponding shallow features

[0108] Step 3.1.2 Use the sub-network N that extracts deep low-frequency features LF Shallow features of hyperspectral support sets and query set shallow features And the shallow features of the target domain training set Perform low-frequency feature extraction to obtain the corresponding low-frequency features

[0109] Step 3.1.3 Use the sub-network N that extracts deep high-frequency features HF Support for hyperspectral and LiDAR shallow features and query set shallow features And the shallow features of the target domain training set Perform high-frequency feature extraction to obtain high-frequency features of the hyperspectral support set and query set as well as high-frequency features of the target domain training set

[0110] Step 3.1.4 Use the fusion sub-network N FUSE Shallow features for hyperspectral support sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain support set fusion features Shallow features for hyperspectral query sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain query set fusion features

[0111] Step 3.1.5 Using the fusion sub-network N FUSE Shallow features of target domain hyperspectral Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain the target domain hyperspectral training set fusion feature F T ;

[0112] Weighted cross entropy Calculate according to formula (15):

[0113]

[0114] Where W, D are query set fusion features Tensor, y i yes The real query sample label;

[0115] Step 3.1.6 Fusion features of source domain hyperspectral support set Fusion features with query set And the target domain hyperspectral fusion features are spliced and sent to the classification sub-network N D Perform domain classification to obtain domain discriminant features described Calculate according to formula (16);

[0116]

[0117] in is the real sample domain label, bs is the number of batch samples;

[0118] Step 3.1.7 takes the total loss function of the source domain as Network N HiF Perform source domain training, let iter→iter+1, if the number of iterations iter>Total_iter, then the trained network N is obtained HiF , go to step 4, otherwise use the back error propagation algorithm based on stochastic gradient descent and loss L s Update N HiF Parameters, and go to step 3.2, Total_iter represents the preset number of training iterations;

[0119] Step 3.2: As the target domain hyperspectral support set and query set, the target domain hyperspectral support set, target domain hyperspectral query set and target domain hyperspectral training set are used. Network N HiF Conduct training;

[0120] Step 3.2.1 Use the sub-network N that extracts shallow features MSCB Hyperspectral support set for target domain and queryset And the target domain training set Perform feature extraction to obtain the corresponding shallow features

[0121] Step 3.2.2 Use the sub-network N that extracts deep low-frequency features LF Hyperspectral support set and query set shallow features And the shallow features of the target domain training set Perform low-frequency extraction to obtain the corresponding low-frequency features

[0122] Step 3.2.3 Use the sub-network N that extracts deep high-frequency features HF Shallow features of hyperspectral support sets and query set shallow features And the shallow features of the target domain training set Perform high-frequency extraction to obtain the corresponding high-frequency features

[0123] Step 3.2.4 Using the fusion sub-network N FUSE Shallow features for hyperspectral support sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain support set fusion features Shallow features for hyperspectral query sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain query set fusion features

[0124] Step 3.2.5 Using the fusion sub-network N FUSE Shallow features of target domain hyperspectral Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain the target domain hyperspectral training set fusion feature F T ;

[0125] Weighted cross entropy Calculate according to formula (17):

[0126]

[0127] Where W, D are query set fusion features Tensor, y i yes The real query sample label;

[0128] Step 3.2.6: Fusing the target domain hyperspectral support set features Fusion features with query set And the target domain hyperspectral training set fusion feature F T Splice and send to the classification sub-network N D Perform domain classification and obtain domain discriminant features described Calculate according to formula (18);

[0129]

[0130] in is the real sample domain label, bs is the number of batch samples;

[0131] Step 3.2.7 takes the total loss function of the target domain as Network N HiF Perform target domain training, set iter→iter+1, if the number of iterations iter>Total_iter, then the trained network N is obtained HiF , go to step 4, otherwise use the back error propagation algorithm based on stochastic gradient descent and loss L T Update N HiFParameters, and go to step 3.1, Total_iter represents the preset number of training iterations.

[0132] The step 4 is specifically as follows:

[0133] Step 4.1: Perform data preprocessing on the hyperspectral image H' to be measured;

[0134] Step 4.1.1 Extract all pixel points from the hyperspectral image H' to be measured to form a set T H ={T H,i |i=1,...,U}, where T H,i Indicates T H The i-th pixel of , U represents the total number of all pixels;

[0135] Step 4.1.2 According to the definition of formula (19), H After standardization, we get

[0136]

[0137] Step 4.1.3 Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks.

[0138] Step 4.2 Use the trained shallow feature extraction sub-network N MSCB Hyperspectral pixel block collection Perform feature extraction to obtain the corresponding shallow features

[0139] Step 4.3 Use the trained sub-network N to extract deep low-frequency features LF Shallow features Perform low-frequency extraction to obtain the corresponding low-frequency features

[0140] Step 4.4 Use the trained deep high-frequency feature extraction sub-network N HF Shallow features Perform high-frequency extraction to obtain the corresponding high-frequency features

[0141] Step 4.5 uses the trained fusion sub-network N FUSE Shallow features Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain fusion features

[0142] Step 4.6 uses KNN classifier to classify the fusion features Classification is performed to calculate the classification prediction results.

[0143] Compared with existing technologies, this paper offers two advantages: First, it proposes a small-sample learning framework based on hierarchical frequency domain fusion. By extracting global and local shallow features, it captures high-frequency difference information, achieves a deep interaction between low-frequency background information and high-frequency detail information, and fully exploits the complementary characteristics of hyperspectral and LiDAR in the frequency domain, significantly improving classification accuracy in complex scenes. Second, to address the challenge of scarce labeled samples in the target domain, it proposes a cross-modal small-sample transfer strategy. By jointly learning multimodal source domain data (hyperspectral and LiDAR), it maps the high-frequency prior knowledge of LiDAR to the high-frequency features of the target domain hyperspectral, thus solving the problem of incomplete single-source data features. Experimental results show that the overall classification accuracy on the Houston 2013, Trento, and MUUFL datasets reaches 97.83%, 94.43%, and 85.20%, respectively, with average accuracies of 96.52%, 92.18%, and 83.11%. This paper effectively improves the classification accuracy of land objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 is the network N of the embodiment of the present invention HiF Schematic diagram of the structure.

[0145] Figure 2 is the sub-network N for extracting shallow features in the embodiment of the present invention MSCB Schematic diagram of the structure.

[0146] Figure 3 This is the fusion sub-network N in the embodiment of the present invention. FUSE Schematic diagram of the structure.

[0147] Figure 4 This is a comparison chart of the visual detection results of the Trento dataset between the embodiment of the present invention and the prior art, where 5 labeled samples per category are used for training and the rest are used as test samples.

[0148] Figure 5 This is a comparison chart of the visual detection results of the MUUFL dataset between the embodiment of the present invention and the prior art, when 5 labeled samples per category are used for training and the rest are test samples.

[0149] Figure 6 This is a comparison chart of the visual detection results of the Houston 2013 dataset between the embodiment of the present invention and the prior art, where 5 labeled samples per category are used for training and the rest are used as test samples. DETAILED DESCRIPTION

[0150] A land cover classification method based on hyperspectral and laser radar layered frequency domain fusion of the present invention is carried out in the following steps:

[0151] Step 1. Create and initialize Figure 1 The network N shown HiF , the network N HiF Contains subnetwork N for extracting shallow features MSCB , sub-network N that extracts deep low-frequency features LF , sub-network N that extracts deep high-frequency features HF , fusion sub-network N FUSE and classification subnetwork N D :

[0152] Step 1.1 Create and initialize Figure 2 The subnetwork N shown to extract shallow features MSCB , the sub-network N that extracts shallow features MSCB Contains two sets of modules L arranged alternately Attention and L FeedForward ;

[0153] The module L Attention By custom module L PC and L MA constitute;

[0154] The custom module L PC Contains Conv 11 、Conv 12 、Conv 21 、Conv 22 , 5 groups of convolutional layers of TConv, 1 layer of activation operation, and the nonlinear activation function ReLU is selected for operation;

[0155] The Conv 11 Contains one layer of convolution operation, containing 32 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0156] The Conv 12 Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0157] The Conv 21 Contains one layer of convolution operation, containing 32 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0158] The Conv 22 Contains one layer of convolution operation, containing 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0159] The TConv consists of one layer of transposed convolution operation, containing 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 2 pixels;

[0160] The custom module L PC The operation is as follows:

[0161] The input four-dimensional tensor X∈R bs×c×p×p At the same time, it is fed into Conv 11 and Conv 21 Calculate X1∈R bs×32×p×p and X3∈R bs×32×p×p , where bs represents the batch size, c represents the number of channels, and p×p represents the block size;

[0162] Add X1 and X3 to get F1, and send F1 and X1 to Conv 12 and Conv 22 Calculate F2∈R bs×64×p×p , X2∈R bs×64×p×p ;

[0163] Add X2 and F2 to get F3, send F3 to TConv and ReLU for calculation, and then get F4 after one layer of activation operation. Then calculate the local attention matrix L according to formula (1):

[0164]

[0165] Perform matrix multiplication on the local attention matrix L and the four-dimensional tensor X to obtain the local attention feature X PC ∈R bs ×c×p×p ;

[0166] The custom module L MA There are three groups of convolutional layers Conv_Q1, Conv_K1, and Conv_V1. Each of the three convolutional layers contains two layers of convolution operations. The first layer contains 192 convolution kernels of size 1×1, and the second layer is depth-wise separable convolution, which contains 192 convolution kernels of size 3×3. Both convolution operations perform convolution operations with a step size of 1 pixel.

[0167] The custom module L MA The operation is as follows:

[0168] (d) Input four-dimensional tensor X∈R bs×c×p×p Send it to the LayerNorm layer for normalization, and then send it to the convolutional layers Conv_Q1, Conv_K1, and Conv_V1 respectively to obtain the feature Q = {Q i |i=1,...,c}∈Rc×p×p , K={K i |i=1,...,c}∈R c×p×p 、V={V i |i=1,...,c}∈R c×p×p ;

[0169] (e) Use the reshape operation to map Q and K to For the i-th channel, i=1,...,c,Q i The jth position in K i The attention weight G between the l-th position in i,j,l Calculated by formula (2), j=1,...,N1, l=1,...,N1;

[0170]

[0171] Where N1 represents the total number of features p×p, represents the transpose of the lth element of the i-th channel of K, represents the transpose of the kth element of the ith channel of K, and then performs matrix multiplication on G and V according to formula (3) to obtain the preliminary global attention feature F MA :

[0172]

[0173] Where V i,l Represents the lth position element of the i-th channel;

[0174] (f) Use the reshape operation to transform F MA Mapping to R bs×c×p×p , and after one layer of two-dimensional convolution operation, 64 convolution kernels of size 1×1, the global attention feature X is obtained MA ∈R bs×c×p×p ;

[0175] (d) Add the global attention feature and the local attention feature to obtain the multi-scale feature X PM , the multi-scale feature X PM Send L FeedForward module;

[0176] The L FeedForward The module operates as follows:

[0177] (a) Multi-scale feature X PM After layer normalization, it undergoes three layers of two-dimensional convolution operations, each layer includes 64 convolution kernels of size 1×1, and operates with a step size of 1 pixel to obtain the feature

[0178] (b) Yes Perform channel maximization operation and GeLU activation function to obtain channel attention feature F ch ∈R bs×1×p×p ,right Perform size maximization operation and GeLU activation function to obtain size attention feature F hw ∈R bs×4c×1×1 ,right After one layer of depth-wise separable convolution operation, including 256 convolution kernels of size 3×3 and a step size of 1 pixel, the feature F is obtained. dconv ∈R bs×4c×[p×[p ;

[0179] (c) Feature F dconv , channel attention feature F ch and size attention feature F hw According to formula (4), the shallow feature Y∈R bs×c×p×p :

[0180] Y = Conv 1_1 (F ch ·F dconv +F hw ·F dconv ) (4)

[0181] Among them, Conv 1_1 Represents a convolution operation with 64 1×1 kernels and a step size of 1 pixel.

[0182] The shallow feature extraction sub-network N MSCB The input four-dimensional tensor X is calculated according to formula (5) to obtain the final shallow feature X MSCB ;

[0183]

[0184] Step 1.2 Establish and initialize the sub-network N for extracting deep low-frequency features LF , the sub-network N that extracts deep low-frequency features LF Contains three groups of convolutional layers Conv_Q2, Conv_K2, Conv_V2 and three groups of custom convolutional layers

[0185] The three groups of convolutional layers Conv_Q2, Conv_K2, and Conv_V2 each contain two layers of convolution operations. The first layer contains 192 convolution kernels of size 1×1, and the second layer is a depth-wise separable convolution, containing 192 convolution kernels of size 3×3. Both layers of convolution operations perform convolution operations with a step size of 1 pixel.

[0186] described Contains 1 layer of convolution operation, containing 128 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0187] described Contains 1 layer of depth-wise separable convolution operation, containing 128 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0188] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0189] The sub-network N that extracts deep low-frequency features LF The specific operations are as follows:

[0190] (a) Input four-dimensional tensor X MSCB Send it to the LayerNorm layer for normalization, and then send it to the convolutional layers Conv_Q2, Conv_K2, and Conv_V2 respectively, to obtain the feature Q2 = {Q2 i |i=1,...,c}∈R c×p×p 、K2={K2 i |i=1,...,c}∈R c×p×p 、V2={V2 i |i=1,...,c}∈R c×p×p ;

[0191] (b) Use the reshape operation to map Q2 and K2 to And calculate the low-frequency attention matrix G according to formula (6) L :

[0192]

[0193] in Indicates that on the i-th channel, Q2 i The j-th position element and K2 i The attention weight between the l-th position elements, N1 represents the total number of features p×p, represents the transpose of the lth element of the i-th channel of K2, represents the transpose of the kth element of the ith channel of K2, and then according to formula (7) G L Perform matrix multiplication with V2 to obtain preliminary low-frequency attention features :

[0194]

[0195] Among them V2 i,lRepresents the lth position element of the i-th channel;

[0196] (c) With X MSCB Add up to get the initial deep low-frequency feature F L , normalized by the LayerNorm layer, and then sent to Get features Re-enter And through the GeLU activation function to get go through Get features and with Add up to get the final deep low-frequency feature X B ;

[0197] Step 1.3 Establish and initialize the sub-network N for extracting deep high-frequency features HF , the sub-network N that extracts deep high-frequency features HF Contains three identical custom modules N FFIM ;

[0198] The custom N FFIM The module contains 4 sets of custom convolutional layers

[0199] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel size performs convolution operation with a step size of 1 pixel;

[0200] described Each layer contains three convolution operations. The first layer of the three convolution operations contains 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operations with a step size of 1 pixel; the second layer of depthwise convolution operations contains 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operations with a step size of 1 pixel; the third layer of convolution operations contains 32 convolution kernels of size 1×1, and each convolution kernel performs convolution operations with a step size of 1 pixel;

[0201] The custom N FFIM The module operation steps are as follows:

[0202] (a) Input four-dimensional tensor X MSCB Send in Calculate the feature f FFIM , and along the dimension channel f FFIM Mapped into two tensors

[0203] (b)f 1,1 and f 2,1 According to formula (8), the shallow high-frequency feature f is calculatedH :

[0204]

[0205] Where CAT(,) represents the splicing operation;

[0206] (c) Shallow high-frequency features f H Loop through the third N FFIM The module obtains deep high-frequency features X D ;

[0207] Step 1.5 Create and initialize Figure 3 The fusion sub-network N shown FUSE , contains six fully connected layers and two convolutional layers The six fully connected layers are Linear1, Linear2, Linear3, Linear4, Linear5, and Linear6, each of which has 1600 output units;

[0208] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0209] described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0210] The fusion sub-network N FUSE The specific operations are as follows:

[0211] (a) The deep low-frequency feature X B Input to Linear1 and Linear2 to get features The shallow feature X MSCB Input to Linear3 and Linear4 to get features The deep high-frequency feature X D Input to Linear5 and Linear6 to get features

[0212] (b) Use the reshape operation to transform Q 1 , K 1 , K 2 , Q 2 Map to And calculate the low-frequency attention matrix AM1 and the high-frequency attention matrix AM2 according to formula (9);

[0213]

[0214] in Indicates that on the i-th channel, The j-th position element of The attention weight between the l-th position elements, N1 represents the total number of features p×p, Indicates that on the i-th channel, The j-th position element of The attention weight between the l-th position elements, K 1 The transpose of the lth element of the ith channel of K 2 The transpose of the lth channel of the i-th element of K 1 The transpose of the kth element of the ith channel of K 2 The transpose of the k-th channel of the i-th element of ;

[0215] (c) Use the reshape operation to transform V 1 、V 2 Mapping to R c×(p×p) , and then according to formula (10) AM1 and V 1 , AM2 and V 2 Perform matrix multiplication to obtain

[0216]

[0217] in Indicates V 1 The vector composed of the lth position element of the ith channel, Indicates V 2 The vector composed of the lth position element of the ith channel;

[0218] (d) Calculate the fusion feature X according to formula (11) Fuse ;

[0219]

[0220] in and Represents a convolution operation, which contains 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0221] Step 1.6: Create and initialize the classification subnetwork N D , contains a linear layer Linear F , LinearF Contains 1 output unit and 4 groups of fully connected layers FC1, FC2, FC3, FC4;

[0222] Each of FC1, FC2, FC3, and FC4 contains a linear layer with 1024 output units and a ReLU activation function and a Dropout operation with a parameter of 0.5;

[0223] X Fuse Enter FC1, FC2, FC3, FC4 in sequence, and then enter Linear F Get the domain classification feature X domain .

[0224] Step 2. Perform data preprocessing on the source domain hyperspectral image training set H, the source domain LiDAR image training set L, the target domain hyperspectral image training set T, the target domain hyperspectral image training set H2, and the manually labeled source domain and target domain pixel point coordinate sets and label sets to obtain the source domain support set and source domain query set, the target domain support set and query set, and the target domain hyperspectral feature training set:

[0225] Step 2.1: Based on the manually labeled source domain pixel point coordinate set and label set, perform data preprocessing on the source domain hyperspectral image training set H and the source domain LiDAR image training set L;

[0226] Step 2.1.1 Based on the manually labeled source domain pixel point coordinate set, extract 1 and 19 labeled pixel points of each class from the source domain hyperspectral image training set H as the source domain hyperspectral support set. and queryset And extract 1 and 19 labeled pixel points of each class from the training set L of the source domain LiDAR image as the source domain LiDAR support set and queryset in express The i-th pixel point in express The i-th pixel point in express The i-th pixel point in express The i-th pixel point in , N represents the total number of pixel points in the source domain support set, and M represents the total number of pixel points in the source domain query set;

[0227] Step 2.1.2. According to the definition of formula (12), After standardization, we can obtain and

[0228]

[0229] Step 2.1.3 is as follows: Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks, which are

[0230] Step 2.1.4 and Integrate into triples

[0231] 1,....,M}, where They represent the pixel pairs composed of the source domain hyperspectral image and the LiDAR image, and the spatial coordinates of the pixel pairs are the same. and Indicates the corresponding true category label;

[0232] Step 2.2: Preprocess the target domain hyperspectral image training set T using the manually labeled target domain pixel point coordinate set and label set;

[0233] Step 2.2.1 Based on the pixel coordinate set of the manually labeled target domain, extract 1 and 19 labeled pixel points of each class from the training set T of the target domain hyperspectral image as the target domain hyperspectral support set. and queryset in express The i-th pixel point in express The i-th pixel point in , B represents the total number of pixel points in the target domain support set, and C represents the total number of pixel points in the target domain query set;

[0234] Step 2.2.2 According to the definition of formula (13), After standardization, we get

[0235]

[0236] Step 2.2.3 is as follows: Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks.

[0237] Step 2.2.4 and Add Gaussian noise with a variance of 0.001 to each pixel block in the target domain hyperspectral training set and

[0238] Step 2.2.5 and Integrate into two groups in the form of Represents a pixel pair consisting of a target domain hyperspectral image and a pixel label, and the spatial coordinates of the pixel pair are the same. and Indicates the corresponding true category label;

[0239] Step 2.3: Preprocess the target domain hyperspectral image training set H2 using the manually labeled target domain pixel point coordinate set and label set;

[0240] Step 2.3.1 Based on the pixel coordinate set of the manually labeled target domain, extract 5 samples of each category from the training set H2 of the target domain hyperspectral image to obtain the target domain hyperspectral training set in express The i-th pixel point in , where B2 represents the total number of pixel points in the target domain training set;

[0241] Step 2.3.2 According to the definition of formula (14), After standardization, we get 1,...,B2};

[0242]

[0243] Step 2.3.3 Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks as the center.

[0244] Step 2.3.4 Add Gaussian noise with a variance of 0.001 to each pixel block in the target domain hyperspectral training set

[0245] Step 3. Use the source domain support set, source domain query set and target domain hyperspectral feature training set to train the network N HiF Perform source domain training; use the target domain support set, target domain query set and target domain hyperspectral feature training set to train the network N HiF Perform target domain training; the source domain training and target domain training are performed alternately to obtain a trained network N HiF :

[0246] Step 3.1: As the source domain support set, As the source domain query set, the source domain support set, the source domain query set and the target domain hyperspectral training set Network N HiF Conduct source domain training;

[0247] Step 3.1.1 Use the sub-network N that extracts shallow features MSCB Support and query sets for source domain hyperspectral and LiDAR and And the target domain training set Perform feature extraction to obtain the corresponding shallow features

[0248] Step 3.1.2 Use the sub-network N that extracts deep low-frequency features LF Shallow features of hyperspectral support sets and query set shallow features And the shallow features of the target domain training set Perform low-frequency feature extraction to obtain the corresponding low-frequency features

[0249] Step 3.1.3 Use the sub-network N that extracts deep high-frequency features HF Support for hyperspectral and LiDAR shallow features and query set shallow features And the shallow features of the target domain training set Perform high-frequency feature extraction to obtain high-frequency features of the hyperspectral support set and query set as well as high-frequency features of the target domain training set

[0250] Step 3.1.4 Use the fusion sub-network N FUSE Shallow features for hyperspectral support sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain support set fusion features Shallow features for hyperspectral query sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain query set fusion features

[0251] Step 3.1.5 Using the fusion sub-network N FUSE Shallow features of target domain hyperspectral Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain the target domain hyperspectral training set fusion feature F T ;

[0252] Weighted cross entropy Calculate according to formula (15):

[0253]

[0254] Where W, D are query set fusion features Tensor, y i yes The real query sample label;

[0255] Step 3.1.6 Fusion features of source domain hyperspectral support set Fusion features with query set And the target domain hyperspectral fusion features are spliced and sent to the classification sub-network N D Perform domain classification to obtain domain discriminant features described Calculate according to formula (16);

[0256]

[0257] in is the real sample domain label, bs is the number of batch samples;

[0258] Step 3.1.7 takes the total loss function of the source domain as Network N HiF Perform source domain training, let iter→iter+1, if the number of iterations iter>Total_iter, then the trained network N is obtained HiF , go to step 4, otherwise use the back error propagation algorithm based on stochastic gradient descent and loss L s Update N HiF Parameters, and go to step 3.2, Total_iter represents the preset number of training iterations;

[0259] Step 3.2: As the target domain hyperspectral support set and query set, the target domain hyperspectral support set, target domain hyperspectral query set and target domain hyperspectral training set are used. Network N HiF Conduct training;

[0260] Step 3.2.1 Use the sub-network N that extracts shallow features MSCB Hyperspectral support set for target domain and queryset And the target domain training set Perform feature extraction to obtain the corresponding shallow features

[0261] Step 3.2.2 Use the sub-network N that extracts deep low-frequency featuresLF Hyperspectral support set and query set shallow features And the shallow features of the target domain training set Perform low-frequency extraction to obtain the corresponding low-frequency features

[0262] Step 3.2.3 Use the sub-network N that extracts deep high-frequency features HF Shallow features of hyperspectral support sets and query set shallow features And the shallow features of the target domain training set Perform high-frequency extraction to obtain the corresponding high-frequency features

[0263] Step 3.2.4 Using the fusion sub-network N FUSE Shallow features for hyperspectral support sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain support set fusion features Shallow features for hyperspectral query sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain query set fusion features

[0264] Step 3.2.5 Using the fusion sub-network N FUSE Shallow features of target domain hyperspectral Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain the target domain hyperspectral training set fusion feature F T ;

[0265] Weighted cross entropy Calculate according to formula (17):

[0266]

[0267] Where W, D are query set fusion features Tensor, y i yes The real query sample label;

[0268] Step 3.2.6: Fusing the target domain hyperspectral support set features Fusion features with query set And the target domain hyperspectral training set fusion feature F T Splice and send to the classification sub-network N D Perform domain classification and obtain domain discriminant features described Calculate according to formula (18);

[0269]

[0270] in is the real sample domain label, bs is the number of batch samples;

[0271] Step 3.2.7 takes the total loss function of the target domain as Network N HiF Perform target domain training, set iter→iter+1, if the number of iterations iter>Total_iter, then the trained network N is obtained HiF , go to step 4, otherwise use the back error propagation algorithm based on stochastic gradient descent and loss L T Update N HiF Parameters, and go to step 3.1, Total_iter represents the preset number of training iterations.

[0272] Step 4. Use the trained sub-network N HiF Treat the test set Perform feature extraction and classification through classifier:

[0273] Step 4.1: Perform data preprocessing on the hyperspectral image H' to be measured;

[0274] Step 4.1.1 Extract all pixel points from the hyperspectral image H' to be measured to form a set Where T H,i Indicates T H The i-th pixel of , U represents the total number of all pixels;

[0275] Step 4.1.2 According to the definition of formula (19), H After standardization, we get

[0276]

[0277] Step 4.1.3 Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks.

[0278] Step 4.2 Use the trained shallow feature extraction sub-network N MSCB Hyperspectral pixel block collection Perform feature extraction to obtain the corresponding shallow features

[0279] Step 4.3 Use the trained sub-network N to extract deep low-frequency features LF Shallow features Perform low-frequency extraction to obtain the corresponding low-frequency features

[0280] Step 4.4 Use the trained deep high-frequency feature extraction sub-network N HF Shallow features Perform high-frequency extraction to obtain the corresponding high-frequency features

[0281] Step 4.5 uses the trained fusion sub-network N FUSE Shallow features Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain fusion features

[0282] Step 4.6 uses KNN classifier to classify the fusion features Classification is performed to calculate the classification prediction results.

[0283] The detection results of the embodiment of the present invention (HiF2-FSLF) on the Trento dataset are compared with those of the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST, and WCPN-FSL methods, where the visual detection results are compared. Figure 4 The results of the three objective evaluation indicators OA, AA and Kappa are shown in Table 1. Figure 4 Here, the methods are the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST, WCPN-FSL and the embodiments of the present invention.

[0284] Table 1

[0285]

[0286] The detection results of the embodiment of the present invention (HiF2-FSLF) on the MUUFL dataset are compared with those of the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST, and WCPN-FSL methods, where the visual detection results are compared. Figure 5 The results of the three objective evaluation indicators OA, AA and Kappa are shown in Table 2. Figure 5 Here, the methods are the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST, WCPN-FSL and the embodiments of the present invention.

[0287] Table 2

[0288]

[0289] The detection results of the embodiment of the present invention (HiF2-FSLF) are compared with those of the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST, and WCPN-FSL methods on the Houston 2013 dataset, where the visual detection results are compared. Figure 6 The results of the three objective evaluation indicators OA, AA and Kappa are shown in Table 3. Figure 6 Here, the methods are the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST, WCPN-FSL and the embodiments of the present invention.

[0290] Table 3

[0291]

[0292] Comparison results show that the overall classification accuracy of the embodiment of the present invention on the Houston 2013, Trento and MUUFL datasets reaches 97.83%, 94.43% and 85.20% respectively, and the average accuracy is 96.52%, 92.18% and 83.11%. Compared with the existing DCFSL, FusionHCT, ADAFSL, Gia-CFSL, MHST and WCPN-FSL methods, the present invention can obtain more accurate detection results.

Claims

1. A land cover classification method based on hyperspectral and lidar layered frequency domain fusion, characterized by Follow the steps below: Step 1. Create and initialize the network N HiF , the network N HiF Contains subnetwork N for extracting shallow features MSCB , sub-network N that extracts deep low-frequency features LF , sub-network N that extracts deep high-frequency features HF , fusion sub-network N FUSE and classification subnetwork N D ; Step 2. Perform data preprocessing on the source domain hyperspectral image training set H, the source domain LiDAR image training set L, the target domain hyperspectral image training set T, the target domain hyperspectral image training set H2, and the manually annotated source domain and target domain pixel point coordinate sets and label sets to obtain the source domain support set and source domain query set, the target domain support set and query set, and the target domain hyperspectral feature training set; Step 3. Use the source domain support set, source domain query set and target domain hyperspectral feature training set to train the network N HiF Perform source domain training; use the target domain support set, target domain query set and target domain hyperspectral feature training set to train the network N HiF Perform target domain training; the source domain training and target domain training are performed alternately to obtain a trained network N HiF ; Step 4. Use the trained sub-network N HiF Treat the test set Perform feature extraction and classification through classifier.

2. The land cover classification method based on hyperspectral and lidar layered frequency domain fusion according to claim 1 is characterized in that The step 1 is specifically as follows: Step 1.1 Establish and initialize the sub-network N for extracting shallow features MSCB , the sub-network N that extracts shallow features MSCB Contains two sets of modules L arranged alternately Attention and L FeedForward ; The module L Attention By custom module L PC and L MA constitute; The custom module L PC Contains Conv 11 、Conv 12 、Conv 21 、Conv 22 , 5 groups of convolutional layers of TConv, 1 layer of activation operation, and the nonlinear activation function ReLU is selected for operation; The Conv 11 Contains one layer of convolution operation, containing 32 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel; The Conv 12 Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; The Conv 21 Contains one layer of convolution operation, containing 32 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; The Conv 22 Contains one layer of convolution operation, containing 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel; The TConv consists of one layer of transposed convolution operation, containing 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 2 pixels; The custom module L PC The operation is as follows: The input four-dimensional tensor X∈R bs×c×p×p At the same time, it is fed into Conv 11 and Conv 21 Calculate X1∈R bs×32×p×p and X3∈R bs×32×p×p , where bs represents the batch size, c represents the number of channels, and p×p represents the block size; Add X1 and X3 to get F1, and send F1 and X1 to Conv 12 and Conv 22 Calculate F2∈R bs×64×p×p , X2∈R bs ×64×p×p ; Add X2 and F2 to get F3, send F3 to TConv and ReLU for calculation, and then get F4 after one layer of activation operation. Then calculate the local attention matrix L according to formula (1): Perform matrix multiplication on the local attention matrix L and the four-dimensional tensor X to obtain the local attention feature X PC ∈R bs ×c×p×p ; The custom module L MA There are three groups of convolutional layers Conv_Q1, Conv_K1, and Conv_V1. Each of the three convolutional layers contains two layers of convolution operations. The first layer contains 192 convolution kernels of size 1×1, and the second layer is depth-wise separable convolution, which contains 192 convolution kernels of size 3×3. Both convolution operations perform convolution operations with a step size of 1 pixel. The custom module L MA The operation is as follows: (a) Input four-dimensional tensor X∈R bs×c×p×p Send it to the LayerNorm layer for normalization, and then send it to the convolutional layers Conv_Q1, Conv_K1, and Conv_V1 respectively to obtain the feature Q = {Q i |i=1,...,c}∈R c×p×p , K={K i |i=1,...,c}∈R c×p×p 、V={V i |i=1,...,c}∈R c×p×p ; (b) Use the reshape operation to map Q and K to For the i-th channel, i=1,...,c,Q i The jth position in K i The attention weight G between the l-th position in i,j,l Calculated by formula (2), j=1,...,N1, l=1,...,N1; Where N1 represents the total number of features p×p, represents the transpose of the lth element of the i-th channel of K, represents the transpose of the kth element of the ith channel of K, and then performs matrix multiplication on G and V according to formula (3) to obtain the preliminary global attention feature F MA : Where V i,l Represents the lth position element of the i-th channel; (c) Use the reshape operation to transform F MA Mapping to R bs×c×p×p , and after one layer of two-dimensional convolution operation, 64 convolution kernels of size 1×1, the global attention feature X is obtained MA ∈R bs×c×p×p ; (d) Add the global attention feature and the local attention feature to obtain the multi-scale feature X PM , the multi-scale feature X PM Send L FeedForward module; The L FeedForward The module operates as follows: (a) Multi-scale feature X PM After layer normalization, it undergoes three layers of two-dimensional convolution operations, each layer includes 64 convolution kernels of size 1×1, and operates with a step size of 1 pixel to obtain the feature (b) Yes Perform channel maximization operation and GeLU activation function to obtain channel attention feature F ch ∈R bs×1×p×p ,right Perform size maximization operation and GeLU activation function to obtain size attention feature F hw ∈R bs×4c×1×1 ,right After one layer of depth-wise separable convolution operation, including 256 convolution kernels of size 3×3 and a step size of 1 pixel, the feature F is obtained. dconv ∈R bs×4c×p×p ; (c) Feature F dconv , channel attention feature F ch and size attention feature F hw According to formula (4), the shallow feature Y∈R bs×c×p×p : Y=Conv 1_1 (F ch ·F dconv +F hw ·F dconv ) (4) Among them, Conv 1_1 Represents a convolution operation with 64 1×1 kernels and a step size of 1 pixel. The shallow feature extraction sub-network N MSCB The input four-dimensional tensor X is calculated according to formula (5) to obtain the final shallow feature X MSCB ; Step 1.2 Establish and initialize the sub-network N for extracting deep low-frequency features LF , the sub-network N that extracts deep low-frequency features LF Contains three groups of convolutional layers Conv_Q2, Conv_K2, Conv_V2 and three groups of custom convolutional layers The three groups of convolutional layers Conv_Q2, Conv_K2, and Conv_V2 each contain two layers of convolution operations. The first layer contains 192 convolution kernels of size 1×1, and the second layer is a depth-wise separable convolution, containing 192 convolution kernels of size 3×3. Both layers of convolution operations perform convolution operations with a step size of 1 pixel. described Contains 1 layer of convolution operation, containing 128 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; described Contains 1 layer of depth-wise separable convolution operation, containing 128 convolution kernels of size 3×3, and each convolution kernel performs convolution operation with a step size of 1 pixel; described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; The sub-network N that extracts deep low-frequency features LF The specific operations are as follows: (a) Input four-dimensional tensor X MSCB Send it to the LayerNorm layer for normalization, and then send it to the convolutional layers Conv_Q2, Conv_K2, and Conv_V2 respectively, to obtain the feature Q2 = {Q2 i |i=1,...,c}∈R c×p×p 、K2={K2 i |i=1,...,c}∈R c×p×p 、V2={V2 i |i=1,...,c}∈R c×p×p ; (b) Use the reshape operation to map Q2 and K2 to And calculate the low-frequency attention matrix G according to formula (6) L : in Indicates that on the i-th channel, Q2 i The j-th position element and K2 i The attention weight between the l-th position elements, N1 represents the total number of features p×p, represents the transpose of the lth element of the i-th channel of K2, represents the transpose of the kth element of the ith channel of K2, and then according to formula (7) G L Perform matrix multiplication with V2 to obtain preliminary low-frequency attention features Among them V2 i,l Represents the lth position element of the i-th channel; (c) With X MSCB Add up to get the initial deep low-frequency feature F L , normalized by the LayerNorm layer, and then sent to Get features Send it in again And through the GeLU activation function to get go through Get features and with Add up to get the final deep low-frequency feature X B ; Step 1.3 Establish and initialize the sub-network N for extracting deep high-frequency features HF , the sub-network N that extracts deep high-frequency features HF Contains three identical custom modules N FFIM ; The custom N FFIM The module contains 4 sets of custom convolutional layers described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel size performs convolution operation with a step size of 1 pixel; described Each layer contains three convolution operations. The first layer of the three convolution operations contains 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operations with a step size of 1 pixel; the second layer of depthwise convolution operations contains 64 convolution kernels of size 3×3, and each convolution kernel performs convolution operations with a step size of 1 pixel; the third layer of convolution operations contains 32 convolution kernels of size 1×1, and each convolution kernel performs convolution operations with a step size of 1 pixel; The custom N FFIM The module operation steps are as follows: (a) Input four-dimensional tensor X MSCB Send in Calculate the feature f FFIM , and along the dimension channel f FFIM Mapped into two tensors (b)f 1,1 and f 2,1 According to formula (8), the shallow high-frequency feature f is calculated H : Where CAT(,) represents the splicing operation; (c) Shallow high-frequency features f H Loop through the third N FFIM The module obtains deep high-frequency features X D ; Step 1.5: Create and initialize the fusion sub-network N FUSE , contains six fully connected layers and two convolutional layers The six fully connected layers are Linear1, Linear2, Linear3, Linear4, Linear5, and Linear6; described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; described Contains one layer of convolution operation, containing 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; The fusion sub-network N FUSE The specific operations are as follows: (a) The deep low-frequency feature X B Input to Linear1 and Linear2 to get features The shallow feature X MSCB Input to Linear3 and Linear4 to get features The deep high-frequency feature X D Input to Linear5 and Linear6 to get features (b) Use the reshape operation to transform Q 1 , K 1 , K 2 , Q 2 Map to And calculate the low-frequency attention matrix AM1 and the high-frequency attention matrix AM2 according to formula (9); in Indicates that on the i-th channel, The j-th position element of The attention weight between the l-th position elements, N1 represents the total number of features p×p, Indicates that on the i-th channel, The j-th position element of The attention weight between the l-th position elements, K 1 The transpose of the lth element of the ith channel of K 2 The transpose of the lth channel of the ith element of K 1 The transpose of the kth element of the ith channel of K 2 The transpose of the k-th channel of the i-th element of ; (c) Use the reshape operation to transform V 1 、V 2 Mapping to R c×(p×p) , and then according to formula (10) AM1 and V 1 , AM2 and V 2 Perform matrix multiplication to obtain in W 1 The vector composed of the lth position element of the ith channel, Indicates V 2 The vector composed of the lth position element of the ith channel; (d) Calculate the fusion feature X according to formula (11) Fuse ; in and Represents a convolution operation, which contains 64 convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; Step 1.6: Create and initialize the classification subnetwork N D , contains a linear layer Linear F , Linear F Contains 1 output unit and 4 groups of fully connected layers FC1, FC2, FC3, FC4; FC1, FC2, FC3, and FC4 each contain a linear layer, a ReLU activation function, and a Dropout operation; X Fuse Enter FC1, FC2, FC3, FC4 in sequence, and then enter Linear F Get the domain classification feature X domain .

3. The land cover classification method based on hyperspectral and lidar layered frequency domain fusion according to claim 2 is characterized in that The step 2 is specifically as follows: Step 2.1: Based on the manually labeled source domain pixel point coordinate set and label set, perform data preprocessing on the source domain hyperspectral image training set H and the source domain LiDAR image training set L; Step 2.1.1 Based on the manually labeled source domain pixel point coordinate set, extract 1 and 19 labeled pixel points of each class from the source domain hyperspectral image training set H as the source domain hyperspectral support set. and queryset And extract 1 and 19 labeled pixel points of each class from the training set L of the source domain LiDAR image as the source domain LiDAR support set and queryset in express The i-th pixel point in express The i-th pixel point in express The i-th pixel point in express The i-th pixel point in , N represents the total number of pixel points in the source domain support set, and M represents the total number of pixel points in the source domain query set; Step 2.1.

2. According to the definition of formula (12), After standardization, we can obtain and Step 2.1.3 is as follows: Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks, which are Step 2.1.4 and Integrate into triples in the form of They represent the pixel pairs composed of the source domain hyperspectral image and the LiDAR image, and the spatial coordinates of the pixel pairs are the same. and Indicates the corresponding true category label; Step 2.2: Preprocess the target domain hyperspectral image training set T using the manually labeled target domain pixel point coordinate set and label set; Step 2.2.1 Based on the pixel coordinate set of the manually labeled target domain, extract 1 and 19 labeled pixel points of each class from the training set T of the target domain hyperspectral image as the target domain hyperspectral support set. and queryset in express The i-th pixel point in express The i-th pixel point in , B represents the total number of pixel points in the target domain support set, and C represents the total number of pixel points in the target domain query set; Step 2.2.2 According to the definition of formula (13), After standardization, we get Step 2.2.3 is as follows: Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks. Step 2.2.4 and Gaussian noise is added to each pixel block in the target domain to obtain the hyperspectral training set and Step 2.2.5 and Integrate into two groups in the form of Represents a pixel pair consisting of a target domain hyperspectral image and a pixel label, and the spatial coordinates of the pixel pair are the same. and Indicates the corresponding true category label; Step 2.3: Preprocess the target domain hyperspectral image training set H2 using the manually labeled target domain pixel point coordinate set and label set; Step 2.3.1 Based on the pixel coordinate set of the manually labeled target domain, extract 5 samples of each category from the target domain hyperspectral image training set H2 to obtain the target domain hyperspectral training set in express The i-th pixel point in , where B2 represents the total number of pixel points in the target domain training set; Step 2.3.2 According to the definition of formula (14), After standardization, we get Step 2.3.3 Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks as the center. Step 2.3.4 Gaussian noise is added to each pixel block in the target domain to obtain the hyperspectral training set 4. The land cover classification method of hyperspectral and lidar layered frequency domain fusion according to claim 3 is characterized in that The step 3 is specifically as follows: Step 3.1: As the source domain support set, As the source domain query set, the source domain support set, the source domain query set and the target domain hyperspectral training set Network N HiF Enter Perform source domain training; Step 3.1.1 Use the sub-network N that extracts shallow features MSCB Support and query sets for source domain hyperspectral and LiDAR and And the target domain training set Perform feature extraction to obtain the corresponding shallow features Step 3.1.2 Use the sub-network N that extracts deep low-frequency features LF Shallow features of hyperspectral support sets and query set shallow features And the shallow features of the target domain training set Perform low-frequency feature extraction to obtain the corresponding low-frequency features Step 3.1.3 Use the sub-network N that extracts deep high-frequency features HF Support for hyperspectral and LiDAR shallow features and query set shallow features And the shallow features of the target domain training set Perform high-frequency feature extraction to obtain high-frequency features of the hyperspectral support set and query set as well as high-frequency features of the target domain training set Step 3.1.4 Use the fusion sub-network N FUSE Shallow features for hyperspectral support sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain support set fusion features Shallow features for hyperspectral query sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain query set fusion features Step 3.1.5 Using the fusion sub-network N FUSE Shallow features of target domain hyperspectral Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain the target domain hyperspectral training set fusion feature F T ; Weighted cross entropy Calculate according to formula (15): Where W, D are query set fusion features Tensor, y i yes The real query sample label; Step 3.1.6 Fusion features of source domain hyperspectral support set Fusion features with query set And the target domain hyperspectral fusion features are spliced and sent to the classification sub-network N D Perform domain classification to obtain domain discriminant features described Calculate according to formula (16); in is the real sample domain label, bs is the number of batch samples; Step 3.1.7 takes the total loss function of the source domain as Network N HiF Perform source domain training, let iter→iter+1, if the number of iterations iter>Total_iter, then the trained network N is obtained HiF , go to step 4, otherwise use the back error propagation algorithm based on stochastic gradient descent and loss L s Update N HiF Parameters, and go to step 3.2, Total_iter represents the preset number of training iterations; Step 3.2: As the target domain hyperspectral support set and query set, the target domain hyperspectral support set, target domain hyperspectral query set and target domain hyperspectral training set are used. Network N HiF Conduct training; Step 3.2.1 Use the sub-network N that extracts shallow features MSCB Hyperspectral support set for target domain and queryset And the target domain training set Perform feature extraction to obtain the corresponding shallow features Step 3.2.2 Use the sub-network N that extracts deep low-frequency features LF Hyperspectral support set and query set shallow features And the shallow features of the target domain training set Perform low-frequency extraction to obtain the corresponding low-frequency features Step 3.2.3 Use the sub-network N that extracts deep high-frequency features HF Shallow features of hyperspectral support sets and query set shallow features And the shallow features of the target domain training set Perform high-frequency extraction to obtain the corresponding high-frequency features Step 3.2.4 Using the fusion sub-network N FUSE Shallow features for hyperspectral support sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain support set fusion features Shallow features for hyperspectral query sets Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain query set fusion features Step 3.2.5 Using the fusion sub-network N FUSE Shallow features of target domain hyperspectral Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain the target domain hyperspectral training set fusion feature F T ; Weighted cross entropy Calculate according to formula (17): Where W, D are query set fusion features Tensor, y i yes The real query sample label; Step 3.2.6: Fusing the target domain hyperspectral support set features Fusion features with query set And the target domain hyperspectral training set fusion feature F T Splice and send to the classification sub-network N D Perform domain classification and obtain domain discriminant features described Calculate according to formula (18); in is the real sample domain label, bs is the number of batch samples; Step 3.2.7 takes the total loss function of the target domain as Network N HiF Perform target domain training, set iter→iter+1, if the number of iterations iter>Total_iter, then the trained network N is obtained HiF , go to step 4, otherwise use the back error propagation algorithm based on stochastic gradient descent and loss L T Update N HiF Parameters, and go to step 3.1, Total_iter represents the preset number of training iterations.

5. The land cover classification method based on hyperspectral and laser radar layered frequency domain fusion according to claim 4 is characterized in that The step 4 is specifically as follows: Step 4.1: Perform data preprocessing on the hyperspectral image H' to be measured; Step 4.1.1 Extract all pixel points from the hyperspectral image H' to be measured to form a set T H ={T H,i |i=1,...,U}, where T H,i Indicates T H The i-th pixel of , U represents the total number of all pixels; Step 4.1.2 According to the definition of formula (19), H After standardization, we get Step 4.1.3 Each pixel point is divided into a series of 9×9 hyperspectral pixel blocks. Step 4.2 Use the trained shallow feature extraction sub-network N MSCB Hyperspectral pixel block collection Perform feature extraction to obtain the corresponding shallow features Step 4.3 Use the trained sub-network N to extract deep low-frequency features LF Shallow features Perform low-frequency extraction to obtain the corresponding low-frequency features Step 4.4 Use the trained deep high-frequency feature extraction sub-network N HF Shallow features Perform high-frequency extraction to obtain the corresponding high-frequency features Step 4.5 uses the trained fusion sub-network N FUSE Shallow features Low-frequency characteristics and high-frequency features Perform feature-level fusion to obtain fusion features Step 4.6 uses KNN classifier to classify the fusion features Classification is performed to calculate the classification prediction results.

Images