A multimodal remote sensing data classification method integrating global and local information

Abstract

The present invention discloses a multimodal remote sensing data classification method that fuses global and local information, belonging to the field of hyperspectral image processing technology. The method comprises obtaining a multimodal remote sensing data set to be classified and inputting the data into a pre-trained multimodal remote sensing data classification model; the multimodal remote sensing data classification model outputs a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified. The present invention solves the problems in the prior art of poor interpretability of multimodal convolution, insufficient cross-modal feature extraction, and difficulty in capturing global information by context-independent convolution structures. The method decomposes multimodal information into common information and unique information through feature decoupling, realizes multimodal information fusion, fully exploits the features of multimodal remote sensing data, and significantly reduces the demand for training data for deep learning models in the field of remote sensing images, thereby achieving more accurate image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image processing, and in particular to a multimodal remote sensing data classification method that integrates global and local information. Background Art

[0002] Hyperspectral imagery can capture rich spatial and spectral details of ground objects. Therefore, its application in land cover classification plays an important role in numerous fields, including geological exploration, urban development planning, forest management, and precision agriculture. LiDAR data not only records the elevation of observed objects but also has the advantage of being less susceptible to interference from environmental factors such as weather and occlusion. Due to differences in their imaging mechanisms, hyperspectral and LiDAR data can be used to capture different aspects of ground objects, and their complementary information can improve the performance of land cover classification tasks. Therefore, the fusion of hyperspectral and LiDAR data can help achieve more accurate image classification, making it a hot research topic. However, capturing complex local and global spatial-spectral correlations is difficult; at the same time, establishing effective interaction between multimodal data is another important issue.

[0003] Multimodal image fusion classification primarily combines the characteristics of each modality to achieve remote sensing image classification tasks, such as identifying physically meaningful highlights and textures. However, multimodal features are currently extracted primarily through convolution, but convolution suffers from poor interpretability and controllability, is inadequate for cross-modal feature extraction, and struggles to extract modality-specific features. Furthermore, context-independent convolutional structures can only extract local information within a relatively small receptive field, making it difficult to capture global information. Furthermore, the network's forward propagation can cause high-frequency information to be lost. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multimodal remote sensing data classification method that integrates global and local information, so as to solve the problems in the prior art of multimodal convolution, such as poor interpretability, difficulty in control, insufficient cross-modal feature extraction, and difficulty in capturing global information by context-independent convolutional structures.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a multimodal remote sensing data classification method that integrates global and local information, comprising:

[0007] Obtain a multimodal remote sensing data set to be classified and input it into a pre-trained multimodal remote sensing data classification model;

[0008] The multimodal remote sensing data classification model outputs a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified;

[0009] The training method of the multimodal remote sensing data classification model includes:

[0010] Obtain original hyperspectral remote sensing data and LiDAR data, perform dimensionality reduction processing on the original hyperspectral remote sensing data, and obtain low-dimensional hyperspectral features;

[0011] Based on the determined number of training samples, randomly select training sets and test sets from low-dimensional hyperspectral features and LiDAR data, build an initial multimodal remote sensing data classification model, and set the number of iterations and parameters of the multimodal remote sensing data classification model;

[0012] Perform multi-scale information fusion of low-dimensional hyperspectral features and LiDAR data to obtain low-dimensional hyperspectral shallow features and LiDAR data shallow features;

[0013] Through the dual-branch Transformer-CNN feature extractor, the low-dimensional hyperspectral shallow features and the LiDAR data shallow features are respectively subjected to global information extraction and local information extraction, thereby obtaining the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data.

[0014] The ECA module is used to fuse the low-frequency global features and high-frequency local features of low-dimensional hyperspectral data with the low-frequency global features and high-frequency local features of LiDAR data. The training set is used to cyclically train the initial multimodal remote sensing data classification model. When the maximum number of iterations is reached, the final multimodal remote sensing data classification model is obtained.

[0015] In combination with the first aspect, further, the dual-branch Transformer-CNN feature extractor includes a SIM-Transformer module and an INN module, wherein the SIM-Transformer module is used to use structural information modeling to learn and process low-frequency global features, and the INN module is used to extract high-frequency local features.

[0016] In combination with the first aspect, further, the multi-scale information fusion of the low-dimensional hyperspectral features and the LiDAR data to obtain the low-dimensional hyperspectral shallow features and the LiDAR data shallow features includes:

[0017] The hyperspectral remote sensing image H and LiDAR L are respectively subjected to 3×3, 5×5 and 7×7 convolution operations to obtain multi-scale information H 3×3 、H 5×5 、H 7×7 , L 3×3 , L 5×5 and L 7×7 ; According to the channel dimension, fuse the three scale features, superimpose the input corresponding to the j-th dimension, and obtain the superimposed feature The formula is:

[0018]

[0019] Where, F c is the multi-scale spatial spectrum feature, k is the feature dimension, Concat is the channel superposition, i is the multi-scale operation, is the superposition feature, j is the number of dimensions;

[0020] right The channel dimension of performs element-wise addition, element-wise averaging, and element-wise maximization to generate three feature maps representing different processing methods and properties. and The formula is:

[0021]

[0022]

[0023]

[0024] Where, For The channel dimension performs element-wise maximization on the generated feature map, For The channel dimension performs element-wise averaging to generate the feature map, For The feature map generated by performing element-wise addition on the channel dimension of

[0025] Will and The multi-scale features are further fused through 3×3 convolution. The formula is:

[0026]

[0027] Where, F j For the general and Features that are superimposed and fused through convolution;

[0028] The feature maps of k groups of three scales based on multi-scale are cascaded to obtain the final low-dimensional shallow features with dimension k. The formula is:

[0029]

[0030] Where H1 is a low-dimensional hyperspectral shallow feature with dimension k.

[0031] In combination with the first aspect, further, the dual-branch Transformer-CNN feature extractor is used to extract global information and local information from the shallow features of the low-dimensional hyperspectral data and the shallow features of the LiDAR data, respectively, to obtain low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and low-frequency global features and high-frequency local features of the LiDAR data, including:

[0032] The acquired shallow features are projected into the D-dimensional embedding through linear mapping, and the learnable cls embedding token is introduced and placed at the beginning of the input sequence embedding. The position information embedding is added to the input sequence embedding to obtain the input of the first layer SIM-Transformer. Then:

[0033]

[0034] Where z0 is the input of the first layer SIM-Transformer, x cls is the learnable cls embedding token, F(·) is a linear mapping, E p To embed location information, is the Nth position token of the patch in the shallow feature H1, where N is the number of patches;

[0035] Then the formula of the lth SIM-Transformer is:

[0036] z′ l =LN(MSA(z l-1 )+z l-1 )

[0037] z l =LN(FFN(z′ l )+z′ l );

[0038] Where LN(·) is layer normalization, FFN is feedforward neural network, MSA is multi-head attention mechanism, z l is the lth SIM-Transformer, z′ l is the intermediate feature of the lth SIM-Transformer.

[0039] Structural information learning is introduced to incorporate object space context information into the Transformer. Assuming that the Transformer layer has H layers, Q and K are the dimension query and key vectors of all tokens in D dimensions. The specific formula is as follows:

[0040]

[0041] Where, h=1, 2, ..., H and N=NH ×N W is the number of patches, D is the dimension, Q is the dimension query of all tokens in D dimension, K is the key vector of all tokens in D dimension, softmax(.) is the activation function, normalizes the tokens, K T Expressed as the transpose of K;

[0042] Extract the attention weights between the patch-token and the cls token of each head, expressed as The corresponding total attention weight is calculated as:

[0043]

[0044] Where A is the corresponding total attention weight, is the attention weight between the patch to token and the cls token of each head;

[0045] To filter unimportant patches, set As the threshold, the new attention weight is:

[0046]

[0047] Where, is the new attention weight, A (x，y) is the relationship weight between the patch token and the cls token at position (x, y), is the threshold.

[0048] The patch with the highest attention weight is considered the most discriminative patch and is set as the reference patch. and the horizontal reference direction, patchP x，y The polar coordinates of are defined as:

[0049]

[0050]

[0051] Where, ρ x，y It is P0 and P x，y The relative distance, θ x，y is the normalized polar angle relative to the horizontal, N H is the length of the patch, N W is the width of the patch, (x0, y0) is N H ×N W The index of the plane, x is N H ×N W The horizontal coordinate position of the plane, y is N H ×N W The ordinate position of the plane;

[0052] Therefore, the structural features are obtained through two layers of graph convolution:

[0053] S=σ(Adj×σ(Adj×X×W 1 )×W 2 );

[0054] Where S is the structural feature, W 1 and W 2 is a learnable parameter, σ is the Sigmod activation function, Adj = A new ×(A new ) T Based on A new The edge weight between the nodes, X is the patch node feature;

[0055] Using INN with Affine Coupling layer, the changes of each reversible layer are:

[0056]

[0057]

[0058]

[0059] Where ⊙ is the Hadamard product, is the 1st to cth channels of the input features of the kth reversible layer, k = 1, ..., K, ψ i is an arbitrary mapping function that does not affect the lossless information transmission of the reversible layer, i = 1, ..., 3, and C is the number of channels of local features.

[0060] In combination with the first aspect, further, the fusing of the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data through the ECA module includes:

[0061] The high-frequency local features H of hyperspectral remote sensing images are fused through the ECA module. h and LiDAR’s high-frequency local features L h , get the high-frequency fusion feature F h ;

[0062] Fusion of low-frequency global features H of hyperspectral remote sensing images l and LiDAR’s low-frequency global features L l , get the low-frequency fusion feature F l ;

[0063] The high-frequency fusion feature F h Fusion with low-frequency features F l Fusion into the final feature Fend ;

[0064] The final feature F is made by the ECA module end Realize cross-channel interactive fusion of high-frequency and low-frequency information.

[0065] In combination with the first aspect, further, the final feature F end The formula is:

[0066] F end =Concat(F h , F l );

[0067] Where, F h is the high-frequency fusion feature, F l is the low-frequency fusion feature, F end is the high-frequency fusion feature F h Fusion with low-frequency features F l The final features after fusion;

[0068] The final feature F is obtained by the ECA module. end The formula for achieving cross-channel interactive fusion of high-frequency and low-frequency information is:

[0069] ω=ρ(C1D k (F end ));

[0070] Where ω is the high-frequency-low-frequency fusion feature, ρ is the Sigmod activation function, and C1D is the 1D convolution.

[0071] In a second aspect, the present invention provides a multimodal remote sensing data classification device that integrates global and local information, comprising:

[0072] The data acquisition module is used to obtain the multimodal remote sensing data set to be classified and input it into the pre-trained multimodal remote sensing data classification model;

[0073] An output result module, configured for the multimodal remote sensing data classification model to output a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified;

[0074] The training method of the multimodal remote sensing data classification model includes:

[0075] Obtain original hyperspectral remote sensing data and LiDAR data, perform dimensionality reduction processing on the original hyperspectral remote sensing data, and obtain low-dimensional hyperspectral features;

[0076] Based on the determined number of training samples, randomly select training sets and test sets from low-dimensional hyperspectral features and LiDAR data, build an initial multimodal remote sensing data classification model, and set the number of iterations and parameters of the multimodal remote sensing data classification model;

[0077] Perform multi-scale information fusion of low-dimensional hyperspectral features and LiDAR data to obtain low-dimensional hyperspectral shallow features and LiDAR data shallow features;

[0078] Through the dual-branch Transformer-CNN feature extractor, the low-dimensional hyperspectral shallow features and the LiDAR data shallow features are respectively subjected to global information extraction and local information extraction, thereby obtaining the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data.

[0079] The ECA module is used to fuse the low-frequency global features and high-frequency local features of low-dimensional hyperspectral data with the low-frequency global features and high-frequency local features of LiDAR data. The training set is used to cyclically train the initial multimodal remote sensing data classification model. When the maximum number of iterations is reached, the final multimodal remote sensing data classification model is obtained.

[0080] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multimodal remote sensing data classification method for fusing global and local information as described in any one of the first aspects is implemented.

[0081] In a fourth aspect, the present invention provides a device comprising:

[0082] a memory for storing instructions;

[0083] A processor is used to execute the instructions so that the device performs operations to implement the multimodal remote sensing data classification method for fusing global and local information as described in any one of the first aspects.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] The present invention discloses a multimodal remote sensing data classification method that integrates global and local information. The method decomposes multimodal information into common information and unique information by means of feature decoupling, realizes multimodal information fusion, fully mines the features of multimodal remote sensing data, trains an optimal multimodal remote sensing data classification model, and then inputs the acquired multimodal remote sensing data set to be classified into a pre-trained multimodal remote sensing data classification model. The multimodal remote sensing data classification model outputs a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified, thereby providing effective feature support for a deep learning classification model and achieving more accurate image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a process of a multimodal remote sensing data classification method that integrates global and local information provided by the first embodiment of the present invention. Figure 1 ;

[0087] Figure 2 This is a process of a multimodal remote sensing data classification method that integrates global and local information provided by the first embodiment of the present invention. Figure 2 ;

[0088] Figure 3 The Houston 2013 hyperspectral image provided in the second embodiment of the present invention;

[0089] Figure 4 The Houston 2013 LiDAR image provided in Example 2 of the present invention;

[0090] Figure 5 This is a schematic diagram of real ground object types in the Houston 2013 hyperspectral image provided by the second embodiment of the present invention;

[0091] Figure 6 Schematic diagram of the classification results of the model under the sample selected by HRWN in the comparative example provided in the second embodiment of the present invention;

[0092] Figure 7 Schematic diagram of the classification results of the model under the sample selected by EndNet in the comparative example provided in the second embodiment of the present invention;

[0093] Figure 8 Schematic diagram of the classification results of the model under the sample selected by AM3Net, a comparative example provided in the second embodiment of the present invention;

[0094] Figure 9 Schematic diagram of the classification results of the model under the sample selected by the comparative example MFT provided in the second embodiment of the present invention;

[0095] Figure 10 2 is a schematic diagram of the classification results of the model under the sample selected by MAHiDFNet, a comparative example provided in the second embodiment of the present invention;

[0096] Figure 11 It is a schematic diagram of the classification results of the model under the samples selected by the comparative example CGLFN provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0097] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. Unless there is a conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0098] The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0099] Example 1:

[0100] like Figures 1 and 2 As shown, this embodiment provides a multimodal remote sensing data classification method that integrates global and local information, including the following steps:

[0101] Step 1: Obtain original hyperspectral remote sensing data and LiDAR data, use PCA to reduce the dimensionality of the original hyperspectral remote sensing data to obtain low-dimensional hyperspectral features;

[0102] Step 2: Based on the determined number of training samples and the ground truth of the hyperspectral remote sensing image, randomly select the training set X and the test set Y from the low-dimensional hyperspectral features and LiDAR data, build the initial multimodal remote sensing data classification model, and set the maximum number of iterations λ and parameter σ of the multimodal remote sensing data classification model;

[0103] Step 3: The low-dimensional hyperspectral feature H and LiDAR data shallow feature L1 are obtained by multi-scale fusion module to perform multi-scale information fusion on the low-dimensional hyperspectral feature H and LiDAR data shallow feature L1.

[0104] The hyperspectral remote sensing image H and LiDAR L are respectively subjected to 3×3, 5×5 and 7×7 convolution operations to obtain multi-scale information H 3×3 、H 5×5 、H 7×7 , L 3×3 , L 5×5 and L 7×7 At the same time, there is similarity between adjacent channels of features, and the spectral dimension is continuous. Taking the hyperspectral remote sensing image H as an example, according to the channel dimension, the three scale features are fused and the j-th dimension corresponding to the input is superimposed to obtain the superimposed feature The formula is:

[0105]

[0106] Where, F cis the multi-scale spatial spectrum feature, k is the feature dimension, Concat is the channel superposition, i is the multi-scale operation, is the superposition feature, and j is the number of dimensions.

[0107] right The channel dimension of performs element-wise addition, element-wise averaging, and element-wise maximization to generate three feature maps representing different processing methods and properties. and In this way, the channel characteristics are further emphasized, and the specific formula is as follows:

[0108]

[0109]

[0110]

[0111] Where, For The channel dimension performs element-wise maximization on the generated feature map, For The channel dimension performs element-wise averaging to generate the feature map, For The feature map generated by performing element-wise addition of the channel dimension.

[0112] Will and The multi-scale features are further fused through 3×3 convolution. The specific formula is as follows:

[0113]

[0114] Where, F j For the general and Features are superimposed and fused via convolution.

[0115] The feature maps of k groups of three scales based on multi-scale are cascaded to obtain the final low-dimensional shallow feature H1 with dimension k. The specific formula is:

[0116]

[0117] Where H1 is a low-dimensional hyperspectral shallow feature with dimension k.

[0118] Similarly, the above process is used to obtain the low-dimensional LiDAR shallow feature L1 with dimension k.

[0119] Step 4: Through the dual-branch Transformer-CNN feature extractor, the low-dimensional hyperspectral shallow feature H1 and the LiDAR data shallow feature L1 are simultaneously input into the SIM-Transformer module and the Invertible Neural Network (INN) module for learning, respectively performing global information extraction and local information extraction to obtain the high-frequency local feature H of the hyperspectral remote sensing image. h and low-frequency global features H l , and the high-frequency local features L of LiDAR h and low-frequency global features L l ;

[0120] The specific steps include:

[0121] The obtained shallow feature H1 is projected into the D-dimensional embedding through the linear mapping F(·). Introducing the learnable cls embedding token x cls , and put it at the beginning of the input sequence embedding. At the same time, the position information is embedded in E p Add to the input sequence embedding to obtain z0 as the input of the first layer of SIM-Transformer, then:

[0122]

[0123] Where z0 is the input of the first layer SIM-Transformer, x cls is the learnable cls embedding token, F(·) is a linear mapping, E p To embed location information, is the Nth position token of the patch in the shallow feature H1, where N is the number of patches.

[0124] The SIM-Transformer feature extractor has L layers of SIM-Transformers. Each layer of SIM-Transformer consists of a multi-head self-attention module and a fully connected layer of feedforward neural network. The l-th SIM-Transformer is calculated as follows:

[0125] z′ l =LN(MSA(z l-1 )+z l-1 )

[0126] z l =LN(FFN(z′ l )+z′ l );

[0127] Where LN(·) is layer normalization, FFN is feedforward neural network, MSA is multi-head attention mechanism, z′ lis the intermediate feature of the lth SIM-Transformer, z l is the lth SIM-Transformer.

[0128] Although the Transformer can achieve good classification results because it has a global receptive field using the self-attention mechanism, it tends to ignore the spatial relationship between patches, which is not conducive to obtaining discriminative patches. Therefore, we introduce structural information learning to incorporate object space context information into the Transformer. Assuming that the Transformer layer has H layers, Q and K are the dimensional query and key vectors of all tokens in D dimensions. The specific formula is as follows:

[0129]

[0130] Where, h=1, 2, ..., H and N=N H ×N W is the number of patches, D is the dimension, Q is the dimension query of all tokens in D dimension, K is the key vector of all tokens in D dimension, softmax(.) is the activation function, normalizes the tokens, K T Expressed as the transpose of K.

[0131] Extract the attention weights between the patch-token and the cls token of each head, expressed as The corresponding total attention weight is calculated as follows:

[0132]

[0133] Where A is the corresponding total attention weight, is the attention weight between the patch-token and the cls token of each head.

[0134] The weight is the relationship between the patch token and the cls token at the (x, y) position, denoted as A (x，y) . To filter unimportant patches, set As the threshold, the new attention weight is as follows:

[0135]

[0136] Where, is the new attention weight, A (x，y) is the relationship weight between the patch token and the cls token at position (x, y), is the threshold.

[0137] The patch with the highest attention weight is considered the most discriminative patch and is set as the reference patch. and the horizontal reference direction, where (x0, y0) is N H ×N W The index of the plane, patchP x，y The polar coordinates of are defined as:

[0138]

[0139]

[0140] Where, ρ x，y It is P0 and P x，y The relative distance, θ x，y is the normalized polar angle relative to the horizontal, N H is the length of the patch, N W is the width of the patch, (x0, y0) is N H ×N W The index of the plane, x is N H ×N W The horizontal coordinate position of the plane, y is N H ×N W The vertical coordinate position of the plane.

[0141] Therefore, the structural features are obtained through two layers of graph convolution:

[0142] S=σ(Adj×σ(Adj×X×W 1 )×W 2 );

[0143] Where S is the structural feature, W 1 and W 2 is a learnable parameter, σ is the Sigmod activation function, Adj = A new ×(A new ) T Based on A new The edge weights between nodes are , and X is the patch node feature.

[0144] In contrast to SIM-Transformer, edge texture information is also important in classification tasks.

[0145] Therefore, INN ensures that the input information is preserved as much as possible by enabling the input and output to generate each other. The specific process is to use INN with Affine Coupling layers. The changes in each reversible layer are as follows:

[0146]

[0147]

[0148]

[0149] Where ⊙ is the Hadamard product, is the 1st to cth channels of the input features of the kth reversible layer, k = 1, ..., K, ψ i is an arbitrary mapping function that does not affect the lossless information transmission of the reversible layer, i = 1, ..., 3, and C is the number of channels of local features.

[0150] Considering the trade-off between computational consumption and feature extraction capability, bottleneck residual block (BRB) is used as ψ i .

[0151] Step 5: Fuse the high-frequency local features H of the hyperspectral remote sensing image through the ECA (Efficient Channel Attention) module h and LiDAR’s high-frequency local features L h , get the high-frequency fusion feature F h , while fusing the low-frequency global features H of the hyperspectral remote sensing image l and LiDAR’s low-frequency global features L l , get the low-frequency fusion feature F l ,Then through the ECA module, the high-frequency and low-frequency information are interactively fused across channels;

[0152] In this step, the high frequency fusion feature F h Fusion with low-frequency features F l Fusion into the final feature F end , the specific formula is:

[0153] F end =Concat(F h , F l );

[0154] Where, F end is the high-frequency fusion feature F h Fusion with low-frequency features F l The final features after fusion.

[0155] Final feature F end The ECA module is used to realize cross-channel interaction between high-frequency and low-frequency information. The specific formula is:

[0156] ω=ρ(C1D k (F end ));

[0157] Where ω is the high-frequency-low-frequency fusion feature, ρ is the Sigmod activation function, and C1D is the 1D convolution.

[0158] Step 6: Repeat steps 3 to 5 to update the multimodal remote sensing data classification model and the parameters of the multimodal remote sensing data classification model;

[0159] Step 7: Use the training set X to cyclically train the initial multimodal remote sensing data classification model. When the number of iterations is 1, obtain the final multimodal remote sensing data classification model, and use the final multimodal remote sensing data classification model to perform final classification on the multimodal remote sensing data set to be classified to obtain the classification result.

[0160] Example 2:

[0161] This example uses the Houston2013 dataset, which was acquired by the National Airborne Laser Mapping Center in June 2012 at the University of Houston campus and nearby urban areas, to obtain hyperspectral and DSM-based LiDSM data for the area. This dataset was used as the dataset for the 2013 GRSS Data Fusion Competition. Figure 3 Displays hyperspectral images from the Houston 2013 dataset, with 144 bands ranging from 0.38 to 1.05 nanometers. Figure 4 The DSM-based LiDAR of the Houston2013 dataset is presented, with a spatial size of 349×1905 pixels and a ground sampling distance of 2.5 meters. Figure 5 The real ground object types are displayed, including 15 ground object categories, and 15029 samples are annotated as ground truth. The multimodal fusion classification comparison experiment uses hierarchical random walk network (HRWN), EndNet, adaptive mutual learning network (AM 3 Net), multimodal fusion Transformer (MFT), multi-attention hierarchical fusion network (MAHiDFNet) and the sampling method (CGLFN) of the present invention are used to perform image classification on the hyperspectral and LiDAR data fusion of the Houston2013 dataset in the embodiment.

[0162] The network learning settings are as follows: 20 samples from each of the 15 categories are selected as the training set D tr , the remaining samples are used as the test set D te The number of iterations I was set to 80. Under this condition, 10 repeated experiments were conducted and the average value was calculated. The classification accuracy is shown in Table 1.

[0163] Table 1 Comparison of image classification accuracy of Houston2013 dataset

[0164]

[0165] The Houton2013 dataset contains 15 types of ground objects, and some areas are obscured by clouds and fog, resulting in a high level of scene complexity. As can be seen from Table 1, the attention mechanism has a positive impact on feature-level fusion. For example, MAHiDFNet and MFT perform better than decision-level fusion methods. A suitable decision-level fusion strategy is better than EndNet, which only extracts one deep feature, which shows the importance of extracting more target features. In contrast, the present invention achieved the best classification results. The present invention mainly extracts feature information of each modality in a hierarchical manner, such as physically meaningful highlight areas and texture details, as well as global semantic information, to effectively screen out information-rich modal features, and interactively fuse the extracted multimodal features to provide high-quality sample features for the deep learning classification model, thereby improving classification performance.

[0166] To visualize the classification results, Figures 6 to 10 HRWN, EndNet, and AM were demonstrated respectively. 3 Classification results of Net, MFT and MAHiDFNet, Figure 11 The classification result diagram of the present invention is shown. It can be seen intuitively that under the common features and unique features of the selected multimodal data of the present invention, the classification model can accurately identify the ground feature category to which the sample belongs.

[0167] The above demonstrates the feasibility of the present invention in multimodal data fusion classification. This invention significantly reduces the training data requirements for deep learning models in the field of remote sensing imagery. This allows deep learning models to achieve excellent performance in extracting both common and unique features from multimodal data, even with minimal training data, effectively improving the accuracy of hyperspectral image classification.

[0168] Example 3:

[0169] Based on the same inventive concept as the first embodiment, this embodiment introduces a multimodal remote sensing data classification device that integrates global and local information, including:

[0170] The data acquisition module is used to obtain the multimodal remote sensing data set to be classified and input it into the pre-trained multimodal remote sensing data classification model;

[0171] An output result module, configured for the multimodal remote sensing data classification model to output a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified;

[0172] The training method of the multimodal remote sensing data classification model includes:

[0173] Obtain original hyperspectral remote sensing data and LiDAR data, perform dimensionality reduction processing on the original hyperspectral remote sensing data, and obtain low-dimensional hyperspectral features;

[0174] Based on the determined number of training samples, randomly select training sets and test sets from low-dimensional hyperspectral features and LiDAR data, build an initial multimodal remote sensing data classification model, and set the number of iterations and parameters of the multimodal remote sensing data classification model;

[0175] Perform multi-scale information fusion of low-dimensional hyperspectral features and LiDAR data to obtain low-dimensional hyperspectral shallow features and LiDAR data shallow features;

[0176] Through the dual-branch Transformer-CNN feature extractor, the low-dimensional hyperspectral shallow features and the LiDAR data shallow features are respectively subjected to global information extraction and local information extraction, thereby obtaining the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data.

[0177] The ECA module is used to fuse the low-frequency global features and high-frequency local features of low-dimensional hyperspectral data with the low-frequency global features and high-frequency local features of LiDAR data. The training set is used to cyclically train the initial multimodal remote sensing data classification model. When the maximum number of iterations is reached, the final multimodal remote sensing data classification model is obtained.

[0178] The specific functional implementation of each of the above modules can be found in the relevant content of the method in Example 1 and will not be elaborated on here.

[0179] Example 4:

[0180] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the multimodal remote sensing data classification method that integrates global and local information as described in any one of the first embodiments.

[0181] Embodiment 5:

[0182] Based on the same inventive concept as other embodiments, this embodiment introduces a device, including:

[0183] a memory for storing instructions;

[0184] The processor is configured to execute the instructions so that the device performs operations to implement the multimodal remote sensing data classification method for fusing global and local information as described in any one of the first embodiments.

[0185] In summary, the present invention provides a multimodal remote sensing data classification method that integrates global and local information. By constructing a classification model, the spatial spectrum feature representation of the integrated multimodal data is obtained. By designing a multimodal feature decomposition and feature fusion structure, the relationship between multimodal data is mined, providing effective feature support for the deep learning classification model. The present invention decomposes multimodal information into common information and unique information by means of feature decoupling, realizes multimodal information fusion, and fully mines the features of multimodal remote sensing data, thereby achieving more accurate image classification. The accuracy of the multimodal data classification results can reach 95.76%. In addition, the experimental results in Example 2 show that on the public hyperspectral classification data set, 20 labeled samples of each class are randomly selected, and the overall accuracy of the classification model is 95.32%.

[0186] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal remote sensing data classification method that integrates global and local information, characterized in that: include: Obtain a multimodal remote sensing data set to be classified and input it into a pre-trained multimodal remote sensing data classification model; The multimodal remote sensing data classification model outputs a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified; The training method of the multimodal remote sensing data classification model includes: Obtain original hyperspectral remote sensing data and LiDAR data, perform dimensionality reduction processing on the original hyperspectral remote sensing data, and obtain low-dimensional hyperspectral features; Based on the determined number of training samples, randomly select training sets and test sets from low-dimensional hyperspectral features and LiDAR data, build an initial multimodal remote sensing data classification model, and set the number of iterations and parameters of the multimodal remote sensing data classification model; Perform multi-scale information fusion of low-dimensional hyperspectral features and LiDAR data to obtain low-dimensional hyperspectral shallow features and LiDAR data shallow features; Through the dual-branch Transformer-CNN feature extractor, the low-dimensional hyperspectral shallow features and the LiDAR data shallow features are respectively subjected to global information extraction and local information extraction, thereby obtaining the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data. The ECA module is used to fuse the low-frequency global features and high-frequency local features of low-dimensional hyperspectral data with the low-frequency global features and high-frequency local features of LiDAR data. The training set is used to cyclically train the initial multimodal remote sensing data classification model. When the maximum number of iterations is reached, the final multimodal remote sensing data classification model is obtained.

2. The multimodal remote sensing data classification method for fusing global and local information according to claim 1, characterized in that: The dual-branch Transformer-CNN feature extractor includes a SIM-Transformer module and an INN module, wherein the SIM-Transformer module is used to use structural information modeling to learn and process low-frequency global features, and the INN module is used to extract high-frequency local features.

3. The multimodal remote sensing data classification method for fusing global and local information according to claim 2, characterized in that: The multi-scale information fusion of low-dimensional hyperspectral features and LiDAR data to obtain low-dimensional hyperspectral shallow features and LiDAR data shallow features includes: The hyperspectral remote sensing image H and LiDAR L are respectively subjected to 3×3, 5×5 and 7×7 convolution operations to obtain multi-scale information H 3×3 、H 5×5 、H 7×7 , L 3×3 , L 5×5 and L 7×7 ; According to the channel dimension, fuse the three scale features, superimpose the input corresponding to the j-th dimension, and obtain the superimposed feature The formula is: Where, F c is the multi-scale spatial spectrum feature, k is the feature dimension, Concat is the channel superposition, i is the multi-scale operation, is the superposition feature, j is the number of dimensions; right The channel dimension of performs element-wise addition, element-wise averaging, and element-wise maximization to generate three feature maps representing different processing methods and properties. and The formula is: Where, For The channel dimension performs element-wise maximization on the generated feature map, For The channel dimension performs element-wise averaging to generate the feature map, For The feature map generated by performing element-wise addition on the channel dimension of Will and The multi-scale features are further fused through 3×3 convolution. The formula is: Where, F j For the general and Features that are superimposed and fused through convolution; The feature maps of k groups of three scales based on multi-scale are cascaded to obtain the final low-dimensional shallow features with dimension k. The formula is: Where H1 is a low-dimensional hyperspectral shallow feature with dimension k.

4. The multimodal remote sensing data classification method for fusing global and local information according to claim 3, characterized in that: The dual-branch Transformer-CNN feature extractor is used to extract global information and local information from the shallow features of the low-dimensional hyperspectral data and the shallow features of the LiDAR data, respectively, to obtain the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data, including: The acquired shallow features are projected into the D-dimensional embedding through linear mapping, and the learnable cls embedding token is introduced and placed at the beginning of the input sequence embedding. The position information embedding is added to the input sequence embedding to obtain the input of the first layer SIM-Transformer. Then: Where z0 is the input of the first layer SIM-Transformer, x cls is the learnable cls embedding token, F(·) is a linear mapping, E p To embed location information, is the Nth position token of the patch in the shallow feature H1, where N is the number of patches; Then the formula of the lth SIM-Transformer is: z′ l =LN(MSA(z l-1 )+z l-1 ) from l =LN(FFN(z l )+z′ l ); Where LN(·) is layer normalization, FFN is feedforward neural network, MSA is multi-head attention mechanism, z l is the global feature of the lth SIM-Transformer, z; l is the intermediate feature of the lth SIM-Transformer; Structural information learning is introduced to incorporate object space context information into the Transformer. Assuming that the Transformer layer has H layers, Q and K are the dimension query and key vectors of all tokens in D dimensions. The specific formula is as follows: Where, and N=N H ×N W is the number of patches, D is the dimension, Q is the dimension query of all tokens in D dimension, K is the key vector of all tokens in D dimension, softmax(.) is the activation function, normalizes the tokens, K T Expressed as the transpose of K; Extract the attention weights between the patch-token and the cls token of each head, expressed as The corresponding total attention weight is calculated as: Where A is the corresponding total attention weight, is the attention weight between the patch to token and the cls token of each head; To filter unimportant patches, set As the threshold, the new attention weight is: Where, is the new attention weight, A (x，y) is the relationship weight between the patch token and the cls token at position (x, y), is the threshold; The patch with the highest attention weight is considered the most discriminative patch and is set as the reference patch. Given a reference patch P0 = P x0，y0 and the horizontal reference direction, patchP x，y The polar coordinates of are defined as: Where, ρ x，y It is P0 and P x，y The relative distance, θ x，y is the normalized polar angle relative to the horizontal, N H is the length of the patch, N W is the height of the patch, (x0, y0) is N H ×N W The index of the plane, x is N H ×N W The horizontal coordinate position of the plane, .y is N H ×N W The ordinate position of the plane; Therefore, the structural features are obtained through two layers of graph convolution: S=σ(Adj×σ(Adj×X×W 1 )×W 2 ); Where S is the structural feature, W 1 and W 2 is a learnable parameter, σ is the activation function, Adj = A new ×(A new ) T Based on A new The edge weight between the nodes, X is the patch node feature; Using INN with Affine Coupling layer, the changes of each reversible layer are: Where ⊙ is the Hadamard product, is the 1st to cth channels of the input features of the kth reversible layer, k = 1, ..., K, ψ i is an arbitrary mapping function that does not affect the lossless information transmission of the reversible layer, i = 1, ..., 3, and C is the number of channels of local features.

5. The multimodal remote sensing data classification method for fusing global and local information according to claim 4, characterized in that: The ECA module is used to fuse the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data with the low-frequency global features and high-frequency local features of the LiDAR data, including: The high-frequency local features H of hyperspectral remote sensing images are fused through the ECA module. h and LiDAR’s high-frequency local features L h , get the high-frequency fusion feature F h ; Fusion of low-frequency global features H of hyperspectral remote sensing images l and LiDAR’s low-frequency global features L l , get the low-frequency fusion feature F l ; The high-frequency fusion feature F h Fusion with low-frequency features F l Fusion into the final feature F end ; The final feature F is made by the ECA module end Realize cross-channel interactive fusion of high-frequency and low-frequency information.

6. The multimodal remote sensing data classification method for fusing global and local information according to claim 5, characterized in that: The final feature F end The formula is: F end =Concat(F h ,F l ); Where, F h is the high-frequency fusion feature, F l is the low-frequency fusion feature, F end is the high-frequency fusion feature F h Fusion with low-frequency features F l The final features after fusion; The final feature F is obtained by the ECA module. end The formula for achieving cross-channel interactive fusion of high-frequency and low-frequency information is: ω=ρ(C1D k (F end )); Where ω is the high-frequency-low-frequency fusion feature, ρ is the Sigmod activation function, and C1D is the 1D convolution.

7. A multimodal remote sensing data classification device that integrates global and local information, characterized in that: include: The data acquisition module is used to obtain the multimodal remote sensing data set to be classified and input it into the pre-trained multimodal remote sensing data classification model; An output result module, configured for the multimodal remote sensing data classification model to output a multimodal remote sensing data classification result based on the multimodal remote sensing data set to be classified; The training method of the multimodal remote sensing data classification model includes: Obtain original hyperspectral remote sensing data and LiDAR data, perform dimensionality reduction processing on the original hyperspectral remote sensing data, and obtain low-dimensional hyperspectral features; Based on the determined number of training samples, randomly select training sets and test sets from low-dimensional hyperspectral features and LiDAR data, build an initial multimodal remote sensing data classification model, and set the number of iterations and parameters of the multimodal remote sensing data classification model; Perform multi-scale information fusion of low-dimensional hyperspectral features and LiDAR data to obtain low-dimensional hyperspectral shallow features and LiDAR data shallow features; Through the dual-branch Transformer-CNN feature extractor, the low-dimensional hyperspectral shallow features and the LiDAR data shallow features are respectively subjected to global information extraction and local information extraction, thereby obtaining the low-frequency global features and high-frequency local features of the low-dimensional hyperspectral data and the low-frequency global features and high-frequency local features of the LiDAR data. The ECA module is used to fuse the low-frequency global features and high-frequency local features of low-dimensional hyperspectral data with the low-frequency global features and high-frequency local features of LiDAR data. The training set is used to cyclically train the initial multimodal remote sensing data classification model. When the maximum number of iterations is reached, the final multimodal remote sensing data classification model is obtained.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal remote sensing data classification method for fusing global and local information as described in any one of claims 1 to 6 is implemented.

9. A device, characterized in that include: a memory for storing instructions; A processor is configured to execute the instructions so that the device performs operations to implement the multimodal remote sensing data classification method for fusing global and local information as described in any one of claims 1 to 6.

Images