A Hyperspectral Image and LiDAR Image Fusion Classification Method Based on State Space Model

Abstract

A hyperspectral image and lidar image fusion classification method based on a state space model, belonging to the technical field of remote sensing image processing and applications. First, a spectrometer and a lidar are used to collect hyperspectral images and lidar images of the same area; secondly, the hyperspectral images and lidar images are respectively used to extract local features using a convolutional neural network, and global features are extracted using a visual state space model; thirdly, the features are sorted and input into a designed state and feature dual fusion neural network to obtain the fused features, and a multi-layer perceptron is used to classify the fused features to obtain the classification result. The present invention can solve the problems of low accuracy and poor generalization ability of existing fusion classification methods, realize high-accuracy and high-generalization ability hyperspectral and lidar image fusion classification, provide an effective solution for hyperspectral and lidar image classification, and improve its reliability and accuracy in practical applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing and applications, and relates to a multi-modal remote sensing image fusion classification method, and particularly to a hyperspectral image and lidar image fusion classification method based on a state space model. Background Technique

[0002] Remote sensing image classification is an important research branch in the technical field of remote sensing image processing and applications. With the development of imaging technology and earth observation technology, the modalities of remote sensing images in the same geographical area are becoming increasingly rich. Hyperspectral images contain the reflected information of ground objects collected in the visible to near-infrared bands in a passive imaging manner, and lidar images contain the elevation information of ground objects obtained in the microwave band in an active imaging manner. By performing multi-modal information fusion analysis on hyperspectral images and lidar images and utilizing the complementarity of information, the classification accuracy of ground objects can be significantly improved, thereby further improving the efficiency of earth observation. A large number of studies have been carried out on the fusion classification method of hyperspectral images and lidar images, and remarkable results have been achieved:

[0003] Li Jiaxin, Hong Danfeng, Gao Lianru, etc. (Reference: LI J, HONG D, GAO L, et al. Deep learning in multimodal remote sensing data fusion: A comprehensive review[J / OL]. International Journal of Applied Earth Observation and Geoinformation, 2022, 112: 102926.) reviewed the research background and significance of hyperspectral image and lidar image fusion classification in the article, and introduced the existing methods, existing problems and the vision for future development.

[0004] Yao Dong, etc. (Chinese invention patent, application patent number: CN202410998576.5) invented a method for hyperspectral and lidar image detection and analysis based on noise filtering and spectral distribution information, which solved the problem that the prior art could not efficiently detect spectral echoes and accurately analyze spectra, thereby realizing the accurate detection and identification of targets. However, its algorithm relies on artificial feature extraction, and the adaptability of the algorithm is low and the portability is poor.

[0005] Wang Wenzhen et al. (Chinese invention patent, application patent number: CN202410717318.5) invented a classification method based on convolutional neural network and prototype enhancement, which solved the problem of data differences from different sources and thus greatly improved the accuracy of ground object classification in complex scenes. However, its model structure is highly linearized and overly relies on data similarity within the same dataset for classification, and the generalization ability of the model needs to be improved. Meng Xiangchao et al. (Chinese invention patent, application patent number: CN202410854745.8) invented a joint classification method for hyperspectral and lidar based on an attention model and category-aware fusion network, which solved the problem that the prior art failed to fully utilize uncertain categories to mine the features of hyperspectral images and lidar images, realized targeted information aggregation, and thus improved the classification accuracy of the overall model. However, its adaptability to special environments is low and it is difficult to meet the classification requirements under complex weather.

[0006] In summary, there is an urgent need to develop a deep learning model with a wide coverage, good adaptability and the ability to represent highly non-linear relationships to effectively extract, represent and fuse the global and local information contained in hyperspectral and lidar. Thus, it can meet the application scenarios of accurate joint classification of hyperspectral images and lidar images under different environments. Summary of the Invention

[0007] The object of the present invention is to overcome the defects of the prior art and propose a fusion classification method for hyperspectral images and lidar images based on a state space model to solve the problem that the existing fusion classification technology is difficult to effectively fuse global and local information, resulting in poor generalization ability of the model. The state space model includes a vision selection state space model (Vision Mamba, ViM), a selection state space model (Mamba) and the state and feature dual fusion neural network proposed by the present invention. This method uses the vision selection state space model to extract image features to solve the problem that the existing methods cannot effectively deeply mine hyperspectral and lidar information; and uses the selection state space model and the state and feature dual fusion neural network to efficiently fuse hyperspectral and lidar information to solve the problem of poor model generalization ability.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] A method for fusing and classifying hyperspectral images and lidar images based on a state space model. The method for fusing and classifying hyperspectral images and lidar images mainly solves two technical problems in the fusion and classification of hyperspectral images and lidar images, namely, the inability to effectively extract features and the inability to effectively fuse features, and proposes a technical solution to effectively solve these two technical problems using a state space model. The method for fusing and classifying hyperspectral images and lidar images is as follows: First, a spectrometer and a lidar are used to collect a hyperspectral image H and a lidar image L of the same area. Second, the hyperspectral image and the lidar image are respectively used by a convolutional neural network to extract local features to obtain H1 and L1, and the hyperspectral image and the lidar image are respectively used by a visual state space model to extract global features to obtain H2 and L2. Third, the features obtained above are sorted as H1, L1, H2, and L2 and then input into the state and feature dual fusion neural network designed by the present invention to obtain the fused feature Z. Finally, a multi-layer perceptron is used to classify the fused feature Z to obtain a classification result. The specific steps are as follows:

[0010] Step 1, extract features from the hyperspectral image and the lidar image;

[0011] The hyperspectral image H is obtained by observing the area G with a spectrometer. The lidar image L is obtained by observing the area G with a lidar.

[0012] Step 1.1, extraction of global features based on the visual selection state space model;

[0013] The visual selection state space model is used to extract global features from the hyperspectral image and the lidar respectively, and the hyperspectral global feature vector sequences h1, h2, …, h N and the lidar global feature vector sequences l1, l2, …, l N are obtained.

[0014] The specific steps of the visual selection state space model are as follows: First, the image H is cut and classified into several hyperspectral image blocks k 11 , k 12 , …, k NN . Second, all the hyperspectral image blocks are arranged in a vector sequence a1, a2, …, a N in the order from top to bottom and from left to right. Third, a1, a2, …, a N are input into two identical visual selection state space models and the outputs are added to obtain its global feature vector sequence.

[0015] Step 1.2, extraction of local features based on the convolutional neural network;

[0016] The convolutional neural network is used to extract local features from the hyperspectral image and the lidar image respectively.

[0017] For hyperspectral images, first, use a 3D convolutional neural network to extract the spatial and spectral local features of the hyperspectral image; second, use a 2D convolution to further extract the local spatial information. Finally, flatten the convolutional output in the spatial dimension to obtain a sequence of hyperspectral local feature vectors p1, p2, …, p N .

[0018] For lidar images, first, use a 2D convolutional neural network to extract the local spatial information; second, use another 2D convolution to further extract the local spatial information. Finally, flatten the convolutional output in the spatial dimension to obtain a sequence of lidar local feature vectors q1, q2, …, q N .

[0019] Step 2, local-global bidirectional feature ranking;

[0020] For the sequence of hyperspectral global feature vectors h1, h2, …, h N and the sequence of lidar global feature vectors l1, l2, …, l N , the sequence of hyperspectral local feature vectors p1, p2, …, p N and the sequence of lidar local feature vectors q1, q2, …, q N obtained in Step 1, arrange them in the order from local to global to get the bottom-up vector sequence F, and arrange them in the order from global to local to get the top-down vector sequence B, where the order of hyperspectral and lidar can be arbitrary.

[0021] Step 3, state and feature dual fusion neural network;

[0022] The present invention proposes a new state and feature dual fusion neural network ( Figure 1 ), which consists of three selective state space models, denoted as Mamba1, Mamba2, Mamba and the state feature dual fusion module StateFusion ( Figure 2 ) designed by the present invention. The specific steps of the state and feature dual fusion neural network are as follows:

[0023] Step 3.1 Bidirectional feature analysis;

[0024] Input the bottom-up vector sequence F and the top-down vector sequence B obtained in Step 2 into two selective state space models Mamba1 and Mamba2 respectively to obtain the output features F1 and B1. The final states s1 and s2 of the two selective state space models Mamba1 and Mamba2 are also used as outputs. That is:

[0025] F1, s1 = Mamba1(F) (3.1)

[0026] B1,s2 = Mamba2(B) (3.2)

[0027] Step 3.2 State fusion;

[0028] Input the final states s1 and s2 of the two selected state space models Mamba1 and Mamba2 obtained in Step 3.1 into the state feature dual fusion module StateFusion to obtain the bidirectional fusion state s. That is:

[0029] s t = A1s1 + A2s2 (3.3)

[0030] s = Cs t + D (3.4)

[0031] where s t is the intermediate state after preliminary fusion, s is the finally fused state, A1, A2, C, and D are all learnable parameters, A1 is the transformation matrix of state s1, A2 is the transformation matrix of state s2, C is the transformation matrix of intermediate state s t and D is the offset vector of intermediate state s t .

[0032] Step 3.3 The fused state is the selected state space model with the initial state;

[0033] Use the bidirectional fusion state s obtained in Step 3.2 as the initial state s0 of the selected state space model Mamba, and splice and input the output features F1 and B1 obtained in Step 3.1 into the selected state space model Mamba to obtain the fused feature vector sequence Z. That is:

[0034]

[0035] Step 4, classify the fused feature vector sequence Z obtained in Step 3;

[0036] Use a multi-layer perceptron as the classifier. Take the last vector of the fused feature vector sequence Z output by the selected state space model Mamba in Step 3.3 for multi-layer perceptron calculation to obtain the classification vector y. Take the dimension number where the maximum component in the classification vector is located as the classification result C.

[0037] Advantages of the present invention:

[0038] First, the present invention utilizes a visual selection state space model and a convolutional neural network. Through the global feature extraction method of the visual selection state space model in step 1 and the local feature extraction method of the convolutional neural network, it effectively extracts, characterizes, and fuses the global and local information contained in hyperspectral and lidar data, solving the problem of ineffective feature extraction in the prior art.

[0039] Second, the present invention designs a method for sorting a sequence of feature vectors. In step 2, the local feature vector sequence of hyperspectral data, the global feature vector sequence of hyperspectral data, the local feature vector sequence of lidar data, and the global feature vector sequence of lidar data are sorted in two directions, from top to bottom and from bottom to top, thereby maximizing the analytical ability of the selection state space model.

[0040] Third, the present invention designs a state and feature dual fusion neural network to efficiently fuse global and local features, thereby improving the classification accuracy and providing a deep learning fusion classification model with a wide coverage range, good adaptability, and the ability to represent highly nonlinear relationships.

[0041] In summary, the present invention can achieve a high-accuracy classification task for the collected hyperspectral images and lidar images, providing an effective solution for high-precision ground object observation, thus meeting the application scenarios of accurate joint classification of hyperspectral images and lidar images in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is the schematic diagram of the present invention.

[0043] Figure 2 is the schematic diagram of the state fusion mechanism designed by the present invention.

[0044] Figure 3 is the true category map and the model classification result map of the method of the present invention; Figure 3 in (a) is the true category of the test data set; Figure 3 in (b) is the model classification result. DETAILED DESCRIPTION OF THE INVENTION

[0045] The following further illustrates the present invention in conjunction with specific implementation cases.

[0046] A method for fusing and classifying hyperspectral images and lidar images based on a state space model. The task of the state space model is to perform pixel-level classification of ground objects based on hyperspectral images and lidar images, specifically including the following steps:

[0047] Step 1, extract features from the hyperspectral image H and the lidar image L;

[0048] The hyperspectral image H is obtained by observing the area G with a spectrometer. The lidar image L is obtained by observing the area G with a lidar.

[0049] The hyperspectral image and the lidar image of the same area G collected using the spectrometer and the lidar are denoted as H and L respectively. The hyperspectral image and the lidar image are collected in and around the University of Houston. The spectral coverage range of the hyperspectral image is 380nm - 1050nm, with a total of 144 bands. The spatial resolutions of both the hyperspectral and lidar are 2.5 meters. The ground truth information contains label information with the same resolution as H and L, which is manually collected and denoted as Y. The labels contain 15 categories of ground objects, namely healthy grassland, stressed grassland, synthetic grassland, trees, soil, water, residential area, commercial area, road, highway, railway, type I parking lot, type II parking lot, stadium, and runway.

[0050] Step 1.1 Global feature extraction based on the state space model;

[0051] The state space model mentioned above is the visual selection state space model. The visual selection state space model is the same deep learning model as the reference (Reference: Zhu L, Liao B, Zhang Q, et al. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model [A / OL]. arXiv, 2024 [2024 - 01 - 19]. http: / / arxiv.org / abs / 2401.09417). The specific steps are as follows:

[0052] Step 1.1.1, Use a sliding window to cut the hyperspectral image H to obtain hyperspectral image patches k with a shape of c×w×w (where c represents the number of channels and w represents the width). 11 , k 12 , …, k NN . The sliding window is centered at the pixel position of the training sample and has a square shape with a side length of 3.

[0053] Step 1.1.2, Sort each hyperspectral image patch k 11 , k 12 , …, k NN obtained by the sliding window cutting in the order from top to bottom and from left to right according to the window sliding order, that is, k 11 , k 12 , …, k NN , and flatten it into a shape of c×l in the image dimension, where l = w×w.

[0054] Step 1.1.3, input the feature sequences into two identical visual selection state space models respectively to analyze the relational characteristics among them. The state dimension of the visual selection state space model is 32. Two feature sequences with the same tensor dimension as the input sequence are obtained.

[0055] Step 1.1.4, perform weighted addition on the outputs of the visual selection state space models in two directions through primary features to obtain a hyperspectral global feature vector sequence h1, h2, …, h with the shape of c1×l1 N 。

[0056] Step 1.1.5, perform Steps 1.1.1 to 1.1.4 on the lidar image to obtain a lidar global feature vector sequence l1, l2, …, l with the shape of c1×l1 N 。

[0057] Step 1.2, extraction of local features of hyperspectral images and lidar images;

[0058] Step 1.2.1, pass the hyperspectral image H through a three-dimensional convolutional layer with a size of 6×3×3, and then through a normalization layer and a rectified linear activation layer to obtain the primary spatial-spectral local texture features of the hyperspectral image.

[0059] Step 1.2.2, pass the primary spatial-spectral local texture features through a two-dimensional convolutional layer with a size of 3×3, and then through a normalization layer to obtain a tensor with the shape of c1×w1×w1. Finally, flatten it in the spatial dimension to c1×l1 to obtain a hyperspectral local feature vector sequence p1, p2, …, p N 。

[0060] Step 1.2.3, pass the lidar image L samples through a two-dimensional convolutional layer with a size of 3×3, and then through a normalization layer to obtain the primary local texture features.

[0061] Step 1.2.4, pass the primary local texture features through a two-dimensional convolutional layer with a size of 3×3 again, and then through a normalization layer to obtain a tensor with the shape of c1×w1×w1. Finally, flatten it in the spatial dimension to c1×l1 to obtain a lidar local feature vector sequence q1, q2, …, q N 。

[0062] Step 2, local-global bidirectional feature ranking;

[0063] For the hyperspectral global feature vector sequence h1, h2, …, h obtained in Step 1 N 、the lidar global feature vector sequence l1, l2, …, l N 、the hyperspectral local feature vector sequence p1, p2, …, p Nand the local feature vector sequences of lidar q1, q2, …, q N , are arranged in the order from local to global to obtain the bottom-up vector sequence F, and are arranged in the order from global to local to obtain the top-down vector sequence B. The order of hyperspectral and lidar can be arbitrary, e.g.:

[0064] F = p1, p2, …, p N , q1, q2, …, q N , h1, h2, …, h N , l1, l2, …, l N (2.1)

[0065] B = l1, l2, …, l N , h1, h2, …, h N , q1, q2, …, q N , p1, p2, …, p N (2.2)

[0066] Step 3, state and feature dual fusion neural network;

[0067] The present invention proposes a new state and feature dual fusion neural network, which consists of three state fusion modules StateFusion designed by selective state space models Figure 2 ). The three selective state space models are denoted as Mamba1, Mamba2, Mamba.

[0068] Step 3.1, input the bottom-up vector sequence F and the top-down vector sequence B obtained in Step 2 into two selective state space models Mamba1 and Mamba2 respectively. At the same time, the final states s1 and s2 of the two selective state space models Mamba1 and Mamba2 are also used as outputs, as shown in formulas (3.1) and (3.2).

[0069] The definition of the selective state space model (reference: Gu A, Dao T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces [A / OL]) is:

[0070] s(t) = As(t - 1) + Bx(t) (3.6)

[0071] y(t) = Cs(t) + D (3.7)

[0072] Among them, both A and D are learnable parameters, B and C are parameters generated by a parameter generation network (composed of one-dimensional convolution and linear layers) according to the input, s(t) is the state sequence, t is the serial number of the feature sequence and the state sequence, x(t) is the input feature vector sequence, and y(t) is the output feature vector sequence. The state vector dimension parameter of the state space model is selected to be 32.

[0073] Step 3.2: Input the final states s1 and s2 of the two selected state space models Mamba1 and Mamba2 obtained in Step 3.1 into the state fusion module StateFusion to obtain the bidirectional fusion state s. As shown in Equation 3.3, its detailed state fusion formula is shown in Equations (3.1) and (3.2).

[0074] Step 3.3: Use the bidirectional fusion state s obtained in Step 3.2 as the initial state s0 of the selected state space model Mamba, and splice and input the output features F1 and B1 obtained in Step 3.1 into the selected state space model Mamba to obtain the fused feature vector sequence Z. As shown in Equation (3.5).

[0075] The state vector dimension parameter of the selected state space model Mamba is 32.

[0076] Step 3.4 classifies the fused feature vector sequence Z obtained in Step 3;

[0077] Step 3.4.1: Take the last one-dimensional vector y0 of the fused feature vector sequence Z with the shape of c2×l2.

[0078] Step 3.4.1: Use a multi-layer perceptron as the classifier to calculate y0 to obtain the classification vector y.

[0079] Step 3.4.2: Obtain the classification result C by calculating the dimension number where the maximum component in the classification vector is located.

[0080] C = argmax(y)(3.8)

[0081] Among them, argmax(y) represents the dimension number where the maximum component among all components y1, y2,..., y of the vector y is located. N in.

[0082] The technical effects of the embodiments of the present invention are described below through simulation experiments:

[0083] First, simulation conditions;

[0084] The experimental settings for both the present invention and the comparative method are as follows: a desktop computer with an Intel Core i5-12400F central processing unit, 64 GB of memory, and a GeForce RTX 3090 graphics processing unit. The software environment is the PyTorch deep learning framework. The number of training cycles for the model is 100, the model optimization algorithm is the Adam algorithm, the learning rate is 0.01, the dataset is the University of Houston 2013 hyperspectral and lidar dataset, and the training set and test set are divided using the default standard division method of the dataset, that is, a small number of samples in the dataset without special cloud and fog occlusion environments are selected as the training set, and the rest are used as the test set.

[0085] The specific comparative method is as follows: a widely studied and used attention-based deep neural network method at present (Reference: WANG X, FENG Y, SONG R, etc. Multi-attentive hierarchical densefusion net for fusion classification of hyperspectral and LiDAR data[J / OL]. Information Fusion, 2022, 82: 1-18. DOI: 10.1016 / j.inffus.2021.12.008 .). First, this method extracts spectral features and spatial features from the hyperspectral image using a convolutional neural network, and at the same time extracts lidar features using a convolutional neural network. Then, the extracted features are put into three attention modules. Finally, the outputs of the three attention modules are concatenated and classified using a multi-layer perceptron.

[0086] Second, the simulation results are shown in Table 1.

[0087] Table 1, the simulation results are

[0088]

[0089]

[0090] According to the simulation result data in Table 1, it can be seen that the classification results of almost all categories of the present invention are better than those of the control method. The two categories with the largest lead are the highway with cloud and fog occlusion and a certain type of parking lot. This shows that the present invention has a significant effect in improving special weather and enhancing the adaptability and discrimination ability of the model. The above embodiments only represent the implementation manners of the present invention, but should not be construed as limiting the scope of the present invention patent. It should be noted that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A hyperspectral image and lidar image fusion classification method based on a state space model, characterized in that, First, use a spectrometer and lidar to collect hyperspectral images of the same area and lidar images ; Second, use a convolutional neural network to extract local features from the hyperspectral image and lidar image respectively to obtain , use a visual state space model to extract global features from the hyperspectral image and lidar image respectively to obtain ; Again, input the features obtained previously into the state and feature dual fusion neural network after sorting them according to to obtain the fused features ; finally, use a multi-layer perceptron to classify the fused features to obtain the classification result; Specifically, it includes the following steps: Step 1, extract features from the hyperspectral image and the lidar image; Observation area by spectrometer Obtain hyperspectral images ; Observation area by lidar Obtain lidar images ; Step 1.1, global feature extraction based on the vision selection state space model; Use a visual selection state space model to perform global feature extraction on hyperspectral images and lidar respectively, and obtain a sequence of hyperspectral global feature vectors and a sequence of lidar global feature vectors ; Step 1.2, local feature extraction based on the convolutional neural network; Use the convolutional neural network to perform local feature extraction on the hyperspectral image and the lidar image respectively; Step 2, local-global bidirectional feature ranking; The hyperspectral global feature vector sequence obtained in step 1 , the lidar global feature vector sequence , the hyperspectral local feature vector sequence and the lidar local feature vector sequence are arranged in the order from local to global to obtain a bottom-up vector sequence and arranged in the order from global to local to obtain a top-down vector sequence , where the order of hyperspectral and lidar can be arbitrary; Step 3, state and feature dual fusion neural network; Design a dual-fusion neural network of state and features, which consists of three select state space models, denoted as and the designed dual-fusion module of state and features ; In the said Step 3, the specific design steps of the state and feature dual fusion neural network are as follows: Step 3.1 Bidirectional feature analysis; The bottom-up vector sequence obtained in step 2 and the top-down vector sequence are respectively input into two selection state space models to obtain output features and The final states of the two selection state space models are also used as outputs; that is: (3.1) ， (3.2) ， Step 3.2 State fusion; The final states of the two selected state space models obtained in step 3.1 are input into the state feature dual fusion module to obtain a bidirectional fusion state ; that is: ​ （3.3）， （3.4）， Among them, is the intermediate state after preliminary fusion, is the state of final fusion, both are learnable parameters, is the state transformation matrix, is the state transformation matrix, is the intermediate state transformation matrix, is the intermediate state offset vector; Step 3.3 The selection state space model with the fused state as the initial state; Take the two-way fusion state obtained in Step 3.2 as the initial state of the selective state space model , and splice the output features obtained in Step 3.1 and input them into the selective state space model to obtain a sequence of fused feature vectors ; that is: ; namely: (3.5)； Step 4. Classify the fused feature vector sequence obtained in Step 3 ; Using a multi-layer perceptron as a classifier, the state space model selected in step 3.3 The output sequence of fused feature vectors , take the last vector for multi-layer perceptron calculation to obtain a classification vector ; Take the dimension number where the maximum component in the classification vector is located as the classification result .

2. A hyperspectral image and lidar image fusion classification method based on a state space model according to claim 1, characterized in that In step 1.1, the specific steps of the visual selection state space model are as follows: First, the image is cut and classified into several hyperspectral image patches . Second, all the hyperspectral image patches are arranged in a vector sequence in the order from top to bottom and from left to right . Finally, is input into two identical visual selection state space models and the outputs are added together to obtain its global feature vector sequence.

3. A hyperspectral image and lidar image fusion classification method based on a state space model according to claim 1, characterized in that, In the said Step 1.2, the specific steps of local feature extraction are: For hyperspectral images, first, a 3D convolutional neural network is used to extract the spatial and spectral local features of the hyperspectral images; second, 2D convolution is used to further extract the local spatial information. Finally, the convolutional output is flattened in the spatial dimension to obtain a sequence of hyperspectral local feature vectors containing spatial and spectral information ; For lidar images, first, a two-dimensional convolutional neural network is used to extract local spatial information; second, another two-dimensional convolution is used to further extract local spatial information; finally, the convolutional output is flattened in the spatial dimension to obtain a sequence of lidar local feature vectors .

Images