A hyperspectral and lidar data fusion classification method based on AGLT network
Through the AGLT network-based method, combined with hyperspectral images and LiDAR-DSM data, the problem of poor classification performance in the existing methods is solved, better spectral feature capture and global feature extraction are achieved, and classification performance is improved.
Patent Information
- Application Number
- CN202311439960.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-01
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-11-01
AI Technical Summary
The existing methods have poor classification performance in the fusion classification of hyperspectral images and lidar data, making it difficult to effectively capture the sequence attributes and global image features of spectral features.
Using the AGLT network-based method, the hyperspectral image data is reduced by principal component analysis, combined with LiDAR-DSM data, the AGLT network is built, and the training network is trained using the training set until the classification accuracy on the test set is no longer improved. Finally, it is used to use The trained AGLT network performs joint processing of hyperspectral images and LiDAR-DSM images.
Through the AGLT network, hyperspectral null-spectral joint features can be captured and learned from hyperspectral images, and elevation features can be obtained from LiDAR-DSM data. The advantages of convolutional neural networks and visual Transformer are fully utilized to improve the classification performance of the model.
Smart Images

Figure CN117475216B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of hyperspectral image classification, and in particular relates to a hyperspectral and laser radar data fusion classification method based on an AGLT network. Background Art
[0002] Remote sensing technology has made significant progress in recent years, increasing the availability of remote sensing images. Usually, data from several remote sensing devices in the same geographic area are available, so multimodal data can be used to analyze land cover information. A variety of different sensor technologies can effectively capture land cover materials with different characteristics. With the acquisition of massive data, the data differences of different modalities can complement each other and effectively improve the effect of remote sensing ground object classification. In the task of multimodal data fusion and land cover interpretation, the fusion interpretability of hyperspectral image (HSI) and light detection and ranging digital surface model (LiDAR-DSM) data has always been an important issue that needs attention. Hyperspectral image sensors can obtain spectral information and geospatial information, and light detection and ranging digital surface models measure the elevation and object height information of the earth's surface. By integrating data from different modalities, more detailed information can be obtained, thereby constructing a complete feature representation.
[0003] The fusion of multiple data sources can improve the accuracy of land cover identification, but there are many technical obstacles, such as different data structures and irrelevant physical features. At present, hyperspectral data and LiDAR-DSM images have been successfully used together. Among them, convolutional neural networks are a common method for joint classification of hyperspectral data and LiDAR-DSM images. Convolutional neural networks are powerful tools for feature extraction and context modeling. However, due to the inherent limitations of their network backbones, they cannot establish long-distance connections of global images, so there are still defects in capturing the sequence properties of spectral features, and the convolution operation in convolutional neural networks can only capture local information, making it difficult to obtain discriminative spectral spatial features from a global perspective. The visual Transformer backbone network can address these challenges and generate new insights in multimodal image classification. The visual Transformer has more similarities between the features obtained in the shallow and deep layers, and is good at capturing the global feature information of the image, but cannot utilize the prior knowledge of the scale, translation invariance and feature locality of the image itself. It is necessary to use large-scale datasets to learn high-quality intermediate representations. However, due to the lack of training data, there are still certain difficulties in combining the convolutional neural network and the visual Transformer backbone network. Therefore, the classification performance of the existing methods is still poor and needs to be further improved. Summary of the invention
[0004] The purpose of the present invention is to solve the problem of poor classification performance of existing methods and propose a hyperspectral and lidar data fusion classification method based on AGLT network.
[0005] The technical solution adopted by the present invention to solve the above technical problems is:
[0006] A hyperspectral and laser radar data fusion classification method based on AGLT network, the method specifically comprises the following steps:
[0007] Step 1, obtaining LiDAR-DSM image data and hyperspectral image data;
[0008] Step 2: Using principal component analysis to perform dimensionality reduction processing on the acquired hyperspectral image data to obtain hyperspectral image data after dimensionality reduction processing;
[0009] Step 3, slicing the acquired LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction, and then dividing the sliced LiDAR-DSM image data and the hyperspectral image data into a training set and a test set;
[0010] Step 4: construct an AGLT network, and use the training set to train the constructed AGLT network until the classification accuracy of the AGLT network on the test set is no longer improved, and then stop the training to obtain a trained AGLT network;
[0011] Step 5: Use the trained AGLT network to jointly process the hyperspectral image and LiDAR-DSM image to be classified to obtain the classification result.
[0012] Furthermore, the AGLT network includes a hyperspectral image data processing branch and a LiDAR-DSM data processing branch, wherein:
[0013] In the hyperspectral image data processing branch, the input image passes through a three-dimensional normalization layer, a first activation function layer, a first three-dimensional convolution layer with a convolution kernel size of 1×1×3, a second three-dimensional convolution layer with a convolution kernel size of 1×3×1, a third three-dimensional convolution layer with a convolution kernel size of 3×1×1, a first two-dimensional normalization layer, a second activation function layer, a first two-dimensional convolution layer with a convolution kernel size of 3×1, a second two-dimensional convolution layer with a convolution kernel size of 1×3, and a first Bi-Former module in sequence;
[0014] In the LiDAR-DSM data processing branch, the input LiDAR-DSM data passes through the second 2D normalization layer, the third activation function layer, the third 2D convolution layer with a convolution kernel size of 3×1, the fourth 2D convolution layer with a convolution kernel size of 1×3, and the second Bi-Former module in sequence;
[0015] The output of the first Bi-Former module and the output of the second Bi-Former module are sent to the cross attention layer to obtain output A and output B. Then output A passes through the first MLP layer, and output B passes through the second MLP layer. Finally, the output of the first MLP layer and the output of the second MLP layer are superimposed. The superposition result is the final classification result.
[0016] Furthermore, the working principle of the Bi-Former module is:
[0017] The input image of the Bi-Former module is mapped into a vector sequence, and an additional learning code is embedded in the head of the vector sequence to obtain the whole sequence, and then the whole sequence is embedded in the position code. The vector sequence after the embedding position code passes through the encoder submodule, and the output of the encoder submodule is used as the output of the Bi-Former module; and the encoder submodule includes N encoders;
[0018] The first encoder works as follows:
[0019] Step 1: The vector sequence after embedding position encoding input by the encoder submodule is the input of the first encoder, and the vector sequence after embedding position encoding is mapped into a query vector, a key vector and a value vector respectively;
[0020] Step 2: Perform multi-head attention calculation on the query vector, key vector and value vector to obtain the multi-head attention calculation result;
[0021] Step 3: Perform residual connection on the multi-head attention calculation results of step 2 and the vector sequence after embedding position encoding, and normalize the residual connection results;
[0022] Step 4: Send the normalized result of step 3 to the Bi feedforward unit, then perform residual connection on the output of the Bi feedforward unit and the normalized result of step 3, and normalize the residual connection result, and use the normalized result as the output of the first encoder;
[0023] The output of the first encoder is then used as the input of the second encoder, and so on, until the output of the Nth encoder is obtained. The output of the Nth encoder is the output of the encoder submodule.
[0024] Furthermore, the calculation method of the multi-head attention is:
[0025] Step 1), the query vector, the key vector and the value vector are respectively passed through the linear layer, and then the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the first scaled dot product attention unit, the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the second scaled dot product attention unit, and the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the third scaled dot product attention unit;
[0026] splicing outputs of the first scaled dot product attention unit, the second scaled dot product attention unit, and the third scaled dot product attention unit to obtain a splicing result;
[0027] Step 2) The concatenated result is then processed by a linear layer, and the output after the linear layer processing is the output of the multi-head attention.
[0028] Furthermore, the calculation method of the first scaled dot product attention unit is:
[0029] Multiply the query vector after passing through the linear layer and the key vector after passing through the linear layer, scale the multiplication result, and continue to mask the scaled result. Finally, pass the masked result through the Softmax activation function.
[0030] Multiply the output of the Softmax activation function with the output of the value vector after passing through the linear layer to obtain the calculation result of the first scaled dot product attention unit.
[0031] Furthermore, the working principle of the Bi feedforward unit is:
[0032] The input of the Bi feedforward unit is sent to two parallel branches, where the first branch is the channel attention branch, which includes a global sub-unit, a linear layer, and a Sigmoid activation function layer; the second branch is the spatial attention branch, which includes a local sub-unit, a linear layer, and a Sigmoid activation function layer;
[0033] In the channel attention branch, the input of the Bi feedforward unit passes through the global subunit, linear layer and Sigmoid activation function layer in sequence to obtain the output X;
[0034] In the spatial attention branch, the input of the Bi feedforward unit first passes through the local sub-unit, and then the output of the local sub-unit and the global sub-unit are spliced. The spliced result then passes through the linear layer and Sigmoid activation function layer of the spatial attention branch in sequence to obtain the output Y;
[0035] Multiply X and Y, and then multiply the multiplication result with the input of the Bi feedforward unit to obtain the final multiplication result, that is, the output of the Bi feedforward unit.
[0036] Furthermore, the global subunit includes an average pooling layer, a linear layer and a GELU activation function layer; the local subunit includes a linear layer and a GELU activation function layer.
[0037] Furthermore, taking the hyperspectral image data as an example, the calculation method of cross attention is:
[0038] The additional learned encoding part of the hyperspectral data in the output of the first Bi-Former module is used as the linear projection function F HSI (·) input, and then the linear projection function F HSI The output of (·) is passed through the matrix W Q Map to query vector;
[0039] Linear projection function F HSI The output of (·) is stacked with the LiDAR-DSM feature encoding part in the output of the second Bi-Former module, and the stacking result is evenly divided into two parts, where the first part is obtained by the matrix W K Mapped to the key vector, the second part is through the matrix W V Map to a vector of values;
[0040] The linear projection function F HSI The query vector corresponding to the output of (·) is multiplied with the key vector corresponding to the first part of the stacking result to obtain the multiplication result a, and the multiplication result is input into the Softmax function; the output of the Softmax function is multiplied with the value vector corresponding to the second part of the stacking result to obtain the multiplication result b, and the multiplication result b is multiplied with the linear projection function F HSI The query vectors corresponding to the output of (·) are superimposed, and the superposition result is then input into the linear back-projection function G HSI (·), and then the linear back projection function G HSI The output of (·) is concatenated with the hyperspectral data feature encoding part in the output of the first Bi-Former module, and the concatenated result is output A;
[0041] Similarly, the additional learned encoding of the LiDAR-DSM data is cross-attention calculated to obtain output B.
[0042] The beneficial effects of the present invention are:
[0043] The present invention can capture and learn hyperspectral spatial-spectral joint features from hyperspectral image data and obtain elevation features from LiDAR-DSM data; and introduce asymmetric convolution kernels into the visual Transformer structure, making full use of the powerful spatial context information extraction capability of convolutional neural networks and the powerful long-range dependency modeling capability of the visual Transformer based on the self-attention mechanism; and design a Bi feedforward unit for the visual Transformer to fully extract the global and local information of the data, thereby improving the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is the model structure diagram of the AGLT network;
[0045] Figure 2 This is the training flow chart of the AGLT network;
[0046] Figure 3 It is the structural diagram of Bi-Former module;
[0047] Figure 4 It is a diagram of the multi-head attention structure;
[0048] Figure 5 It is the Bi feedforward unit structure diagram;
[0049] In the figure, (a) is channel attention, (b) is spatial attention;
[0050] Figure 6 It is a diagram of the cross-attention structure;
[0051] Figure 7 It is a TR data display chart;
[0052] In the figure, (a) is a hyperspectral false color image, (b) is a DSM grayscale image, and (c) is a true value image;
[0053] Figure 8 It is a graph showing MU data;
[0054] In the figure, (a) is a hyperspectral false color image, (b) is a DSM grayscale image, and (c) is a true value image;
[0055] Fig. 9 This is a graph showing AU data;
[0056] In the figure, (a) is a hyperspectral false color image, (b) is a DSM grayscale image, and (c) is a true value image;
[0057] Fig.10 This is the classification result diagram of different data by the AGLT network model. DETAILED DESCRIPTION
[0058] The present application is further described in detail below through specific implementations in combination with the accompanying drawings. Obviously, the described implementations are only part of the implementations of the present invention, not all of the implementations. Based on the implementations in the present invention, other implementations obtained by ordinary technicians in the field without making creative work are all within the scope of protection of the present invention.
[0059] Specific implementation method 1. Combination Figure 2 This embodiment describes a hyperspectral and laser radar data fusion classification method based on an AGLT network, and the method specifically includes the following steps:
[0060] Step 1: Obtain LiDAR-DSM image data and hyperspectral image data (the data obtained are currently public data);
[0061] Step 2: Using principal component analysis to perform dimensionality reduction processing on the acquired hyperspectral image data to obtain hyperspectral image data after dimensionality reduction processing;
[0062] Dimensionality reduction can reduce the number of bands in the third dimension of three-dimensional raw data, reduce data redundancy, and speed up running time;
[0063] Step 3, slicing the acquired LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction, and then dividing the sliced LiDAR-DSM image data and the hyperspectral image data into a training set and a test set;
[0064] Step 4: construct an AGLT network, and use the training set to train the constructed AGLT network until the classification accuracy of the AGLT network on the test set is no longer improved, and then stop the training to obtain a trained AGLT network;
[0065] Step 5: Use the trained AGLT network to jointly process the hyperspectral image and LiDAR-DSM image to be classified to obtain the classification result.
[0066] The present invention first combines convolutional neural network with visual transformer, which can capture and learn hyperspectral spatial-spectral joint features from hyperspectral image data and obtain elevation features from LiDAR-DSM data; secondly, introduce asymmetric convolution kernel into the visual transformer structure, making full use of the powerful spatial context information extraction ability of convolutional neural network and the powerful long-range dependency modeling ability of visual transformer based on self-attention mechanism; finally, design Bi feedforward unit for visual transformer to fully extract global and local information of data. It is used to fuse multi-source heterogeneous information and improve joint classification performance.
[0067] Specific implementation method 2: Combination Figure 1 This embodiment is different from the first embodiment in that the AGLT network includes a hyperspectral image data processing branch and a LiDAR-DSM data processing branch, wherein:
[0068] In the hyperspectral image data processing branch, the input image passes through a three-dimensional normalization layer, a first activation function layer, a first three-dimensional convolution layer with a convolution kernel size of 1×1×3, a second three-dimensional convolution layer with a convolution kernel size of 1×3×1, a third three-dimensional convolution layer with a convolution kernel size of 3×1×1, a first two-dimensional normalization layer, a second activation function layer, a first two-dimensional convolution layer with a convolution kernel size of 3×1, a second two-dimensional convolution layer with a convolution kernel size of 1×3, and a first Bi-Former module in sequence;
[0069] In the LiDAR-DSM data processing branch, the input LiDAR-DSM data passes through the second 2D normalization layer, the third activation function layer, the third 2D convolution layer with a convolution kernel size of 3×1, the fourth 2D convolution layer with a convolution kernel size of 1×3, and the second Bi-Former module in sequence;
[0070] The output of the first Bi-Former module and the output of the second Bi-Former module are sent to the cross attention layer to obtain output A and output B. Then output A passes through the first MLP layer, and output B passes through the second MLP layer. Finally, the output of the first MLP layer and the output of the second MLP layer are superimposed. The superposition result is the final classification result.
[0071] The other steps and parameters are the same as those in the first embodiment.
[0072] The convolution kernel of the hyperspectral data branch of the present invention adopts the form of a three-dimensional asymmetric convolution kernel (1×1×3, 1×3×1, 3×1×1) converted to a two-dimensional asymmetric convolution kernel (1×3, 3×1). At the same time, the DSM data branch adopts the form of a two-dimensional asymmetric convolution (1×3, 3×1), so that the multi-source image features can be fully learned.
[0073] Specific implementation method three: Combination Figure 3 This embodiment is different from the first or second embodiment in that the working principle of the Bi-Former module is as follows:
[0074] The input image of the Bi-Former module is mapped into a vector sequence, and an extra learable embedding is embedded in the head of the vector sequence to obtain the whole sequence, and then the embedding position of the whole sequence is encoded. The vector sequence after the embedding position encoding passes through the encoder submodule, and the output of the encoder submodule is used as the output of the Bi-Former module; and the encoder submodule includes N encoders;
[0075] The first encoder works as follows:
[0076] Step 1: The vector sequence after embedding position encoding input by the encoder submodule is the input of the first encoder, and the vector sequence after embedding position encoding is mapped into a query (Q) vector, a key (K) vector, and a value (V) vector respectively;
[0077] Step 2: Perform multi-head attention calculation on the query vector, key vector and value vector to obtain the multi-head attention calculation result;
[0078] Step 3: Perform residual connection on the multi-head attention calculation results of step 2 and the vector sequence after embedding position encoding, and normalize the residual connection results;
[0079] Step 4: Send the normalized result of step 3 to the Bi feedforward unit, then perform residual connection on the output of the Bi feedforward unit and the normalized result of step 3, and normalize the residual connection result, and use the normalized result as the output of the first encoder;
[0080] The output of the first encoder is then used as the input of the second encoder, and so on, until the output of the Nth encoder is obtained. The output of the Nth encoder is the output of the encoder submodule.
[0081] The other steps and parameters are the same as those in the first or second embodiment.
[0082] Specific implementation method four: Combination Figure 4 The present embodiment is different from the first to third embodiments in that the calculation method of the multi-head attention is:
[0083] Step 1), the query vector, the key vector and the value vector are respectively passed through the linear layer, and then the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the first scaled dot product attention unit, the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the second scaled dot product attention unit, and the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the third scaled dot product attention unit;
[0084] splicing outputs of the first scaled dot product attention unit, the second scaled dot product attention unit, and the third scaled dot product attention unit to obtain a splicing result;
[0085] Step 2) The concatenated result is then processed by a linear layer, and the output after the linear layer processing is the output of the multi-head attention.
[0086] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0087] Specific implementation mode 5: This implementation mode is different from any one of the specific implementation modes 1 to 4 in that the calculation method of the first scaled dot product attention unit is:
[0088] Multiply the query vector after passing through the linear layer and the key vector after passing through the linear layer, scale the multiplication result, and continue to mask the scaled result. Finally, pass the masked result through the Softmax activation function.
[0089] Multiply the output of the Softmax activation function with the output of the value vector after passing through the linear layer to obtain the calculation result of the first scaled dot product attention unit.
[0090] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0091] The calculation method of the second scaled dot product attention unit and the third scaled dot product attention unit is the same as the first scaled dot product attention unit.
[0092] Specific implementation method six: Combination Figure 5 This embodiment is different from the first to fifth embodiments in that the working principle of the Bi feedforward unit is:
[0093] The input of the Bi feedforward unit is sent to two parallel branches, where the first branch is the channel attention branch, which includes a global sub-unit, a linear layer, and a Sigmoid activation function layer; the second branch is the spatial attention branch, which includes a local sub-unit, a linear layer, and a Sigmoid activation function layer;
[0094] In the channel attention branch, the input of the Bi feedforward unit passes through the global subunit, linear layer and Sigmoid activation function layer in sequence to obtain the output X;
[0095] In the spatial attention branch, the input of the Bi feedforward unit first passes through the local sub-unit, and then the output of the local sub-unit and the global sub-unit are spliced. The spliced result then passes through the linear layer and Sigmoid activation function layer of the spatial attention branch in sequence to obtain the output Y;
[0096] Multiply X and Y, and then multiply the multiplication result with the input of the Bi feedforward unit to obtain the final multiplication result, that is, the output of the Bi feedforward unit.
[0097] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.
[0098] Specific implementation seven: This implementation differs from any one of specific implementations one to six in that the global subunit includes an average pooling layer, a linear layer and a GELU activation function layer; the local subunit includes a linear layer and a GELU activation function layer.
[0099] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0100] Specific implementation method eight: Combination Figure 6 The present embodiment is different from the first to seventh embodiments in that, taking the hyperspectral image data as an example, the calculation method of the cross attention is:
[0101] The additional learned encoding part of the hyperspectral data in the output of the first Bi-Former module is used as the linear projection function F HSI (·) input, and then the linear projection function F HSI The output of (·) is obtained by learning the matrix W Q Mapped to query (Q) vector; linear projection function F HSI The output of (·) is stacked with the LiDAR-DSM feature encoding part in the output of the second Bi-Former module, and the stacking result is evenly divided into two parts, where the first part is obtained by learning the matrix W K Mapped to the key (K) vector, the second part is to learn the matrix W V Mapped to a value (V) vector;
[0102] The linear projection function F HSI The query vector corresponding to the output of (·) is multiplied with the key vector corresponding to the first part of the stacking result to obtain the multiplication result a, and the multiplication result is input into the Softmax function; the output of the Softmax function is multiplied with the value vector corresponding to the second part of the stacking result to obtain the multiplication result b, and the multiplication result b is multiplied with the linear projection function F HSI The query vectors corresponding to the output of (·) are superimposed, and the superposition result is then input into the linear back-projection function G HSI (·), and then the linear back projection function G HSI The output of (·) is concatenated with the hyperspectral data feature encoding part in the output of the first Bi-Former module, and the concatenated result is output A;
[0103] Linear projection function F HSI (·) and the linear back-projection function G HSI The purpose of (·) is for dimension alignment;
[0104] Similarly, cross-attention calculation is performed on the additional learned code of the LiDAR-DSM data (the parameters in this case are the additional learned code of the LiDAR-DSM data and the feature code of the hyperspectral data, that is, the additional learned code of the LiDAR-DSM data is used to replace the position of the additional learned code of the hyperspectral data, and the feature code of the hyperspectral data is used to replace the feature code part of the LiDAR data), and the output B is obtained.
[0105] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0106] Since the additional learned encoding for the hyperspectral data has already learned abstract information among all feature encodings in the hyperspectral data, interacting with the feature encodings from the LiDAR-DSM data helps to learn the complementary information. Since the additional learned encoding for the LiDAR-DSM data has already learned abstract information among all feature encodings in the LiDAR-DSM data, interacting with the feature encodings from the hyperspectral data helps to learn the complementary information.
[0107] Example
[0108] The present invention proposes a hyperspectral and lidar data fusion classification method based on AGLT network. The implementation process of the method is shown in Table 1:
[0109] Table 1 AGLT network structure algorithm flow
[0110]
[0111] The specific implementation steps are as follows:
[0112] Step 1: Obtain hyperspectral image data and LiDAR-DSM data (using public data).
[0113] Step 2: Use the PCA principal component analysis method to perform dimensionality reduction processing on the obtained hyperspectral image data, which can reduce the number of bands in the third dimension of the three-dimensional original data, reduce data redundancy, and speed up the running time.
[0114] Step 3: Preprocess the LiDAR-DSM data and the reduced hyperspectral data. According to the number of labeled pixel samples in the hyperspectral image and the LiDAR-DSM image, specify the number of training sets and test sets, and store them in two sets of data with the same format as the LiDAR-DSM data and the reduced hyperspectral data. Then, slice the two sets of data.
[0115] Step 4: Training and classification of AGLT network
[0116] Step 4.1: Construct the AGLT network. The training set and test set images used in the present invention all adopt a resolution of 11×11. Figure 1 As shown in the figure, H×W represents the spatial dimension, C represents the third-dimensional band. After PCA processing and data preprocessing, a series of 11×11×L hyperspectral images (L represents the third-dimensional band after dimensionality reduction) and 11×11 LiDAR-DSM images are obtained.
[0117] Step 4.2, the AGLT network structure is mainly divided into three parts, the first is the asymmetric three-dimensional convolution for hyperspectral data, the asymmetric two-dimensional convolution after data reconstruction, and the asymmetric two-dimensional convolution for LiDAR-DSM data. Asymmetric three-dimensional convolution mainly includes 1×1×3, 1×3×1 and 3×1×1; asymmetric two-dimensional convolution mainly includes 1×3 and 3×1. While extracting the spatial-spectral joint feature information, the number of parameters is reduced.
[0118] Step 4.3: The outputs of the asymmetric convolutional neural network of the hyperspectral data branch and the LiDAR-DSM data branch are respectively fed into the Bi-Former. Figure 5 As shown in the figure, the Bi feedforward unit in the Bi-Former enables the AGLT network to obtain a lighter computational burden in a small channel dimension, increase cross-channel information interaction, and fuse global information with local information, further improving the model performance and accuracy.
[0119] Step 4.4: The output of the Bi-Former of the hyperspectral data branch and the LiDAR-DSM data branch is sent to the cross attention to interact with feature information and fully extract the multimodal data fusion features.
[0120] Step 5: Classify the images.
[0121] Experimental Section
[0122] The computer configuration used to train the network model and verify the classification results is as follows: CPU is Intel Core i9-12900K, memory is 32GBDDR5 5200, graphics card is NVIDIA GeForce RTX 3090Ti, hard disk is 1TBSSD plus 4T HDD, and the system is Windows 11 Professional Edition. Three well-known multimodal feature fusion datasets are used, namely the Trento dataset (TR), the University of Mississippi and the University of Florida golf course dataset (MU) and the Augsburg dataset (AU). The dataset display diagram is as follows Figure 7 , Figure 8 and Fig. 9As shown in the figure, different colors represent different types of land features. Figure 7 The corresponding detailed data information is shown in Table 2. Figure 8 The corresponding detailed data information is shown in Table 3. Fig. 9 The corresponding detailed data information is shown in Table 4.
[0123] Table 2
[0124]
[0125]
[0126] Table 3
[0127]
[0128] Table 4
[0129]
[0130] 1. The TR dataset was captured in the rural area around Trento, Italy. The dataset contains hyperspectral images and lidar images. The hyperspectral image size is 600×166 pixels, contains 63 bands, covers the spectral band of 420.89-989.09nm, has a spectral resolution of 9.2nm, and a spatial resolution of 1m. The lidar image is a single-channel image that contains the altitude of the corresponding ground position. The image size is the same as the hyperspectral image. There are 6 categories in the annotation information of the dataset.
[0131] 2.MU dataset is a registered aerial hyperspectral-lidar dataset. The two modal images of this dataset were acquired simultaneously in a single aerial flight in November 2010 in Mississippi, USA. The image size is 325×220 pixels. The hyperspectral image contains 64 spectral bands. The lidar image is a single-channel image containing the altitude of the corresponding ground position, and the image size is the same as the hyperspectral image. The data annotation information contains 11 categories.
[0132] The 3.AU dataset was captured over the city of Augsburg, Germany. The HSI data were acquired by the DAS-EOC HySpex sensor, based on LiDAR-DSM data collected by the DLR-3K system. The spatial resolution of both images was downsampled to a unified resolution of 30m to adequately manage multimodal fusion. In this dataset, the HSI data consists of 180 bands ranging from 0.4-2.5μm, while the LiDAR-DSM data has only one raster. The size of this dataset is 332×485 pixels. This dataset depicts seven different land cover classes.
[0133] For the evaluation indicators of the model, the three most widely used objective evaluation indicators in the industry are selected: Overall accuracy (OA), average accuracy (AA) and Kappa coefficient (K).
[0134] The objective classification results of the AGLT network model used in the present invention on three data sets are shown in Table 5.
[0135] Table 5
[0136]
[0137] From left to right, the first column is the data set; the second column is the OA data of the classification results; the third column is the AA data of the classification results; the fourth column is the K data of the classification results; the fifth column is the training time of the classification; and the sixth column is the test time of the classification. Fig.10 As shown, Fig.10 In the figure, (a) represents the classification effect of the TR dataset, (b) represents the classification effect of the MU dataset, and (c) represents the classification effect of the AU dataset. The higher the accuracy, the less salt and pepper noise in the image.
[0138] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A hyperspectral and lidar data fusion classification method based on AGLT network, characterized in that: The method specifically comprises the following steps: Step 1, obtaining LiDAR-DSM image data and hyperspectral image data; Step 2: Using principal component analysis to perform dimensionality reduction processing on the acquired hyperspectral image data to obtain hyperspectral image data after dimensionality reduction processing; Step 3, slicing the acquired LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction, and then dividing the sliced LiDAR-DSM image data and the hyperspectral image data into a training set and a test set; Step 4: construct an AGLT network, and use the training set to train the constructed AGLT network until the classification accuracy of the AGLT network on the test set is no longer improved, and then stop the training to obtain a trained AGLT network; The AGLT network includes a hyperspectral image data processing branch and a LiDAR-DSM data processing branch, wherein: In the hyperspectral image data processing branch, the input image passes through a three-dimensional normalization layer, a first activation function layer, a first three-dimensional convolution layer with a convolution kernel size of 1×1×3, a second three-dimensional convolution layer with a convolution kernel size of 1×3×1, a third three-dimensional convolution layer with a convolution kernel size of 3×1×1, a first two-dimensional normalization layer, a second activation function layer, a first two-dimensional convolution layer with a convolution kernel size of 3×1, a second two-dimensional convolution layer with a convolution kernel size of 1×3, and a first Bi-Former module in sequence; In the LiDAR-DSM data processing branch, the input LiDAR-DSM data passes through the second 2D normalization layer, the third activation function layer, the third 2D convolution layer with a convolution kernel size of 3×1, the fourth 2D convolution layer with a convolution kernel size of 1×3, and the second Bi-Former module in sequence; The output of the first Bi-Former module and the output of the second Bi-Former module are sent to the cross attention layer to obtain output A and output B. Then output A passes through the first MLP layer, and output B passes through the second MLP layer. Finally, the output of the first MLP layer is superimposed with the output of the second MLP layer. The superposition result is the final classification result. The working principle of the Bi-Former module is: The input image of the Bi-Former module is mapped into a vector sequence, and an additional learning code is embedded in the head of the vector sequence to obtain the whole sequence, and then the whole sequence is embedded in the position code. The vector sequence after the embedding position code passes through the encoder submodule, and the output of the encoder submodule is used as the output of the Bi-Former module; and the encoder submodule includes N encoders; The first encoder works as follows: Step 1: The vector sequence after embedding position encoding input by the encoder submodule is the input of the first encoder, and the vector sequence after embedding position encoding is mapped into a query vector, a key vector and a value vector respectively; Step 2: Perform multi-head attention calculation on the query vector, key vector and value vector to obtain the multi-head attention calculation result; Step 3: Perform residual connection on the multi-head attention calculation results of step 2 and the vector sequence after embedding position encoding, and normalize the residual connection results; Step 4: Send the normalized result of step 3 to the Bi feedforward unit, then perform residual connection on the output of the Bi feedforward unit and the normalized result of step 3, and normalize the residual connection result, and use the normalized result as the output of the first encoder; The output of the first encoder is then used as the input of the second encoder, and so on, until the output of the Nth encoder is obtained. The output of the Nth encoder is the output of the encoder submodule; The calculation method of the multi-head attention is: Step 1), the query vector, the key vector and the value vector are respectively passed through the linear layer, and then the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the first scaled dot product attention unit, the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the second scaled dot product attention unit, and the output of the query vector after passing through the linear layer, the output of the key vector after passing through the linear layer and the output of the value vector after passing through the linear layer are sent to the third scaled dot product attention unit; splicing outputs of the first scaled dot product attention unit, the second scaled dot product attention unit, and the third scaled dot product attention unit to obtain a splicing result; Step 2), the concatenated result is processed by the linear layer, and the output after the linear layer processing is the output of the multi-head attention; The calculation method of the first scaled dot product attention unit is: Multiply the query vector after passing through the linear layer and the key vector after passing through the linear layer, scale the multiplication result, and continue to mask the scaled result. Finally, pass the masked result through the Softmax activation function. Multiply the output of the Softmax activation function by the output of the value vector after passing through the linear layer to obtain the calculation result of the first scaled dot product attention unit; The working principle of the Bi feedforward unit is: The input of the Bi feedforward unit is sent to two parallel branches, where the first branch is the channel attention branch, which includes a global sub-unit, a linear layer, and a Sigmoid activation function layer; the second branch is the spatial attention branch, which includes a local sub-unit, a linear layer, and a Sigmoid activation function layer; In the channel attention branch, the input of the Bi feedforward unit passes through the global subunit, linear layer and Sigmoid activation function layer in sequence to obtain the output X; In the spatial attention branch, the input of the Bi feedforward unit first passes through the local sub-unit, and then the output of the local sub-unit and the global sub-unit are spliced. The spliced result then passes through the linear layer and Sigmoid activation function layer of the spatial attention branch in sequence to obtain the output Y; Multiply X and Y, and then multiply the multiplication result with the input of the Bi feedforward unit to obtain the final multiplication result, that is, the output of the Bi feedforward unit; The global subunit includes an average pooling layer, a linear layer and a GELU activation function layer; the local subunit includes a linear layer and a GELU activation function layer; Taking the hyperspectral image data as an example, the calculation method of cross attention is: The additional learned encoding part of the hyperspectral data in the output of the first Bi-Former module is used as the linear projection function F HSI (·) input, and then the linear projection function F HSI The output of (·) is passed through the matrix W Q Map to query vector; Linear projection function F HSI The output of (·) is stacked with the LiDAR-DSM feature encoding part in the output of the second Bi-Former module, and the stacking result is evenly divided into two parts, where the first part is obtained by the matrix W K Mapped to the key vector, the second part is through the matrix W V Map to a vector of values; The linear projection function F HSI The query vector corresponding to the output of (·) is multiplied with the key vector corresponding to the first part of the stacking result to obtain the multiplication result a, and the multiplication result is input into the Softmax function; the output of the Softmax function is multiplied with the value vector corresponding to the second part of the stacking result to obtain the multiplication result b, and the multiplication result b is multiplied with the linear projection function F HSI The query vectors corresponding to the output of (·) are superimposed, and the superposition result is then input into the linear back-projection function G HSI (·), and then the linear back projection function G HSI The output of (·) is concatenated with the hyperspectral data feature encoding part in the output of the first Bi-Former module, and the concatenated result is output A; Similarly, the additional learning code of the LiDAR-DSM data is used to replace the position of the additional learning code of the hyperspectral data, and the feature code of the hyperspectral data is used to replace the feature code part of the LiDAR data. The additional learning code of the LiDAR-DSM data is cross-attention calculated to obtain the output B; Step 5: Use the trained AGLT network to jointly process the hyperspectral image and LiDAR-DSM image to be classified to obtain the classification result.
Citation Information
Patent Citations
Hyperspectral and laser radar fusion classification method based on shuffle feature enhancement
CN114972864A
Remote sensing hyperspectral image and laser radar image fusion classification method and device
CN115331110A
