Hyperspectral image and LiDAR data collaborative classification based on double-branch Swin Transformer

Abstract

The invention discloses a hyperspectral image and LiDAR data collaborative classification method based on double-branch Swin Transformer, and the method comprises the steps: S1, employing 3D-2D mixed convolution to extract the shallow space-spectrum information of a hyperspectral image after principal component analysis and dimension reduction according to the characteristic of image and spectrum integration of the hyperspectral image; and S2, an improved Swin Transform network is adopted, a ResNet module is adopted in the network, and SSR (Spectral Segment Recombination) is introduced in the network, so that information loss in feature transmission is eliminated, and deep space-spectrum features of the hyperspectral image are further mined. S3, carrying out 2D-2D convolution on the LiDAR data of the other branch, and mining elevation information through an improved Swin Transform network; and S4, aiming at the hyperspectral and LiDAR double branches, improving a CAM (Cross Attention Module) into double-branch input, through hyperspectral and LiDAR feature interaction, by establishing a dynamic attention mechanism, effectively capturing associated information between two modals, and fusing the associated information into uniform feature representation. The network is tested on two public hyperspectral and LiDAR data sets, and shows good classification performance.

Description

Technical Field

[0001] The invention discloses a collaborative classification method of hyperspectral image and LiDAR data using a dual-branch Swin Transformer, which belongs to the field of image classification. Background Art

[0002] Remote sensing imaging technology has made significant progress in recent years, making the availability of HSI (Hyperspectral image) increasingly available. Often, data from multiple remote sensing devices for the same geographic area are available, making it possible to analyze land cover information using multimodal data. A variety of different sensor technologies can efficiently capture different features of surface materials. For example, hyperspectral image sensors can obtain spectral information as well as geospatial information, while light detection and ranging or digital surface models can measure elevation and object height information on the surface. By integrating data from these different modalities, we can obtain more detailed information to construct an overall feature representation. This approach helps interpret remote sensing images in many research and application areas, including land cover classification, object detection, semantic segmentation, environmental monitoring, mineral exploration, urban planning, medical diagnosis, precision agriculture, and food safety. Land cover classification has always been a particularly prominent application. Early remote sensing data classification work mainly relied on a single data type. For example, HSI classification only uses HSI data. However, HSI classification methods have difficulty distinguishing objects with similar spectral features but different heights, such as concrete roofs and roads. In contrast, LiDAR data has the natural advantage of recording the height information of ground objects, which can provide supplementary information for further improving and refining the classification of land cover. Therefore, based on the above characteristics of hyperspectral remote sensing images and LiDAR data, in the intelligent analysis of remote sensing data, the coordination and feature fusion of these two types of data can maximize their respective characteristics.

[0003] At present, many classification algorithms based on deep learning tasks still have problems such as data redundancy, insufficient fine feature extraction, and loss of global and local spectral information. There is also the problem of poor classification accuracy. Summary of the invention

[0004] The object of the present invention is achieved in that:

[0005] Step 1: Use the principal component analysis method to remove the spectral redundancy of the hyperspectral image, and then use the 3D-2D hybrid convolution to fully extract the shallow spatial-spectral features of the hyperspectral image;

[0006] Step 2: The shallow spatial-spectral features of the hyperspectral image are sent to the improved Swin Transformer network, which adds the ResNet module and SSR (Spectral Segmentation Recombination) to the network to eliminate the information loss in feature transmission and further explore the deep spatial-spectral features of the hyperspectral image.

[0007] Step 3: The LiDAR data of the other branch is passed through 2D-2D convolution and improved Swin Transformer network to mine the elevation information of the LiDAR data;

[0008] Step 4: The deep features extracted from the hyperspectral and LiDAR branches are interacted through CAM (Cross attention module). By establishing a dynamic attention mechanism, the correlation information between the two modalities is effectively captured and fused into a unified feature representation.

[0009] Step 5: Send the hyperspectral and LiDAR features after interactive information into MLP (Multilayer Perceptron), and finally obtain the collaborative classification result of bimodal data.

[0010] The above-mentioned dual-branch Swin Transformer hyperspectral image and LiDAR data collaborative classification method, step 2 specifically includes the following steps:

[0011] Step 21: In order to eliminate information loss during the transmission process, the spectral features extracted in step 1 are passed through the ResNet structure, which consists of three stages. Each stage consists of a conv2D convolution, ReLU, another conv2D and a residual structure.

[0012] Step 22: The obtained hyperspectral features are spliced ​​on the window through SSR (Spectral Segmentation Recombination) and the weights are calculated for weighted processing;

[0013] Step 2 and 3: Send the weighted spliced ​​features to the SSP (Spatial Pyramid Pooling) module to fuse local features and global features;

[0014] Step 24: Use Swin Transformer to calculate the attention within the window and the attention of the sliding window to obtain the largest possible global receptive field while reducing calculations.

[0015] In the above-mentioned dual-branch Swin Transformer method for collaborative classification of hyperspectral images and LiDAR data, step three specifically includes the following steps:

[0016] Step 31: Extract shallow features from LiDAR data through 2D-2D convolution;

[0017] Step 32: Use the extracted shallow features to extract spatial features and elevation information through the improved Swin Transformer.

[0018] The above-mentioned dual-branch Swin Transformer hyperspectral image and LiDAR data collaborative classification method, step 4 specifically includes the following steps:

[0019] Step 41: Before entering the Attention layer, add an additional class token to the serialized features, and the hyperspectral branch is HSI cls token, LiDAR branch is LiDAR cls Token, cls token will participate in the training together with all tokens. After the training is completed, cls token is extracted separately for classification.

[0020] Step 4.2: Mark the features of LiDAR cls Token、HSI cls The token is connected with LiDAR features and hyperspectral features and global dependencies are captured through multi-headed self-attention;

[0021] Step 43: LiDAR cls The token is projected to the same dimension as the hyperspectral feature using a size-aligned linear projection function, and then concatenated and attention is calculated;

[0022] Step 4. HSI cls The token is projected to the same dimension as the LiDAR feature using a size-aligned linear projection function, and then concatenated with the LiDAR feature and self-attention is calculated.

[0023] In the above-mentioned dual-branch Swin Transformer method for collaborative classification of hyperspectral images and LiDAR data, step five specifically includes the following steps:

[0024] Step 51: weighted splicing of the hyperspectral and LiDAR marker features for feature exchange;

[0025] Step 52: Classify the output through MLP, which consists of LN (Layer Normalization) and a fully connected layer, and uses ReLU activation function to obtain the final classification result.

[0026] Beneficial effects:

[0027] The invention discloses a dual-branch Swin Transformer hyperspectral image and LiDAR data collaborative classification method, the method comprising: S1, according to the "graph-spectrum integration" characteristic of hyperspectral image, it is proposed to use 3D-2D hybrid convolution to extract shallow spatial-spectral information of hyperspectral image after principal component analysis dimension reduction. S2, an improved Swin Transformer network is adopted, ResNet module is adopted in the network, and SSR (Spectral Segmentation Recombination) is introduced to eliminate information loss in feature transmission, and further mine deep spatial-spectral features of hyperspectral image. S3, the LiDAR data of another branch is mined through 2D-2D convolution and improved Swin Transformer network to mine elevation information; S4, according to the dual branches of hyperspectral and LiDAR, CAM (Cross attention module) is improved to dual-branch input, through the interaction of hyperspectral and LiDAR features, by establishing a dynamic attention mechanism, the correlation information between the two modalities is effectively captured, and it is fused into a unified feature representation. The network was tested on two public hyperspectral and LiDAR datasets and showed good classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the collaborative classification method of hyperspectral image and LiDAR data using the dual-branch Swin Transformer in the method of the present invention.

[0029] Figure 2 It is a schematic diagram of the principle of the collaborative classification method of hyperspectral image and LiDAR data using the dual-branch Swin Transformer in the method of the present invention.

[0030] Figure 3 It is a schematic diagram of the SPP (Spatial Pyramid Pooling) principle in the method of the present invention.

[0031] Figure 4 It is a schematic diagram of the principle of the Swin Transformer block in the method of the present invention.

[0032] Figure 5It is a schematic diagram of the CAM (cross attention module) principle in the method of the present invention.

[0033] Figure 6 This is a diagram of the classification results of the Houston 2013 data set in the method of the present invention.

[0034] Figure 7 It is a classification result diagram of the Trento data set in the method of the present invention. DETAILED DESCRIPTION

[0035] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0036] The flow chart of the collaborative classification method of hyperspectral image and LiDAR data using dual-branch Swin Transformer in this specific implementation is as follows: Figure 1 As shown, the network principle diagram is as follows Figure 2 As shown, the following steps are included:

[0037] Step 1: Use the principal component analysis method to remove the spectral redundancy of the hyperspectral image, and then use the 3D-2D hybrid convolution to fully extract the shallow spatial-spectral features of the hyperspectral image; specifically:

[0038] In this specific implementation, two public datasets are used, namely the Houston 2013 dataset and the Trento dataset.

[0039] (1) The Houston2013 dataset was provided by the 2013 IEEE GRSS data fusion competition. This dataset covers the University of Houston campus and the surrounding urban area, and depicts 15 common urban features. The HSI data contains 144 spectral bands with spectral resolutions ranging from 0.38 to 1.05 μm. For the same area, there is only one band of LiDAR data. Both HSI and LiDAR data consist of 349×1905 pixels with a spatial resolution of 2.5 m.

[0040] (2) The Trento dataset was taken in a rural area south of Trento, Italy. The HSI data was acquired by the AISA Eagle sensor of the AISA Eagle system, with 63 spectral bands and spectral resolution from 0.42 to 0.99 μm, while the LiDAR data was collected by the Optech ALTM 3100EA sensor. The dataset size is 600×166 pixels, with a spatial resolution of 1 m, and contains six types of ground objects.

[0041] Step 2: The shallow spatial-spectral features of the hyperspectral image are fed into the improved Swin Transformer network, which adds the ResNet module and SSR to the network to eliminate the information loss in feature transmission and further mine the deep spatial-spectral features of the hyperspectral image; specifically:

[0042] Step 21: In order to eliminate information loss during the transmission process, the spectral features extracted in step 1 are passed through the ResNet structure, which consists of three stages. Each stage consists of a conv2D convolution, ReLU, another conv2D and a residual structure.

[0043] The application of traditional Transformer in the field of image processing mainly refers to the self-attention mechanism, which models the correlation of pixels in the image and extracts features through the attention mechanism. Swin Transformer is mainly aimed at low-dimensional and large-size images, while remote sensing images are high-dimensional and deep-level images. In order to more deeply extract the correlation between the deep-level feature information of remote sensing images, two-layer convolution and residual structure are used to extract spectral features.

[0044] Step 22: The obtained hyperspectral features are spliced ​​on the window through SSR (Spectral Segmentation Recombination) and the weights are calculated for weighted processing;

[0045] The extracted high-dimensional features are windowed to facilitate the subsequent calculation of attention in the windowed form. After any given information enters the network module, it is transformed as shown in the following formula.

[0046]

[0047]

[0048] In the formula, F(.) represents a split operation (for the split on the input feature matrix dimension), Respectively represent the image features after segmentation, It represents the result of splicing the segmented feature matrix. The input image features originally have spectral hierarchical features. Through dimension segmentation and windowed splicing, the high-dimensional features of the features are flattened into low-dimensional features, and the high-dimensional feature information is retained.

[0049] Step 2 and 3: Send the weighted spliced ​​features to the SPP (Spatial Pyramid Pooling) module to fuse local features and global features; the structure diagram is as follows Figure 3 As shown, the details are as follows:

[0050] The obtained features are processed by convolution and then processed by maximum pooling with kernels of sizes 5×5, 9×9, and 13×13 to obtain the receptive field of the local area of ​​the feature map and the receptive field close to the global area, and then feature fusion is performed through convcat. This operation of fusing receptive fields of different scales can effectively enrich the expression ability of the feature map, enhance the acceptance range of the output features of the backbone network, and separate important context information. The specific formula is as follows:

[0051] Y out =concat(conv(X in ),mp 5×5 (conv(X in )),mp 9×9 (conv(X in )),mo 13×13 (conv(X in ))

[0052] Where X in represents the input feature map, Y out It represents the output feature map after SSP processing, conv represents convolution processing, and mp represents the maximum pooling processing of the corresponding kernel size.

[0053] Step 24: Use the Batch Normalization (BN) layer and W-MSA to window the image features and extract features through the attention mechanism. W-MSA stands for the Window-based Multi-head Self-Attention mechanism. The output features are then processed by LN and MLP. W-MSA stands for the Window-based Multi-head Self-Attention mechanism, which only considers the input within a part of the local window when calculating self-attention. The flattened features of the obtained spectral information are divided into 7×7 windows. For each window, a multi-head self-attention calculation is performed to obtain the correlation score of each element in the window with other elements in the window. These scores are called "attention weights". Since the number of elements in each window is relatively small, such calculations can greatly reduce the computational complexity. The calculation amount of W-MSA is as follows:

[0054]

[0055] In the formula, h×ω represents the length and width of the input image, M represents the size of the partition window, and C represents the number of channels. It can be seen from the above formula that the window mechanism reduces the amount of calculation from the square relationship to the linear relationship related to the window size, which greatly reduces the amount of calculation.

[0056] Step 25: After the obtained features are processed by the LN layer, the shifted-window multi-head self-attention layer (SW-MSA) is used to further extract deep features, and then the features are passed through the LN layer and the MLP layer. The structure of SW-MSA is shown in the figure. The specific method of SW-MSA is to shift the windows divided in step 44 to the left and right by window size / 2 units, and then re-join the divided windows into new windows and re-extract features through the multi-head self-attention mechanism. The principle diagram of Swin Transformer block is shown in the figure. Figure 4 shown.

[0057] Step 3: The LiDAR data of the other branch is passed through 2D-2D convolution and improved Swin Transformer network to mine the elevation information of the LiDAR data;

[0058] Step 4: The deep features extracted from the hyperspectral and LiDAR branches are interacted through CAM (Cross attention module). By establishing a dynamic attention mechanism, the correlation information between the two modalities is effectively captured and fused into a unified feature representation; the principle diagram is shown in the figure. Figure 5 As shown, specifically:

[0059] Step 41: Before entering the Attention layer, add an additional class token to the serialized features, and the hyperspectral branch is HSI cls token, LiDAR branch is LiDAR cls Token, cls token will participate in the training together with all tokens. After the training is completed, cls token is extracted separately for classification.

[0060] Step 42: First, use the classification token in the HSI branch as a proxy to exchange information with the patch token in another branch, and then project it back to its own branch. Because the classification token can represent the abstract information of all patch tokens in its own branch, interacting with patch tokens in other branches helps to exchange information in a context-aware manner, thereby enriching the representation of each patch token. First, connect the hyperspectral classification token with the LiDAR patch, and its expression is as follows:

[0061]

[0062] Represents the classification token from the hyperspectral spectrum, Represents the classification token from the LiDAR feature.

[0063] Step 43: For x HSI′ and The multi-head self-attention mechanism is used for calculation, and the formula is as follows:

[0064] K = x HSI′ W k ,V=x HSI′ W v

[0065]

[0066] Among them, W q , W k , W k are three learnable parameter matrices, C is the embedding feature dimension, and h corresponds to the number of heads. The classification token processed by the multi-head self-attention mechanism is then concatenated with the hyperspectral patch token. The same operation is then performed on the LiDAR features. The specific formula is as follows:

[0067]

[0068]

[0069] The use of the cross-attention module allows the hyperspectral features and LiDAR features to no longer be simply fused, but to exchange classification tokens before performing attention calculations. Through this method, the obtained features not only contain single-branch features, but also integrate the receptive field of another branch, greatly improving the recognition accuracy.

[0070] Step 5: Send the hyperspectral and LiDAR features after interactive information into a multi-layer perceptron, and finally obtain the collaborative classification results of the bimodal data.

[0071] To verify the effectiveness of the algorithm, the experiment was conducted on the Windows 10 operating system, and the classification method was implemented using Python language and PyTorch library. The experimental environment is a Gen Intel(R) Core(TM) i5-12400F 2.50GHz processor, 48GB memory and GeForce GTX 3060Ti graphics card. The classification accuracy of the Houston2013 dataset and the Trento dataset are shown in Table 1 and Table 2, respectively. Three classification evaluation indicators are used: OA (overall accuracy), AA (average accuracy) and Kappa coefficient. The classification results of the Houston2013 dataset and the Trento dataset are respectively Figure 6 , 7 The experimental results show that the method proposed in the present invention can make full use of the rich information of hyperspectral and LiDAR to fully train the network, improve the fitting ability and robustness of the network, thereby improving the classification accuracy of the network and greatly reducing the area of ​​misclassification.

[0072] Table 1 Houston2013 dataset sample classification accuracy (%)

[0073]

[0074] Table 2 Comparison of classification accuracy of samples in the Trento dataset (%)

Claims

1. A dual-branch Swin Transformer method for collaborative classification of hyperspectral images and LiDAR data, characterized in that: The following steps are involved: Step 1: Use the principal component analysis method to remove the spectral redundancy of the hyperspectral image, and then use the 3D-2D hybrid convolution to fully extract the shallow spatial-spectral features of the hyperspectral image; Step 2: The shallow spatial-spectral features of the hyperspectral image are sent to the improved Swin Transformer network, which adds the ResNet module and SSR (Spectral Segmentation Recombination) to the network to eliminate the information loss in feature transmission and further explore the deep spatial-spectral features of the hyperspectral image. Step 3: The LiDAR data of the other branch is passed through 2D-2D convolution and improved Swin Transformer network to mine the elevation information of the LiDAR data; Step 4: The deep features extracted from the hyperspectral and LiDAR branches are interacted through CAM (Cross attention module). By establishing a dynamic attention mechanism, the correlation information between the two modalities is effectively captured and fused into a unified feature representation. Step 5: Send the hyperspectral and LiDAR features after interactive information into MLP (Multilayer Perceptron), and finally obtain the collaborative classification result of bimodal data.

2. The method for collaborative classification of hyperspectral imagery and LiDAR data using a dual-branch Swin Transformer according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 21: In order to eliminate information loss during the transmission process, the spectral features extracted in step 1 are passed through the ResNet structure, which consists of three stages. Each stage consists of a conv2D convolution, ReLU, another conv2D and a residual structure. Step 22: The obtained hyperspectral features are spliced ​​on the window through SSR, and the weights are calculated for weighted processing; Step 2 and 3: Send the weighted spliced ​​features to the SSP (Spatial Pyramid Pooling) module to fuse local features and global features; Step 24: Use Swin Transformer to calculate the attention within the window and the attention of the sliding window for the obtained features, so as to obtain the largest possible global receptive field while reducing the calculation.

3. The method for collaborative classification of hyperspectral image and LiDAR data using dual-branch Swin Transformer according to claim 1, characterized in that: Step three specifically includes the following steps: Step 31: Extract shallow features from LiDAR data through 2D-2D convolution; Step 32: Use the extracted shallow features to extract spatial features and elevation information through the improved Swin Transformer module.

4. A collaborative classification method of hyperspectral image and LiDAR data based on a dual-branch Swin Transformer, characterized in that: Step 4 specifically includes the following steps: Step 41: Before entering the Attention layer, add an additional class token to the serialized features, and the added hyperspectral branch is HSI cls token, LiDAR branch is LiDAR cls Token, class token will participate in the training together with all tokens. After the training is completed, the cls token is extracted separately for classification. Step 4.2: Mark the features of LiDAR cls Token、HSI cls The token is connected with LiDAR features and hyperspectral features and global dependencies are captured through multi-headed self-attention. Step 43: LiDAR cls The token is projected to the same dimension as the hyperspectral feature using a size-aligned linear projection function, and then concatenated and attention is calculated; Step 4. HSI cls The token is projected to the same dimension as the LiDAR feature using a size-aligned linear projection function, and then concatenated with the LiDAR feature and self-attention is calculated.

5. The method for collaborative classification of hyperspectral imagery and LiDAR data using a dual-branch Swin Transformer according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 51: weighted splicing of feature interactive hyperspectral and LiDAR marker features; Step 52: Classify the output through MLP, which consists of layer normalization and a fully connected layer, and uses ReLU activation function to finally obtain the classification result.

Images