Hyperspectral and lidar image cross-scene classification method with semi-central attention

Abstract

The application discloses a hyperspectral and laser radar image cross-scene classification method based on semi-central attention, and comprises the following steps: step S1, a semi-central attention module is designed; step S2, an encoder and a decoder are constructed by taking the semi-central attention module as a basic unit, and the encoder and the decoder are combined into a hyperspectral and laser radar image self-learning network; step S3, an HSI information embedding module is constructed by taking a hyperspectral image as input, a position information embedding module is constructed by taking a pixel position as input, and an elevation information embedding module is constructed by taking a laser radar image as input; the HSI information embedding module, the position information embedding module and the elevation information embedding module are simply superimposed and fused as input of the encoder and the decoder; and the application can obtain robust features independent of a scene, thereby improving cross-scene classification effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of radar image cross-scene, and particularly relates to a hyperspectral and laser radar image cross-scene classification method based on semi-central attention. BACKGROUND

[0002] The information processing of hyperspectral image recognition is based on the characteristics of hyperspectral data itself. First, the continuous and narrow spectral resolution characteristics make hyperspectral data contain rich spectral information, which enhances the ability of fine classification of ground objects. However, in addition to the noise introduced in the acquisition process, due to the difference in ground object composition, changes in lighting conditions, environment, atmospheric conditions, neighborhood effects and other factors, there is a spectral drift phenomenon in hyperspectral data, that is, there is a non-noise spectral difference between the same type of ground objects. Therefore, the cross-scene classification based on hyperspectral images is not satisfactory.

[0003] The purpose of cross-scene classification research is to reduce cross-scene differences, obtain robust features independent of scenes, and improve cross-scene classification accuracy. Hyperspectral image cross-scene classification is a sub-topic of domain adaptation. For the research on domain adaptation problem, researchers have proposed many methods. These methods can be roughly divided into four categories: statistical distribution alignment method, geometric structure alignment method, active learning method and deep learning method. Some deep learning frameworks are combined to design optimization objectives to eliminate the data differences between the source domain and the target domain. However, the existing methods do not consider the real source of the difference-spectral drift, so these methods cannot fundamentally reduce the cross-scene difference.

[0004] In order to reduce the cross-scene difference caused by spectral drift, information-stable laser radar images can be used to assist hyperspectral image cross-scene classification. By measuring the time difference between the emission and reception of laser, the active imaging laser radar provides elevation information, which is not easily affected by lighting conditions, can resist cloudy weather, is not easily disturbed by the atmosphere, and has less difference in cross-scene, so it has an information stability advantage compared with hyperspectral images. In addition, the elevation information also has certain ground object recognition ability. For example, the main component of grass and trees is chlorophyll, and the spectral curve shows similarity, but the height of the two is different, the height of grass is close to the ground height, and the height of trees is usually several meters to tens of meters, so grass and trees can be distinguished by the elevation information provided by the laser radar image. Therefore, through the cooperation of hyperspectral images and laser radar images, the stability of laser radar images can be used to suppress spectral drift caused by sensor parameter differences and ground object spatial distribution differences and other unrepresentative factors, and the discrimination performance of hyperspectral images and laser radar images is integrated, and finally a good collaborative cross-scene classification effect is obtained.

[0005] However, the cross-scene classification based on hyperspectral and laser radar images also faces some difficulties. First, the hyperspectral image and the laser radar image are heterogeneous with each other, how to effectively fuse the information of the two to obtain features conducive to classification is the first challenge. Second, although the laser radar image has the characteristic of information stability compared with the hyperspectral image, the information amount is low. How to use the laser radar image with low information amount to assist the hyperspectral image with rich information, suppress spectral drift, reduce cross-scene differences, and obtain robust features independent of the scene, and obtain high-precision cross-scene classification effect is also a big challenge. SUMMARY

[0006] The present application proposes a hyperspectral and laser radar image cross-scene classification method based on semi-central attention, which can ensure that robust features independent of the scene are obtained, thereby improving the cross-scene classification effect.

[0007] The technical scheme of the present application is as follows:

[0008] The hyperspectral and laser radar image cross-scene classification method based on semi-central attention comprises the following steps:

[0009] Step S1: design a semi-central attention module;

[0010] Step S2: take the semi-central attention module as a basic unit, construct an encoder and a decoder, and combine the encoder and the decoder into a hyperspectral and laser radar image self-learning network HLNet;

[0011] Step S3: construct an HSI information embedding module with a hyperspectral image as input, a position information embedding module with a pixel position as input, and an elevation information embedding module with a laser radar image as input; and simply superimpose and fuse the HSI information embedding module, the position information embedding module, and the elevation information embedding module as the input of the encoder and the decoder;

[0012] Step S4: form a self-supervised training by using the spectral correlation and spatial correlation of the module, use all data of the source scene and the target scene as training samples to perform self-supervised training on the HLNet, and obtain a trained encoder;

[0013] Step S5: connect the trained encoder with a classifier, fix the parameters of the encoder, then train the encoder with the labeled data of the source scene as training samples, and obtain a classification model of the target scene;

[0014] Step S6: classify the remote sensing image of the target scene by using the trained classification model of the target scene.

[0015] Preferably, step S1 comprises the following substeps:

[0016] Step S11: constructing a pixel feature extraction module, which includes a 1x1 convolutional layer, a normalization layer composed of BatchNorm, and an activation layer composed of ReLU, for extracting the features of the pixels;

[0017] Step S12: constructing a pixel key extraction module, which includes a 1x1 convolutional layer and a normalization layer composed of BatchNorm; by performing multi-head grouping on the obtained pixel keys, performing a corresponding inner product, and activating using a softmax function, the final pixel relationship weight is obtained.

[0018] Step S13: calculating the mixed feature. The features obtained by the feature extraction module and the pixel relationship weight obtained by the key extraction module are multiplied to obtain the final mixed pixel feature.

[0019] Preferably, step S2 includes the following sub-steps:

[0020] Step S21: constructing an encoder, which is composed of a plurality of semi-central attention modules connected in series, and the number of semi-central attention modules included can be any integer value greater than 1. The input of the encoder is the superposition of the HSI information embedding, the position information embedding, and the height information embedding. In the self-supervised training process, all pixel features output by the encoder will be used as the input of the decoder. In the supervised training process, the central pixel feature output by the encoder will be used as the input of the classifier.

[0021] Step S22: constructing a decoder, which is composed of a plurality of semi-central attention modules connected in series, and the number of semi-central attention modules included can be any integer value greater than 1. The input of the decoder is the depth feature embedding, the height information embedding, the position information embedding, and the label missing information embedding. The output of the decoder is an image with a size consistent with the spatial size of the hyperspectral image and the number of spectral bands.

[0022] Preferably, step S3 includes the following sub-steps:

[0023] Step 31: constructing an HSI information embedding module, which includes a 1x1 convolutional layer;

[0024] Step 32: constructing a position information embedding module, which is composed of trainable parameters, and has the same spatial size and channel number as the output of the HSI information embedding module;

[0025] Step 33: constructing a height information embedding module, which includes a 3x3 convolutional layer;

[0026] Step 34: constructing a missing label information embedding module; the module is composed of trainable parameters, and has the same spatial size and channel number as the output of the HSI information embedding module.

[0027] Preferably, step S4 comprises the following sub-steps:

[0028] Step S41: the spectral correlation self-supervised learning is to set the spectral values of part of the bands of all pixels in the image to 0, and restore these spectra;

[0029] Step S42: the spatial correlation self-supervised learning is to remove part of the pixels in the image, and restore the spectral vectors of these pixels;

[0030] Step S43: the spectral correlation self-supervised learning and the spatial correlation self-supervised learning are performed synchronously, both of which take the deep features, the position information and the height information embedded by the encoder as inputs, and restore the original spectral vectors by using the decoder, and in the process of the correlation self-supervised learning, the loss function is

[0031] Loss = Loss se + Loss sa

[0032] wherein Loss represents the total loss, Loss se represents the mean square error between the original band spectral values set to 0 and the band spectral values restored by the model, and Loss sa represents the mean square error between the spectral values of the removed original bands and the spectral values of the restored bands.

[0033] Preferably, the center pixel features obtained by the encoder in step S5 are the inputs of the classifier, and the final outputs of the center pixels are the logical values, and these output logical values are the training samples of the label data of the source scene, and the classifier is trained by the following loss function, and the expression is:

[0034]

[0035] wherein n s is the number of the label data of the source scene, y s is a category hot code vector, the value of the corresponding category position is 1, and the rest is 0, and r s is the logical value output by the classifier.

[0036] Preferably, step S6 comprises the following sub-steps:

[0037] Step S61: the classification model of the target scene is trained and classified by the self-supervised learning and the supervised learning, and this way comprises: image cropping: cropping the hyperspectral and lidar image block with the pixel to be classified as the center, and the size of the image block is an odd number;

[0038] Step S62: after prediction, the output results of step S61 are filled into the classification map of the hyperspectral image according to the position.

[0039] The semi-central attention hyperspectral and laser radar image cross-scene classification method has the following advantages:

[0040] 1. The method proposes a complete hyperspectral and laser radar image joint cross-scene classification model, which can use the label data of the source scene and the data of the target scene to classify the target scene and obtain accurate classification results, thereby reducing the labeling work of the target scene.

[0041] 2. The method proposes a semi-central attention module, which extracts features of target pixels in the block while considering the features of surrounding pixels, which is beneficial to the classification of the pixels.

[0042] 3. The method proposes a spectrum and space correlation self-built learning, which can effectively learn the spectral correlation and spatial correlation information of the hyperspectral image, and is beneficial to learning the robust features independent of the scene and improving the cross-scene classification effect. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a classification flowchart of the semi-central attention hyperspectral and laser radar image cross-scene classification method of the application;

[0044] Figure 2 is an overall structure diagram of encoding, decoding and classification constructed by the application;

[0045] Figure 3 is an encoder and decoder constructed by the application;

[0046] Figure 4 is a semi-central attention module structure diagram constructed by the application;

[0047] Figure 5 is a spectrum correlation self-supervised learning process constructed by the application;

[0048] Figure 6 is a space correlation self-built learning process constructed by the application;

[0049] Figure 7 is one of the application examples, which contains hyperspectral images, laser radar images and label images of source scenes and target scenes;

[0050] Figure 8 is a classification effect diagram of the trained encoder-classifier on the target scene in the example. DETAILED DESCRIPTION

[0051] The application will be further illustrated below in combination with specific examples and drawings. The embodiments are implemented on the premise of the technical scheme of the application, and it should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application.

[0052] As Figure 1 shown, the hyperspectral and laser radar image cross-scene classification method with semi-central attention includes the following steps:

[0053] Step 1: build a semi-central attention module, and build an encoder and a decoder with the semi-central attention module as a basic unit;

[0054] Step 2: input the data of the source scene and the target scene after being cropped into the encoder-decoder for spectral correlation and spatial correlation self-supervised learning training;

[0055] Step 3: save the encoder parameters obtained by training, and load them into the encoding-classifier;

[0056] Step 4: connect the trained encoder to the classifier, fix the parameters of the encoder, then train the encoder with the labeled data of the source scene as training samples, and finally obtain the classification model of the target scene;

[0057] Step 5: apply the trained encoding-classifier to the target scene to classify the target scene and obtain the classification result.

[0058] Specifically, step 1 includes the following steps:

[0059] Step 1.1: build a pixel feature extraction branch, which includes a 1×1 convolution layer, a BatchNorm normalization, and an activation layer, and its calculation process includes the following formulas:

[0060] I out1 =conv(I in )#(1)

[0061] I out2 =BatchNorm(I out1 )#(2)

[0062] I out3 =ReLU(I out2 )#(3)

[0063] Where I in is the input of the branch, I out3 is the output of the branch, conv is a one-dimensional convolution operation with a convolution kernel size of 1×1 and a step size of 1; BatchNorm is a batch normalization function that normalizes the features of each channel, and the formula is:

[0064]

[0065]

[0066]

[0067]

[0068] ReLU is a ReLU function, and the formula is:

[0069] S(x) = max(0, x) # (8)

[0070] Step 1.2: Construct a pixel key extraction branch, which is a 1x1 convolutional layer, a BatchNorm normalization, and an inner product operation, for obtaining the relationship weight between pixels. The operation formula of the branch is:

[0071] I out4 = conv(I in ) # (9)

[0072] I out5 = BatchNorm(I out5 ) # (10)

[0073]

[0074] In the formula, I in is the input of the branch, W is the output of the branch, and d is the feature channel number of the pixel; SoftMax is a softmax function, and the formula is:

[0075]

[0076] Step 1.3: Construct a pixel home aliasing module, and the operation formula of the module is:

[0077] I out = I out3 * W # (13)

[0078] I out3 is the output of step 1.1 branch, W is the output of step 1.2 branch, and the two outputs are the inputs of the module, I out is the output of the module.

[0079] As Figure 2 shown, specifically, step 2 includes the following steps:

[0080] Step 2.1: Obtain the waveband that needs to be set to 0 using a random seed, and set the corresponding waveband of the hyperspectral data of the source scene and the target scene to 0;

[0081] Step 2.2: On the basis of step 2.2, obtain the pixel position that needs to be removed using a random seed, and remove the corresponding pixel position of the hyperspectral data of the source scene and the target scene;

[0082] As Figure 3 shown, specifically, step 2 includes the following steps:Figure 4 Step 2.3: initialize the parameters of the encoder-decoder as shown in FIG. 2B;

[0083] As shown in FIG. 2C, step 2.4: construct the spectral correlation and spatial correlation self-supervised learning loss function. In the correlation self-supervised learning process, the loss function is Figure 5 Figure 6 As shown in FIG. 2C, step 2.4: construct the spectral correlation and spatial correlation self-supervised learning loss function. In the correlation self-supervised learning process, the loss function is

[0084] Loss = Loss se + Loss sa

[0085] wherein Loss represents the total loss, Loss se represents the mean square error between the original band spectral value with the value set to 0 and the model recovered band spectral value, and Loss sa represents the mean square error between the spectral value of the original band and the spectral value of the model recovered band.

[0086] Step 2.5: set the training times, iterators and other training tools, input the hyperspectral data and lidar data of the source scene and the target scene with random bands set to 0 and random pixels removed into the encoder-decoder for training, so as to obtain the trained encoder.

[0087] Specifically, step 3 includes the following steps:

[0088] Step 3.1: save the trained encoder network parameters;

[0089] Step 3.2: construct an encoder-classifier and initialize the corresponding network parameters;

[0090] Step 3.3: load the trained encoder network parameters into the encoder-classifier, and make the network parameters of the encoder become untrainable parameters, i.e. not participate in the following training.

[0091] Specifically, step 4 includes the following steps:

[0092] Step 4.1: construct the data-label pair of the source scene;

[0093] Step 4.2: construct the training loss function of the encoder-classifier, and the specific loss function calculation formula is:

[0094]

[0095] wherein n s is the number of source scene label data, y s is a category hot code vector (the value of the corresponding category position is 1, and the rest is 0), and r s is the logic value output by the classifier.

[0096] ​Step 4.3: Set up the training tools such as the number of training iterations and iterators, input the data-label pairs from the source scene into the encoder-classifier for training, and finally obtain the trained encoder-classifier.

[0097] Specifically, step 5 includes the following steps:

[0098] Step 5.1: Use the trained encoder-classifier to classify the patch containing the test pixel in the target scene to obtain the corresponding classification result.

[0099] Step 5.2: Segment the classification results of each pixel to obtain a classification result image with the same spatial size as the hyperspectral image of the target scene. Figure 7 The imagery includes hyperspectral images of the source and target scenes (Nashua->Hanover), LiDAR images, and a ground truth map of the labels. Figure 8 This paper presents a label map generated on the Hanover dataset using an HLCS model trained on Nashua data, label data, and Hanover data, demonstrating the excellent cross-scene classification performance of the HLCS model.

Claims

1. A cross-scene classification method for hyperspectral and lidar images using semi-central attention, characterized in that, Includes the following steps: Step S1: Design a semi-central attention module; Step S2: Using the semi-central attention module as the basic unit, construct the encoder and decoder, and combine the encoder and decoder into the hyperspectral and lidar image self-learning network HLNet; Step S3: Construct an HSI information embedding module using hyperspectral images as input, construct a location information embedding module using pixel locations as input, and construct an elevation information embedding module using lidar images as input; The HSI information embedding module, the location information embedding module, and the elevation information embedding module are simply superimposed and fused as the input to the encoder and decoder; Step S4: Utilize the spectral and spatial correlations of the modules to form self-supervised training. Use all data from the source and target scenes as training samples to perform self-supervised training on HLNet and obtain the trained encoder. Step S5: Connect the trained encoder to the classifier, fix the encoder parameters, and then use the labeled data of the source scene as training samples to train the encoder to obtain the classification model of the target scene. Step S6: Classify the remote sensing image of the target scene using the trained target scene classification model; Step S1 includes the following sub-steps: Step S11: Construct a pixel feature extraction module, which includes a 1×1 convolutional layer, a normalization layer composed of BatchNorm, and an activation layer composed of ReLU, for extracting pixel features. Step S12: Construct a pixel key extraction module, which contains a 1×1 convolutional layer and a normalization layer composed of BatchNorm; by grouping the obtained pixel keys into multiple groups, performing the corresponding inner product, and activating them using the softmax function, the final pixel relationship weights are obtained. Step S13: Calculate the aliasing features. Multiply the features obtained by the feature extraction module and the pixel relationship weights obtained by the key extraction module by matrix multiplication to obtain the final aliased pixel features.

2. The semi-central attention hyperspectral and lidar image cross-scene classification method according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S21: Construct an encoder. The encoder is composed of semi-central attention modules connected in series. The number of semi-central attention modules it contains can be any integer value greater than 1. The input of the encoder is the superposition of HSI information embedding, position information embedding and elevation information embedding. During the self-supervised training process, all pixel features output by the encoder will be used as the input of the decoder. During supervised training, the center pixel feature of the encoded output will be used as the input to the classifier; Step S22: Construct a decoder. The decoder is composed of semi-central attention modules connected in series. The number of semi-central attention modules it contains can be any integer value greater than 1. Its input is depth feature embedding, elevation information embedding, location information embedding, and label missing information embedding. Its output is an image with the same spatial size and number of spectral bands as the hyperspectral image.

3. The semi-central attention hyperspectral and lidar image cross-scene classification method according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step 31: Construct the HSI information embedding module, which contains a 1×1 convolutional layer; Step 32: Construct a location information embedding module, which consists of training parameters and has the same spatial size and number of channels as the output of the HSI information embedding module; Step 33: Construct an elevation information embedding module, which contains a 3×3 convolutional layer; Step 34: Construct the missing label information embedding module; this module consists of trainable parameters, and its spatial size and number of channels are the same as the output of the HSI information embedding module.

4. The method for cross-scene classification of hyperspectral and lidar images with semi-central attention as described in claim 1, characterized in that, Step S4 includes the following sub-steps: Step S41: Spectral association self-supervised learning involves setting the spectral values ​​of all pixel bands in the image to 0 and recovering these spectra. Step S42: Spatial correlator-supervised learning involves removing some pixels from the image and recovering the spectral vectors of these pixels; Step S43: Spectral correlation self-supervised learning and spatial correlation sub-supervised learning are performed simultaneously. Both take the depth features, location information embedding, and elevation information embedding obtained by the encoder as input, and the decoder recovers the original spectral vector. In the correlation self-supervised learning process, the loss function is: ， in, Indicates the total loss. This represents the mean square error between the original band spectral values ​​set to 0 and the band spectral values ​​recovered by the model. This represents the mean square error between the spectral values ​​of the original band removed and the spectral values ​​of the band recovered by the model.

5. The semi-central attention hyperspectral and lidar image cross-scene classification method according to claim 1, characterized in that, In step S5, the center pixel features obtained by the encoder serve as the input to the classifier, ultimately outputting the logical values ​​of the center pixels. These output logical values ​​use the label data of the source scene as training samples, and the classifier is evaluated using the following loss function, the expression of which is: ， in, The number of source scene label data. This is a category hot code vector, where the value at the corresponding category position is 1, and the rest are 0. The logical value output by the classifier.

6. The semi-central attention hyperspectral and lidar image cross-scene classification method according to claim 1, characterized in that, Step S6 includes the following sub-steps: Step S61: The classification model for the target scene is trained and classified through self-supervised learning and supervised learning. This method includes: image cropping: cropping out hyperspectral and lidar image patches centered on the pixels to be classified. The size of the image patch is an odd number. Step S62: After prediction, the output of step S61 is filled into the classification map of the image space size and the hyperspectral image according to the position.

Images