Hyperspectral and laser radar image fusion classification method based on position information enhancement

Abstract

The invention discloses a hyperspectral and laser radar image fusion classification method based on position information enhancement, and belongs to the field of remote sensing image processing. The implementation method comprises the following steps of: marking pixels by using a preset window, marking the pixels in the window in the same way, marking different windows in sequence, and normalizing the windows to obtain absolute position coding maps corresponding to all the pixels; and performing sliding slicing on the hyperspectrum and laser radar image and the absolute position coding map to obtain a spatial spectrum cube, an elevation information cube and an absolute position information cube. Extracting abstract expressions from the three types of feature cubes through a convolutional layer, inputting the abstract expressions into an absolute position enhancement module, and outputting fusion features which have absolute position information enhancement and are fused with elevation information and spatial spectrum information; and pooling the output feature map of the convolutional layer to reserve prominent features and overall features, and splicing the prominent features and the overall features as supplementary information with the fusion features to obtain a final classification result through a full connection layer.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing and relates to a hyperspectral and laser radar image fusion classification method based on position information enhancement. Background Art

[0002] Hyperspectral images are high-dimensional data with spatial information and spectral features collected under hundreds of continuous spectral bands. Different materials in the image can be distinguished by the difference in spectral features. However, due to the lack of altitude information, it is impossible to distinguish the difference of the same material at different altitudes using only hyperspectral images, while LiDAR images can provide complementary elevation information for hyperspectral images. Therefore, making full use of the information in hyperspectral and LiDAR images can be effectively used for land cover classification.

[0003] With the rise of deep learning, especially convolutional neural networks, which can learn high-level semantic information and deep abstract representations in images, remote sensing image processing technology has been further developed. Nowadays, more and more deep learning algorithms are also used for hyperspectral and lidar image fusion classification. For example, Shen et al. used convolutional networks and channel shuffling methods in the paper "SFE-FN: A Shuffle Feature Enhancement-Based Fusion Network for Hyperspectral and LiDAR Classification. IEEE Geoscience and Remote Sensing Letters, 2023.3243036." to achieve the enhancement and interaction of hyperspectral and lidar image features. This paper focuses on the local receptive field features of the image, while Roy et al. proposed a multimodal fusion transformer (MFT) network inspired by vision transformers in the paper "Multimodal Fusion Transformer for Remote Sensing Image Classification. IEEE Transactions on Geoscience and Remote Sensing, 2023.3286826." The network starts from the global receptive field and uses a multi-head cross-patch attention mechanism to effectively fuse and classify hyperspectral and lidar image features. Wang et al. proposed a multiscale pyramid fusion framework in "MS2CANet: Multiscale Spatial–Spectral Cross-Modal Attention Network for Hyperspectral Image and LiDAR Classification. IEEE Geoscience and Remote Sensing Letters, 2024.3350633.", using multiple convolution branches to extract features of hyperspectral images and lidar images from different scales, and designed an effective feature recalibration module to increase the information interaction between the two modalities.

[0004] However, these algorithms currently ignore the absolute position information relationship of pixels. Most Transformer-based models use randomly generated matrices to represent the position relationship between pixels, but this method is difficult to achieve strong representation. Hyperspectral and LiDAR image fusion classification is a pixel-level classification method. The absolute position correlation of each pixel in the image can also be used as complementary information of the two modal image features, thereby improving the overall classification performance of the network model. In addition, in the process of extracting features from images using convolutional layers, some information will inevitably be lost, which will affect the final classification accuracy to a certain extent. The present invention performs maximum pooling and average pooling operations on the convolutional layers in the feature extraction process, thereby retaining the key features for the final classification of supplementary information. Summary of the invention

[0005] In order to solve the problem of insufficient utilization of absolute position correlation information between different pixels in the existing research on hyperspectral and lidar image fusion classification, the purpose of the present invention is to provide a hyperspectral and lidar image fusion classification method based on position information enhancement, which represents the position correlation between pixels through a pixel-level absolute position encoding method, and obtains an abstract representation with absolute position information characteristics through a convolutional neural network, and then enhances the absolute position characteristics of hyperspectral and lidar data through an absolute position enhancement module, thereby extracting multimodal fusion features with absolute position correlation information, spatial spectral information of hyperspectral images and elevation information of lidar images, so as to achieve effective classification of hyperspectral and lidar images.

[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is:

[0007] The hyperspectral and laser radar image fusion classification method based on position information enhancement disclosed in the present invention uses a window of preset size to mark pixels. The pixel marks in the window are the same, and different windows are marked in sequence. The marked matrix diagram is normalized, and finally the absolute position coding diagram corresponding to all pixels can be obtained. Then the hyperspectral and laser radar images and the absolute position coding diagram are mirror-filled and slidingly sliced ​​to obtain the spatial spectrum cube of the hyperspectral, the elevation information cube of the laser radar, and the absolute position information cube. Subsequently, an absolute position enhancement network APENet (Absolute Position Enhancement Network) model for the fusion classification of hyperspectral and laser radar images is constructed, which includes a feature extraction module, an absolute position enhancement module APEM (Absolute Position Enhancement Module), a supplementary information feature map extraction module, and a classification module. The APENet model extracts the abstract representation of the three types of feature cubes obtained by slicing through the convolution layer of the feature extraction module, and then inputs it into APEM to obtain the spatial spectrum feature matrix HE with three types of fusion information: spatial spectrum, elevation and absolute position; in the supplementary information feature map extraction module, the feature map output by the convolution layer is subjected to maximum pooling and average pooling operations to retain the prominent features and overall structural information of the two modal images as the supplementary information matrix for classification. Finally, the feature map HE and the supplementary information matrix are activated through the fully connected layer and softmax function in the classification module to obtain the final classification result.

[0008] The hyperspectral and laser radar image fusion classification method based on location information enhancement disclosed in the present invention comprises the following steps:

[0009] Step 1: Construct the absolute position coding map POS. First, set a matrix map of the same size as the hyperspectral image and a window pos_windows of a preset size. In the matrix map, all pixels are divided into different sub-regions in sequence using the pos_windows window, and the pixels in these sub-regions are marked in sequence as absolute position codes, and the pixels in the same sub-region are marked with the same code. At the same time, in order to eliminate the influence of excessive order of magnitude and simplify the calculation, the matrix map after coding is normalized. The coding map is normalized according to the following formula to reduce the amount of calculation:

[0010]

[0011] POS i,j represents the pixel value of the i-th row and j-th column in the coding image POS, where i = 1, 2, 3 ... H; j = 1, 2, 3 ... W; H, W represent the height and width of the hyperspectral image respectively; N(POS i,j) indicates that the pixel values ​​of the i-th row and j-th column of the matrix are normalized; while min(POS) and max(POS) respectively represent the minimum and maximum values ​​in the coding map POS. Finally, the absolute position coding map corresponding to all pixels in the hyperspectral image or lidar image is obtained.

[0012] Step 2: The absolute position coding map constructed in step 1 is mirrored, filled, and slide-sliced ​​together with the hyperspectral image and lidar image to obtain an absolute position information cube, a hyperspectral spatial spectrum cube, and a lidar elevation information cube.

[0013] Step three, construct an absolute position enhancement network APENet (Absolute Position Enhancement Network) model, and the APENet model includes a feature extraction module, an absolute position enhancement module APEM (Absolute Position Enhancement Module), a supplementary information feature map extraction module and a classification module. The feature extraction module is composed of four convolutional layers, and the structure of each convolutional layer is composed of two-dimensional convolution or three-dimensional convolution, batch normalization and GELU. APEM fuses and enhances various features through operations such as nonlinear activation functions, matrix products and softmax functions to obtain the spatial spectral feature map HE; the supplementary information feature map extraction module retains the prominent features and overall structural information of the feature map output by the convolutional layer through pooling operations, and uses it as supplementary information for classification; the classification module integrates the obtained HE and supplementary information and then processes them through the fully connected layer to obtain the final classification result;

[0014] The APENet model first extracts abstract representations from the three types of feature cubes obtained by slicing through a feature extraction module. Four layers of convolution layers consisting of two-dimensional convolution, batch normalization, and GELU are used to extract features from the elevation information cube and the absolute position information cube to obtain the lidar feature map L and the absolute position feature map P. For the spatial spectrum cube, one three-dimensional convolution layer and three two-dimensional convolution layers are used to fully extract the hyperspectral feature map H.

[0015] Step 4: Input the three types of features extracted in step 3 into the Absolute Position Enhancement Module (APEM) of the APENet model to fuse and enhance the information of the three types of features. The extracted feature map H contains rich spatial spectral features, the feature map L has rich elevation information, and the feature map P contains the absolute position information between pixels. The APEM module is used to fully explore the features of the absolute position correlation between each pixel in the hyperspectral image and the lidar image. By capturing the absolute position information and elevation features in the feature maps P and L, and using them to enhance the feature map H, H is enhanced with absolute position information and fused with elevation information in the lidar image to obtain a more compact and effective multimodal fusion feature.

[0016] First, the feature maps P and L are respectively passed through the Tanh function to introduce nonlinearity into the network model. The convolution layer used in the feature extraction process is a linear combination operation. The introduction of the nonlinear Tanh function can improve the nonlinear fitting ability of the module, thereby enhancing the expression ability of the APENet model. Then the size of the feature maps P and L is adjusted through the reshaping operation, because the size of the feature maps P and L obtained after the above convolution layer satisfies P, L∈R c×h×w , where c, h, and w represent the number of channels, height, and width of the feature map, respectively. Two feature maps of the same size cannot be used for subsequent matrix product operations, so their sizes need to be adjusted. The size of the feature map after reshaping is L∈R c×l , P∈R l×c , where l = h × w.

[0017] Step 5: Perform matrix product operation on the adjusted feature maps P and L. The size of the feature map PL after matrix product is R l×l , l represents the product of the width and height of the original input image. Then, the number of dimensions is reduced to reduce the amount of calculation by a one-dimensional convolution operation with an output channel of 1 and a convolution kernel size of 1. The convolution formula is as follows:

[0018]

[0019] In the above formula, Ψ(·) represents a one-dimensional convolution with depth, w∈R l×1×1 represents the weight parameter of Ψ(·), the depth of the one-dimensional convolution is l, the convolution kernel size is 1×1, and the feature map PL = {X1,X2,X3,…X l}, where X i ∈R l ={x1,x2,x3,…x l}.

[0020] Step 6: Use the reshaping operation to ensure that the feature map PL and the feature map H∈Rc×h×w The size of the aligned feature map is PL∈R 1×h×w , and then the weight parameters with absolute position information and elevation information are learned from the feature map PL through the softmax function, and then the fused and enhanced spatial spectral feature map HE is obtained through element-wise multiplication. The whole process is as follows:

[0021]

[0022] in Represents matrix product, "⊙" represents multiplication by corresponding elements, T(·) represents Tanh nonlinear function activation operation, and σ(·) represents softmax function. The obtained spatial spectral feature map HE is enhanced by the absolute position information of the feature map P, and integrates the multimodal fusion features of the elevation information in the lidar image feature map L and the spatial spectral information in the hyperspectral image feature map H.

[0023] Step 7: Use the maximum pooling and average pooling operations on the output feature map of the convolution layer of the feature extraction module in step 3 to retain the prominent features and overall feature information of the hyperspectral image and lidar image at different layers from shallow to deep, and obtain the hyperspectral S H With LiDAR S L The two feature maps are used as supplementary features for image classification. They are concatenated with the feature map HE obtained in step 6 and used in the classification module of the APENet model to obtain the classification probability. Finally, the obtained classification probability and label value are used in step 8 to train the APENet model based on position information enhancement through the cross entropy loss function.

[0024] Step eight, for the constructed APENet model based on location information enhancement, configure the parameters required for APENet model training and testing, train the APENet model, and perform hyperspectral and lidar image fusion classification based on the trained APENet model, that is, hyperspectral and lidar image fusion classification based on location information enhancement.

[0025] Beneficial effects:

[0026] 1. The hyperspectral and lidar image fusion classification method based on position information enhancement disclosed in the present invention uses a window of preset size to mark pixels. The pixels in the window have the same mark, and different windows are marked in sequence. The marked matrix image is normalized, and finally the absolute position coding image corresponding to all pixels can be obtained. The absolute position information cube is obtained by mirror filling and dynamic slicing, which provides the absolute position enhancement APENet model with rich features of absolute position correlation between different pixels, thereby effectively improving the classification accuracy of the APENet model.

[0027] 2. The hyperspectral and lidar image fusion classification method based on position information enhancement disclosed in the present invention aims to solve the problem that the existing model does not sufficiently utilize the absolute position correlation features between pixels. The absolute position information cube is added as the input auxiliary classification of the model. The absolute position enhancement module APEM fully mines and utilizes the features of the absolute position correlation between each pixel in the hyperspectral image and the lidar image. The hyperspectral feature map is enhanced by extracting the absolute position information and elevation information of the absolute position coding map and the lidar image, and a richer, more compact and effective multimodal fusion feature HE is obtained, thereby improving the APENet model's ability to distinguish between different materials in the image.

[0028] 3. The hyperspectral and lidar image fusion classification method based on location information enhancement disclosed in the present invention performs maximum pooling and average pooling operations in the feature map output by the convolution layer of the feature extraction module to retain the prominent features and overall structural information of the two modal images to obtain the hyperspectral S H With LiDAR S L The feature map is used as supplementary information for classification, and it is used with the fused and enhanced feature map HE in the classification module to output the classification probability. The classification probability and label value are then used to train the APENet model through the cross entropy loss function to improve the prediction accuracy of the APENet model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart of the hyperspectral and lidar image fusion classification method based on location information enhancement.

[0030] Figure 2 A schematic diagram of constructing an absolute position coding diagram according to an embodiment of the present invention.

[0031] Figure 3 The figure is an overall network structure diagram of an embodiment of the present invention.

[0032] Figure 4 FIG. 4 is a structural diagram of an absolute position enhancement module according to an embodiment of the present invention.

[0033] Figure 5 This is a structural diagram of a supplementary information feature map extraction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings to better understand the hyperspectral and laser radar image fusion classification method based on position information enhancement of the present invention.

[0035] Hyperspectral and LiDAR image fusion classification is a detailed pixel-level classification processing technology. It constructs rich feature representations by efficiently utilizing the complementary information between the two modal images, thereby achieving effective classification of objects. How to fully explore and utilize the useful information in hyperspectral images and LiDAR images is a key issue in classification. However, many hyperspectral and LiDAR image fusion classification methods only focus on the spectral and spatial information in the image, while ignoring the positional correlation information between each pixel in the image; at the same time, in feature extraction, especially in convolution operations, some details or important features may be lost, thus affecting the final recognition accuracy.

[0036] In order to solve the above technical challenges and improve the accuracy of hyperspectral and laser radar image fusion classification, an embodiment of the present invention provides a hyperspectral and laser radar image fusion classification method based on location information enhancement, such as Figure 1 The following is a basic flow chart of this embodiment, including:

[0037] 1. Construct a two-dimensional matrix image of the same size as the hyperspectral image, use a sliding window to encode the absolute position of the matrix image, and normalize the absolute position encoding image;

[0038] 2. Mirror and fill hyperspectral images, lidar images and absolute position coding maps and slide slice to construct spatial spectral cubes, elevation information cubes and absolute position information cubes;

[0039] 3. Input the three types of cubes into the feature extraction module to compress and extract information, and obtain three types of feature maps H, L and P of spatial spectrum, elevation information and absolute position information;

[0040] 4. Input the three types of feature matrices into the absolute position enhancement module to obtain a spatial spectrum feature matrix HE with enhanced absolute position information and fused with elevation information;

[0041] 5. In addition, a pooling operation is performed on the output feature map of the convolutional layer in the feature extraction module to extract the prominent features and overall structural features of the hyperspectral and lidar images as supplementary information for classification;

[0042] 6. The enhanced fused spatial spectral feature matrix HE and the supplementary information matrix are spliced ​​and classified to finally obtain the classification result.

[0043] like Figure 1 As shown, the hyperspectral and laser radar image fusion classification method based on location information enhancement disclosed in this embodiment has the following specific implementation steps:

[0044] Step 1: In order to make full use of the absolute position correlation information of each pixel in the image, an absolute position coding map is constructed to assist classification. Figure 2 As shown in the figure, first set a two-dimensional matrix image of the same size as the hyperspectral image. The hyperspectral image is W pixels wide and H pixels high, so the absolute position coding map also has H×W pixels. In the public dataset of hyperspectral and lidar image fusion classification, the hyperspectral image and the lidar image are of the same size, so only one absolute position coding map needs to be constructed. Then set the window pos_windows=7 to perform sliding coding on the two-dimensional matrix image. The specific coding method is to use the pos_windows window to slide slice the two-dimensional matrix image into different sub-areas. The pixels in the same sub-area are all coded in the same way, while the pixels in different sub-areas are coded in sequence, such as Figure 2 As shown, c and r satisfy the following relationship:

[0045]

[0046] in It means rounding up to ensure that each pixel has a corresponding absolute position code. Finally, the absolute position code map POS can be constructed. The maximum code value is c×r. Since the value of c×r is generally large, the code map POS is normalized according to the following relationship for the convenience of calculation:

[0047]

[0048] POS i,j Represents the pixel value of the i-th row and j-th column in the coded image POS, where i = 1, 2, 3 ... H; j = 1, 2, 3 ... W; N (POS i,j ) indicates that the pixel values ​​of the i-th row and j-th column of the matrix are normalized; while min(POS) and max(POS) respectively represent the minimum and maximum values ​​in the coding map POS. Finally, the absolute position coding map corresponding to all pixels in the hyperspectral image or lidar image is obtained.

[0049] Step 2: Construct a dataset for training and testing. Mirror and fill the public hyperspectral and lidar datasets and the absolute position coding map, and set a sliding window of patch_size=11 to crop and slice to obtain a hyperspectral cube, lidar cube, and absolute position information cube of size 11×11.

[0050] Step 3: Build the Absolute Position Enhancement Network (APENet) model. Figure 3As shown in the figure, the APENet model includes a feature extraction module, an absolute position enhancement module APEM (Absolute Position Enhancement Module), a supplementary information feature map extraction module and a classification module. The feature extraction module consists of four convolutional layers; APEM fuses and enhances various features through operations such as nonlinear activation functions, matrix products and softmax functions to obtain the spatial spectral feature map HE; the supplementary information feature map extraction module retains the prominent features and overall structural information of the feature map output by the convolution layer through pooling operations, and uses it as supplementary information for classification; the classification module integrates the obtained HE and supplementary information and processes them through the fully connected layer to obtain the final classification result.

[0051] Furthermore, the hyperspectral cube, lidar cube and absolute position information cube obtained by slicing are input into the feature extraction module for feature extraction. The lidar image cube and the absolute position information cube are respectively input into four layers of convolution to extract abstract representations L and P from shallow to deep. The convolution layer structure consists of two-dimensional convolution, batch normalization and GELU activation function. The number of channels of the four-layer convolution increases gradually from 4, 8, 16, and 32. The convolution kernel size is 3×3 and the convolution output size remains unchanged. Finally, the feature map L, P∈R is extracted. 32×11×11 , in the process of laser radar cube feature extraction, the feature after one layer of convolution And the features after two and three layers of convolution and It will be used in the subsequent supplementary information feature map extraction module to extract supplementary information.

[0052] For the hyperspectral image cube, one 3D convolution layer and three 2D convolution layers are set to extract the important feature H. The 3D convolution layer consists of 3D convolution, batch normalization, GELU activation function, maximum pooling, and average pooling. The 2D convolution layer structure is the same as above, and the number of channels is gradually reduced to 128, 64, and 32, and the convolution output size remains unchanged. After one layer of two-dimensional convolution and two layers of two-dimensional convolution, we get and Finally, after three layers of two-dimensional convolution, the hyperspectral feature map H∈R 32×11×11 The whole feature extraction process of the three types of original cubes, namely hyperspectral, lidar and absolute position information, is as follows: Figure 3 As shown in the feature extraction module in .

[0053] Step 4: The three types of abstract representations H, L and P extracted above are input into the Absolute Position Enhancement Module (APEM) to fully exploit the absolute position correlation characteristics between each pixel in the hyperspectral image and the lidar image, thereby fusing and enhancing the information of the three types of feature maps. Figure 4 As shown in the figure, first, the feature map P and the feature map L are respectively subjected to the Tanh function to introduce nonlinearity into the network model. The convolution layer used in the feature extraction process is a linear combination operation. The introduction of the nonlinear Tanh function can improve the nonlinear fitting ability of the module and enhance the expression ability of the APENet model.

[0054] Then the size of the feature map is adjusted by reshaping operation, because the size of the feature map P and feature map L obtained after the above convolution layer satisfies P, L∈R 32×11×11 , two feature maps of the same size cannot perform subsequent matrix product operations, so their sizes need to be adjusted. The size of the reshaped feature map is L∈R 32×121 ,P∈R 121×32 .

[0055] Step 5: Perform matrix product operation on the adjusted feature maps P and L:

[0056]

[0057] in, represents matrix product, T(·) represents Tanh nonlinear function activation operation; the feature map R after matrix product L Size R 121×121 .

[0058] Furthermore, in order to reduce the number of parameters and computation of the APENet model, it is necessary to reduce the number of dimensions through a one-dimensional convolution operation with an output channel number of 1 and a convolution kernel size of 1. The formula is as follows:

[0059]

[0060] Among them, Ψ(·) represents a one-dimensional convolutional layer with depth, w∈R 121×1×1 represents the weight parameter of Ψ(·), the depth of the one-dimensional convolution is 121, the convolution kernel size is 1×1, and the feature map RL={X1,X2,X3,…X l}, where X i ∈R 121 ={x1,x2,x3,…x l}, i = 1, 2, 3…l, so that the feature map size after convolution is PL∈R 1×121 .

[0061] Step 6: Use the reshaping operation to ensure that the feature map PL and the feature map H∈R 32×11×11 Alignment, the size of the aligned feature map is PL∈R 1×11×11 , so that the element-wise multiplication operation of the feature map PL and the feature map H can be realized. Then the weight parameters with absolute position information and elevation information are learned through the softmax function, and then the fused and enhanced spatial spectral feature map HE is obtained through the element-wise multiplication operation, and its size is HE∈R 32×11×11 , the whole process is as follows:

[0062]

[0063] Where “⊙” represents multiplication of corresponding elements, and σ(·) represents the softmax function. The final feature map HE is enhanced by the absolute position information of the feature map P and fused with the elevation information in the feature map L. The feature map HE is then used for classification and recognition in the classification module.

[0064] Step 7: In order to solve the problem that some details or important information are inevitably lost in the feature extraction process of hyperspectral images and lidar images, thus affecting the classification performance to a certain extent, a supplementary information feature map extraction module is designed to use the maximum pooling and average pooling operations on the feature map output by the convolution layer to learn the prominent features and overall feature information of hyperspectral images and lidar images at different layers from shallow to deep.

[0065] Specifically, the feature map extracted from the feature extraction module and Input Figure 5 The supplementary information is extracted from the supplementary information feature map extraction module shown in FIG. First, the feature maps after different layers of convolution are subjected to maximum pooling and average pooling to extract salient features and overall structural information, and then the two pooled feature maps are spliced ​​to obtain and As follows:

[0066] l n =Concat(Maxpool(T n ),Avgpool(T n )), n=1,2,3

[0067] Among them, Concat means feature concatenation, Maxpool and Avgpool mean maximum pooling and average pooling respectively. Then, the feature map of the three types of pooling results is concatenated. or After splicing, it is processed by two-dimensional convolution, batch normalization and GELU activation function. The process is as follows:

[0068] F = Conv2D(Concat(l1,l2,l3))

[0069] Among them, Conv2D represents the layer structure composed of two-dimensional convolution, batch normalization and GELU activation function. Finally, it is reshaped into one-dimensional data to obtain the feature map F H and F L , and then align it with the feature map HE through the fully connected layer and reshape operation to obtain a size of R 32×11×11 The feature map F H and F L Finally, the information is compressed into a one-dimensional sequence through the average pooling operation, and the final supplementary information feature map matrix S H , S L ∈R 32×1 .

[0070] Step 8: The spatial spectrum feature map HE and the supplementary information matrix S obtained by the above enhancement fusion are combined H and S L Classification and recognition are performed through the classification module. First, the feature map HE is averaged and pooled to compress it into one-dimensional data to obtain HE∈R 32×1 , and then the feature maps HE and S H and S L Perform feature concatenation to obtain a new feature map Y∈R 96×1 , perform full connection layer operation and softmax operation on the obtained feature map Y to obtain the classification result. The classification result and label are used in the subsequent APENet model training process through the cross entropy loss function. The entire network model architecture is as follows Figure 3 shown.

[0071] During the training process, the present invention configures the hyperparameters required for network model training and testing for the constructed APENet model based on location information enhancement, and uses the data set and modules constructed in the above steps to train the APENet model for hyperspectral and lidar image fusion classification.

[0072] The trained APENet model is used for hyperspectral and lidar image classification. The two public hyperspectral and lidar datasets, Houston and Trento, are classified using the APENet model with and without the absolute position information enhancement module APEM. The overall classification accuracy (OA), average classification accuracy (AA) and Kappa coefficient (K) of different datasets in different networks are shown in Table 1. Not using APEM means that the HSI feature map is enhanced only by the features of the LiDAR image data.

[0073] Table 1: Comparative experimental accuracy of APEM on different datasets

[0074]

[0075] From the data in the table, we can see that compared with not using APEM, the method of using APEM to fuse the absolute position coding map with the LiDAR features and then enhance the HSI features performs better on both datasets. In the Houston dataset experiment, the OA index is improved by about 0.9%, and the AA and K coefficients are also improved by about 1%; on the Trento dataset, the three indicators are also improved by about 0.3%.

[0076] In summary, the hyperspectral and lidar image fusion classification method based on position information enhancement in the present invention can fully mine and utilize the useful absolute position information in hyperspectral images and lidar images, thereby effectively improving the overall classification accuracy of the APENet model for different categories, and then effectively used in the field of land cover classification.

[0077] The above is a specific embodiment of the present invention, and the technical scheme and beneficial effects of the present invention are further described in detail. It should be understood that the protection scope of the present invention is not limited to the above embodiment. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention and without departing from the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hyperspectral and lidar image fusion classification method based on location information enhancement, characterized in that: The following steps are included: Step 1: Construct an absolute position coding map POS. First, set a matrix map of the same size as the hyperspectral image and a window pos_windows of a preset size. In the matrix map, all pixels are divided into different sub-regions in sequence using the pos_windows window, and the pixels in these sub-regions are marked in sequence as absolute position codes, and the pixels in the same sub-region are marked with the same code. At the same time, in order to eliminate the influence of excessive order of magnitude and simplify the calculation, the encoded matrix map is normalized to obtain the absolute position coding map corresponding to all pixels in the hyperspectral image or lidar image. Step 2: The absolute position coding map constructed in step 1, the hyperspectral image and the lidar image are subjected to mirror filling and sliding slicing to obtain an absolute position information cube, a hyperspectral spatial spectrum cube and a lidar elevation information cube; Step 3, construct an Absolute Position Enhancement Network (APENet) model, which includes a feature extraction module, an Absolute Position Enhancement Module (APEM), a supplementary information feature map extraction module and a classification module; the feature extraction module consists of four convolutional layers, and each convolutional layer structure consists of two-dimensional convolution or three-dimensional convolution, batch normalization and GELU; APEM slices and fuses various features through nonlinear activation function, matrix product and softmax function to obtain a spatial spectral feature map HE; the supplementary information feature map extraction module retains the prominent features and overall structural information of the convolutional layer output feature map through pooling operation, and uses it as supplementary information for classification; The classification module integrates the obtained HE and supplementary information and processes them through the fully connected layer to obtain the final classification result; The APENet model first extracts abstract representations from the three types of feature cubes obtained by slicing through a feature extraction module; it uses a four-layer convolution layer consisting of two-dimensional convolution, batch normalization, and GELU to extract features from the elevation information cube and the absolute position information cube to obtain the lidar feature map L and the absolute position feature map P; and uses one three-dimensional convolution layer and three two-dimensional convolution layers to fully extract the hyperspectral feature map H from the spatial spectrum cube; Step 4: Input the three types of features extracted in step 3 into the Absolute Position Enhancement Module (APEM) of the APENet model to fuse and enhance the information of the three types of features. The extracted feature map H contains rich spatial spectral features, the feature map L has rich elevation information, and the feature map P contains the absolute position information between pixels. The APEM module is used to fully explore the features of the absolute position correlation between each pixel in the hyperspectral image and the lidar image. By capturing the absolute position information and elevation features in the feature maps P and L, and using them to enhance the feature map H, H is enhanced with absolute position information and fused with the elevation information in the lidar image to obtain a more compact and effective multimodal fusion feature. First, the feature maps P and L are processed by the Tanh function, and then the sizes of the feature maps P and L are adjusted by the reshaping operation, because the sizes of the feature maps P and L obtained after the above convolutional layers satisfy P, L∈R c×h×w , where c, h, and w represent the number of channels, height, and width of the feature map respectively; two feature maps of the same size cannot be used for subsequent matrix product operations, so their sizes need to be adjusted. The size of the feature map after reshaping is L∈R c×l , P∈R l×c , where l = h × w; Step 5: Perform matrix product operation on the adjusted feature maps P and L. The size of the feature map PL after matrix product is R l ×l , l represents the product of the width and height of the original input image; then the number of dimensions is reduced to reduce the amount of calculation through a one-dimensional convolution operation with an output channel of 1 and a convolution kernel size of 1; Step 6: Use the reshaping operation to ensure that the feature map PL and the feature map H∈R c×h×w The size of the aligned feature map is PL∈R 1×h×w , and then learn the weight parameters with absolute position information and elevation information from the feature map PL through the softmax function, and then obtain the fused and enhanced spatial spectral feature map HE through element multiplication operation; Step 7: Use the maximum pooling and average pooling operations on the output feature map of the convolution layer of the feature extraction module in step 3 to retain the prominent features and overall feature information of the hyperspectral image and lidar image at different layers from shallow to deep, and obtain the hyperspectral S H With LiDAR S L The two feature maps are used as supplementary features for image classification, and are processed together with the feature map obtained in step 6 in the classification module of the APENet model to obtain the classification probability; Step eight, for the constructed APENet model based on location information enhancement, configure the parameters required for APENet model training and testing, train the APENet model, and perform hyperspectral and lidar image fusion classification based on the trained APENet model, that is, hyperspectral and lidar image fusion classification based on location information enhancement.

2. The hyperspectral and laser radar image fusion classification method based on location information enhancement according to claim 1 is characterized in that: The encoding graph is normalized according to the following formula to reduce the amount of calculation: POS i,j represents the pixel value of the i-th row and j-th column in the coding image POS, where i = 1, 2, 3 ... H; j = 1, 2, 3 ... W; H, W represent the height and width of the hyperspectral image respectively; N(POS i,j ) means that the pixel values ​​in the i-th row and j-th column of the matrix image are normalized; while min(POS) and max(POS) respectively represent the minimum and maximum values ​​in the POS of the coding image.

3. The hyperspectral and laser radar image fusion classification method based on location information enhancement according to claim 1 is characterized in that: In the absolute position enhancement module APEM, the feature map P and feature map L are firstly subjected to the Tanh function to introduce nonlinearity into the network model. The convolution layer used in the feature extraction process is a linear combination operation. The introduction of the nonlinear Tanh function improves the nonlinear fitting ability of the module, thereby enhancing the expression ability of the APENet model.

4. The hyperspectral and laser radar image fusion classification method based on location information enhancement according to claim 1 is characterized in that: In step 5, the convolution formula is as follows: In the above formula, Ψ(·) represents a one-dimensional convolution with depth, w∈R l×1×1 represents the weight parameter of Ψ(·), the depth of the one-dimensional convolution is l, the convolution kernel size is 1×1, and the feature map PL = {X1,X2,X3,…X l }, where X i ∈R l ={x1,x2,x3,…x l }.

5. The hyperspectral and laser radar image fusion classification method based on location information enhancement according to claim 1 is characterized in that: The spatial spectrum characteristic graph HE is as follows: in represents matrix product, "⊙" represents multiplication by corresponding elements, T(·) represents the activation operation of Tanh nonlinear function, and σ(·) represents the softmax function. The obtained spatial spectral feature map HE is enhanced by the absolute position information of the feature map P, and integrates the multimodal fusion features of the elevation information in the lidar image feature map L and the spatial spectral information in the hyperspectral image feature map H.

6. The method for fusion classification of hyperspectral and laser radar images based on location information enhancement according to claim 1 is characterized in that: In step seven, the salient features and overall feature information of the feature map output by each convolutional layer are retained through the supplementary information feature map extraction module, which is used as a supplementary feature and concatenated with the feature map HE obtained in step six, and then the classification probability is output through the classification module. The obtained classification probability and label value are used through the cross entropy loss function to train the APENet model based on position information enhancement in step eight.

Images