A hyperspectral image classification method based on full-spectral correlation learning network

Abstract

The present invention provides a hyperspectral image classification method based on a full-spectral correlation learning network. Aiming at the correlation between adjacent spectral bands and non-adjacent spectral bands, a full-spectral correlation adaptive learning module is designed, and it is used as the basic structural unit to construct a backbone feature extraction network to extract deep semantic spatial-spectral features that enhance the joint spectral correlation and retain the intrinsic geometric structure; extract traditional manual features with detail information and interpretability information, and design an asymmetric fusion module with the help of the gated structure idea to align and fuse the above two multimodal spatial-spectral features at low complexity; combine the above two modules to propose a full-spectral correlation learning network to achieve effective classification of hyperspectral images. The classification algorithm proposed by the present invention can make full use of the spectral attribute characteristics of hyperspectral images and effectively realize the classification of ground objects.

Description

Technical Field

[0001] The invention belongs to the field of airborne remote sensing intelligent processing, and in particular relates to a hyperspectral image classification method based on a full-spectrum correlation learning network. Background Art

[0002] With the rapid development of airborne remote sensing technology, sensor technology and imaging technology, remote sensing images have gradually evolved from panchromatic remote sensing images, multispectral remote sensing images to hyperspectral remote sensing images, with increasing spatial resolution and spectral resolution, and are widely used in airborne remote sensing fields such as earth resource exploration, climate disaster monitoring, military reconnaissance, wetland monitoring, food safety, and biomedicine. Among them, hyperspectral image remote sensing intelligent analysis and processing technology, especially feature extraction and classification technology, is the most critical information acquisition technology.

[0003] Existing hyperspectral remote sensing image classification algorithms can be divided into two types according to the different implementation processes: a two-stage classification method of "feature extraction + classifier" and an "end-to-end" integrated classification method. With the rapid development of computer vision and artificial intelligence technology, the integrated classification method based on deep learning has achieved better classification performance than the traditional two-stage classification method with its strong feature extraction ability, parameter autonomous learning ability and model generalization ability. However, due to factors such as sensor accuracy and climate environment, problems such as "same object, different spectrum, same spectrum, different objects" in hyperspectral remote sensing images, limited label samples, and noise / abnormal data interference have limited the classification algorithm, especially the classification performance of the integrated classification method based on deep learning.

[0004] The existing hyperspectral image classification algorithms based on deep learning have the following shortcomings: (1) The existing algorithms use attention mechanisms, recurrent neural networks, etc. to learn the spectral correlation of hyperspectral images, but do not fully consider the impact of spectral attributes between adjacent spectral bands and non-adjacent spectral bands on the classification performance of ground objects from a global and local perspective. They are often only used as an additional submodule to assist the feature extraction process of the entire algorithm, and are rarely designed as basic structural units to build a new backbone network; (2) The combination of the above two classification algorithms can make up for the problem of insufficient feature detail information in deep learning algorithms, but most of the existing multimodal feature fusion classification algorithms have relatively complex structures, which increases the difficulty of optimizing the entire classification algorithm. Therefore, how to design an intelligent classification algorithm that can fully utilize the inherent attribute information of hyperspectral images is a key problem that technicians in the field of airborne remote sensing intelligent processing need to solve. Summary of the invention

[0005] The purpose of the present invention is to make full use of the inherent attribute information of hyperspectral remote sensing images and maximize the classification accuracy of ground objects, and provide an airborne hyperspectral remote sensing image classification method based on a full-spectral correlation learning network, which is applied to the field of airborne remote sensing intelligent processing, improves the hyperspectral remote sensing image classification performance, and solves the key problems mentioned in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a hyperspectral image classification method based on a full spectrum correlation learning network, comprising the following steps:

[0007] S1. Based on the characteristics of hyperspectral remote sensing images, such as rich spatial spectrum (space spectrum), strong spectral correlation and multimodal features, a three-dimensional data cube of each ground object and its spectral grouping are constructed;

[0008] S2, using spectral grouping of three-dimensional data as input data, combined with group convolution and spatial spectrum convolution long short-term memory network, designed a full spectrum correlation adaptive learning module to dynamically learn the spectral correlation of hyperspectral images;

[0009] S3: Taking the full-spectral correlation adaptive learning module as the basic structural unit, a backbone feature extraction network is built to extract deep semantic features enhanced by spectral information and retaining the intrinsic geometric structure;

[0010] S4, using 3D data cube as input data, constructing an asymmetric fusion module based on gated structure to align and integrate deep semantic features and traditional manual features, while suppressing noise and abnormal data interference;

[0011] S5. Integrate the above two modules to build a full-spectral correlation learning network to achieve hyperspectral image classification.

[0012] In the step S2, based on the three-dimensional data cube sequence obtained by grouping along the spectral dimension of the hyperspectral remote sensing image in step S1, the respective advantages of grouped convolution and spatial spectral convolution long short-term memory network in feature extraction are drawn upon to design a full-spectral correlation adaptive learning module, including: a short-term spectral correlation learning submodule, a long-term spectral correlation learning submodule, and a grouped spatial spectral correlation learning submodule. At the same time, a skip connection structure is used to avoid the gradient vanishing or gradient exploding problem, and the spectral information and correlation of the hyperspectral image are adaptively learned by grouping the obtained three-dimensional data spectrum.

[0013] In step S3, the full spectrum correlation adaptive learning module designed in step S2 is used as the basic structural unit to build a backbone feature extraction network, and dynamically extract deep semantic spatial spectral features with enhanced spectral correlation and retained intrinsic geometric structure information from the three-dimensional data spectral grouping obtained by executing step S1.

[0014] In step S5, based on the backbone feature extraction network designed in step S3 and the asymmetric fusion module designed in step S4, a full-spectral correlation learning network is constructed, and spatial-spectral features that take into account both semantic information and detail information are extracted from the three-dimensional data cube and its spectral grouping obtained by executing step S1, so as to realize hyperspectral image classification and verify the feasibility of the proposed classification method.

[0015] The full-spectrum correlation learning network proposed in the present invention has good spatial-spectral feature extraction capability. The designed full-spectrum correlation adaptive learning module can be used as a general module to build a backbone feature extraction network for feature extraction of space-time or space-spectral data, thereby improving the intelligent processing performance in the field of airborne remote sensing.

[0016] The present invention provides a hyperspectral image classification method based on a full-spectral correlation learning network. With the goal of reducing complexity and improving classification accuracy, the present invention comprehensively considers the data characteristics of hyperspectral images in the field of airborne remote sensing and constructs a new classification model based on a full-spectral correlation learning network. Compared with the existing classification model, the classification algorithm proposed in the present invention can make full use of the spectral attribute characteristics of hyperspectral images and effectively realize the classification of ground objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly explain the purpose, design ideas and innovation of the airborne hyperspectral remote sensing image classification method proposed in the present invention, the present invention will be described in detail with reference to the accompanying drawings and attached tables.

[0018] Figure 1 This is a flow chart of the airborne hyperspectral remote sensing image classification method proposed in the present invention.

[0019] Figure 2 This is the structural diagram of the full-spectrum correlation learning network proposed in the present invention.

[0020] Figure 3 This is a structural diagram of the full-spectrum correlation adaptive learning module proposed in the present invention.

[0021] Figure 4 This is a comparison chart of the visualized experimental results of the method proposed in this invention and five benchmark methods in the past five years. DETAILED DESCRIPTION

[0022] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1

[0024] This embodiment is a hyperspectral image classification method based on a full spectrum correlation learning network provided by the present invention. The method flow is as follows Figure 1 As shown, the following steps are included:

[0025] S1. Based on the characteristics of hyperspectral remote sensing images, such as rich spatial spectral (space spectrum) information, strong spectral correlation and multimodal features, a three-dimensional data cube and its spectral grouping are constructed for each ground object.

[0026] Assume that the dimension of the original hyperspectral image is W×H×D, where W, H, and D are the width, height, and number of spectral bands, respectively.

[0027] With the continuous development of remote sensing technology, imaging technology and sensor technology, the special imaging mechanism makes hyperspectral images contain richer spectral information and spatial information. First, for the i-th pixel in the image, select the s×s local neighborhood area centered on it as its spatial information, and combine it with the spectral information to obtain a three-dimensional data cube As the spatial spectrum information of the pixel, a 3×3×3 three-dimensional convolutional layer is used to extract its primary spatial spectrum features. C 1 Indicates the number of feature channels.

[0028] Then, in order to facilitate the subsequent joint learning of local and global spectral correlations and simplify the parameter analysis process, based on the feature extraction characteristics of hyperspectral images, the spectral dimension is used to extract the local and global spectral correlations. Perform grouping to obtain three-dimensional data spectral grouping: Where m = 1, 2, ... N s , d represents the number of spectral bands in each spectral grouping, represents the number of spectral groups, Indicates rounding down.

[0029] Based on the above preprocessing process, for each pixel point, the input spatial spectrum information is a three-dimensional data cube X i and its spectral grouping Therefore, the training set is constructed:

[0030]

[0031] And the test set:

[0032]

[0033] The corresponding training set labels are:

[0034]

[0035] And the test set labels:

[0036]

[0037] in, and represent the number of training samples and the number of test samples respectively.

[0038] S2, using 3D data spectral grouping as input data, combined with group convolution and space spectrum convolution long short-term memory network, design full spectrum correlation adaptive learning module, such as Figure 3 As shown, the spectral correlation of hyperspectral images is dynamically learned.

[0039] There are rich spectral correlations between adjacent spectral bands and non-adjacent spectral bands in hyperspectral images. Making full use of the attribute characteristics of spectral information can improve the classification accuracy of ground objects. To this end, based on the respective advantages of grouped convolution and spatial spectral convolution long short-term memory networks, the present invention designs a full spectral correlation adaptive learning module, such as Figure 2 As shown in FIG, it includes: a short-term spectral correlation learning submodule, a long-term spectral correlation learning submodule and a grouped spatial spectral correlation learning submodule. Assume that the input data of this module is C l Input channel number for the l-th layer full spectrum correlation adaptive learning module.

[0040] First, X l As the input of the short-term spectral correlation learning submodule, the short-term spectral correlation is learned through 1 layer of 1×1×1 and 1 layer of 3×3×3 grouped convolutional layers to obtain the spatial spectrum feature

[0041] Then, since the spatial spectral convolutional long short-term memory network has the advantage of capturing the long-term correlation of data, a long-term spectral correlation learning submodule is constructed, using a 1×1×1 spatial spectral convolutional long short-term memory network to learn the long-term spectral correlation from the input data. Extracting spatial spectral features enhanced by long-term spectral correlations

[0042] Then, the spatial-spectral feature F enhanced by joint spectral information is obtained by further grouping the spatial-spectral correlation learning submodule. l , the specific calculation process is designed as follows:

[0043] T=f CBR1 (f concat (X l )) (1)

[0044]

[0045]

[0046]

[0047] In formula (1), f concat (·) is a cascade operation, f CBR1 (·) represents a 1×1×1 3D convolutional layer (including batch normalization layer and ReLU function).

[0048] In formula (2) represents the element-wise product operation, f spe (T) represents the grouped self-spectral attention learning module.

[0049] In formula (3), f int (T) represents the self-intrinsic structure attention learning module.

[0050] The calculation process of the above two attention learning modules is designed as follows:

[0051] f spe (T) = G σ (f C1 (softmax(f CBR1 (T))⊙f CBR1 (T))) (5)

[0052] f int (T) = σ(f C1 (softmax(f GAP (f CBR1 (T)))⊙f CBR1 (T))) (6)

[0053] Among them, in formula (5), σ(·), G σ (·) and softmax(·) are Sigmoid, grouped Sigmoid and Softmax functions respectively, ⊙ is the matrix multiplication operation, f C1 (·) is a 1×1×1 3D convolutional layer.

[0054] In formula (6), f GAP (·) indicates a global average pooling layer.

[0055] Finally, in order to avoid the gradient vanishing or gradient exploding problem in the entire deep classification network due to the deepening of the network, the idea of ​​skip connection is introduced to reuse the input data X l , and obtain the output spatial spectrum feature X of the entire full spectrum correlation adaptive learning module l+1 :

[0056] X l+1 =f CBR1 (F l )+f concat (X l ). (7)

[0057] S3. Using the full-spectral correlation adaptive learning module as the basic structural unit, a backbone feature extraction network is built to extract deep semantic features that are enhanced by spectral information and retain the intrinsic geometric structure.

[0058] For each pixel point in step S1, the three-dimensional data spectrum is grouped Taking the full spectrum correlation adaptive learning module in step S2 as the basic structural unit, a new backbone feature extraction network is designed by stacking l layers of the basic structural unit to extract deep semantic spatial-spectral features that can simultaneously characterize long-term and short-term spectral information from hyperspectral images.

[0059] S4. Using a three-dimensional data cube as input data, we construct an asymmetric fusion module based on a gated structure to align and integrate deep semantic features and traditional manual features, while suppressing noise and abnormal data interference.

[0060] Different from deep semantic features, traditional manual features have the advantages of rich detail information and strong interpretability. With the advantage of three-dimensional Gabor filter that can depict the local signal changes of hyperspectral images in space, spectrum and spatial spectrum domain, the gated structure idea is introduced to construct an asymmetric fusion module, including detail feature learning submodule and channel feature learning submodule, to extract spatial spectrum features that take into account both semantic information and detail information. In addition, the gating mechanism can assist the entire network to suppress interference such as noise and abnormal data to a certain extent.

[0061] First, the three-dimensional Gabor filter contains three important parameters: frequency f and direction Among them, according to the analysis of existing literature, θ and The value range of is fixed as follows: n = 4, and the value of f is selected and set according to the specific hyperspectral image dataset. After removing some redundant filters, each frequency f will correspond to 13 three-dimensional Gabor filters. In order to simplify the experimental analysis and network structure design, the number of three-dimensional Gabor filters is defined as N g =13, d is fixed to 13 in step S1.

[0062] Then, for each pixel in step S1, the 3D data cube The three-dimensional Gabor filter designed above is used to perform convolution operation with it to extract the traditional manual spatial spectrum features of the hyperspectral image.

[0063] Finally, the asymmetric fusion module is used to learn the deep semantic features F(deep) through the channel feature learning submodule and the detail feature learning submodule. i And the traditional manual feature F(hand) i The attribute information of each modality is combined to realize the dimension alignment and information fusion of the two modal spatial spectrum features, and the spatial spectrum feature F(fuse) that takes into account both semantic information and detail information is obtained. i , which is used for the subsequent classification of ground objects. The specific calculation process is designed as follows:

[0064]

[0065] Among them, f cha (·) represents the channel feature learning submodule, f det (·) represents the detail feature learning submodule, and its calculation process is designed as follows:

[0066] f cha (·)=softmax(f C1 (ReLU(f C1 (f GAP (·)))) (9)

[0067] f det (·)=softmax(f C1 (·)) (10)

[0068] S5. Integrate the above two modules to build a full-spectral correlation learning network to achieve hyperspectral image classification.

[0069] First, based on steps S3 and S4, considering the problems of large number of parameters and high training difficulty of the traditional "fully connected layer + Softmax function" classification network, we further use the global pooling layer and orthogonal Softmax function to design a low-complexity classification network and construct a new full-spectral correlation learning network.

[0070] Therefore, the loss function of the proposed full-spectral correlation learning network is The definition is as follows:

[0071] Y P =f OSL (f GAP (F)) (11)

[0072]

[0073] In formula (11), F is the fusion feature extracted by formula (8) in step S4, and f OSL (·) is the orthogonal Softmax function.

[0074] In formula (12), Y P and Y are the predicted labels output by the model and the true category labels of the input data, respectively.

[0075] Then, based on the training set {X train , Y train}, the loss function in formula (12) is optimized by adaptive momentum optimization algorithm End-to-end training and optimization are performed to enable the entire network to fully learn the inherent attribute information of hyperspectral remote sensing images, while achieving effective classification of ground objects, thereby obtaining a trained full-spectral correlation learning network model.

[0076] Finally, the test set X test Input into the above trained full spectrum correlation learning network model to predict its category label And the category prediction label With the test set label Y test Comparison is made to evaluate the classification performance of the proposed network model.

[0077] Example 2

[0078] Based on Example 1, this implementation selects a public airborne hyperspectral remote sensing image dataset (Indian Pines dataset) to conduct simulation experiments on the airborne hyperspectral remote sensing image classification algorithm proposed in the present invention to verify its feasibility and effectiveness. The dataset is a hyperspectral remote sensing image of a certain area taken by an airborne visible light / infrared imaging spectrometer, with 224 spectral bands and a spatial resolution of 145×145 pixels. After removing some invalid bands and background pixels, the entire dataset contains 200 spectral bands, 10,249 pixels and 16 types of ground objects for experimental research and comparative analysis.

[0079] In addition, this implementation method selects five novel hyperspectral remote sensing image classification algorithms in the past five years as comparison methods, such as Figure 4. Including: Composite Kernel Support Vector Machine (SVM-CK.Remote Sens.2018), Spatial-Spectral 3-D Convolutional Long Short-Term Memory Neural Network (SSCL3DNN.IEEE Trans.Geosci.Remote Sens.2020), 3-D Octave Convolution with theSpatial-Spectral Attention Network (3DOC-SSAN.IEEE Trans.Geosci.RemoteSens.2021), 3-D Gabor Convolutional Neural Network (3DG-CNN.IEEETrans.Geosci.Remote Sens.2022), Pseudo Complex-Valued Deformable ConvLSTMNeural Network with Mutual Attention Learning (APDCLNN.IEEETrans.Geosci.Remote Sens.2022).

[0080] Table 1 shows the quantitative experimental results of the classification algorithm proposed in the present invention and the above five benchmark methods on the data set, including the classification accuracy of each category, the overall classification accuracy (OA), the average classification accuracy (AA) and the Kappa coefficient, and the results shown are the average values ​​of 10 random experiments. Among them, 10 labeled samples are randomly selected from each type of ground object target for algorithm training, and the remaining samples are used for testing and verification. That is, the training set has 160 labeled samples and the test set has 11116 labeled samples.

[0081] According to Table 1, compared with other comparison methods, the algorithm of the present invention achieves at least 3.54%, 3.40% and 3.92% improvement in OA, AA and Kappa coefficient indicators respectively under the premise of moderate model complexity, and the classification performance of most ground objects is improved, especially the classification performance of ground objects with strong spectral correlation (for example: Class 2 and Class 3, Class 5 and Class 6).

[0082] Table 1 Classification results of different classification algorithms under Indian Pines dataset (%)

[0083]

[0084]

[0085] Aiming at the field of airborne remote sensing intelligent processing, the present invention proposes a hyperspectral remote sensing image classification method based on a full-spectral correlation learning network. The effectiveness of the algorithm of the present invention in reducing the complexity of deep models and improving classification performance is proved through network structure derivation, experimental results and comparative analysis.

Claims

1. A hyperspectral image classification method based on full-spectral correlation learning network, It is characterized in that The following steps are included: S1. Based on the spatial spectrum of the hyperspectral remote sensing image, construct a three-dimensional data cube for each ground object and its spectral grouping; S2, using spectral grouping of three-dimensional data as input data, combined with group convolution and spatial spectrum convolution long short-term memory network, designed a full spectrum correlation adaptive learning module to dynamically learn the spectral correlation of hyperspectral images; Full spectrum correlation adaptive learning module, including: short-term spectrum correlation learning submodule, long-term spectrum correlation learning submodule, grouped space spectrum correlation learning submodule; Input data X l As the input of the short-term spectral correlation learning submodule, the short-term spectral correlation is learned through 1 layer of 1×1×1 and 1 layer of 3×3×3 grouped convolutional layers to obtain the spatial spectrum feature Construct a long-term spectral correlation learning submodule, using a 1×1×1 spatial spectral convolutional long short-term memory network to learn from the input data Extraction of spatial spectral features with enhanced long-term spectral correlations; By grouping the spatial-spectral correlation learning submodule, the spatial-spectral features with enhanced spectral information are obtained; S3: Taking the full-spectral correlation adaptive learning module as the basic structural unit, a backbone feature extraction network is built to extract deep semantic features enhanced by spectral information and retaining the intrinsic geometric structure; S4, with three-dimensional data cube X i As input data, an asymmetric fusion module based on a gated structure is constructed, and the asymmetric fusion module includes a detail feature learning submodule and a channel feature learning submodule; S5. Integrate the backbone feature extraction network designed based on step S3 and the asymmetric fusion module designed based on step S4 to construct a full-spectral correlation learning network to achieve hyperspectral image classification.

2. A hyperspectral image classification method based on a full spectrum correlation learning network as claimed in claim 1, It is characterized in that In the step S2, based on the three-dimensional data cube sequence obtained by grouping along the spectral dimension of the hyperspectral remote sensing image in step S1, a full-spectral correlation adaptive learning module is designed in combination with group convolution and spatial-spectral convolution long short-term memory network; at the same time, a skip connection structure is used to avoid the gradient vanishing or gradient exploding problem, and the spectral information and correlation of the hyperspectral image are adaptively learned by grouping the obtained three-dimensional data spectrum.

Images