A hyperspectral remote sensing image classification method based on attention joint network

Abstract

The present application relates to a kind of hyperspectral remote sensing image classification method based on attention joint network, compared with prior art, it solves the defect of insufficient classification performance caused by limited training samples.The present application includes the following steps: training sample acquisition and preprocessing;Attention joint network is built;Attention joint network training;Classification sample acquisition and preprocessing;The acquisition of hyperspectral remote sensing image classification result.The present application extracts spatial spectral feature and spatial feature from the hyperspectral image after PCA dimension reduction, introduces the attention module consisting of channel attention and spatial attention to refine features, by connecting the output of two branches, enhanced spatial spectral feature is obtained, finally classification is carried out using Softmax classifier, still has good classification performance under the condition of limited training samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hyperspectral remote sensing image, in particular to a hyperspectral remote sensing image classification method based on attention joint network. BACKGROUND

[0002] Hyperspectral remote sensing image has rich spatial and spectral information, which can show the light emission characteristics of objects in the medium and long infrared range, and has been widely used in target recognition, environmental monitoring and other fields. However, in practical application, the hyperspectral image has the problems of strong redundancy, multiple bands and limited training samples, which makes the classification of hyperspectral image still has great challenges.

[0003] In order to solve the above problems, dimension reduction is used to convert high-dimensional spectral data into low-dimensional spectral data while retaining potential spectral information. The classical dimension reduction methods include principal component analysis (PCA), independent component analysis (ICA) and so on. Ahmad et al. compared the influence of different dimension reduction methods on the classification of hyperspectral image, and proved that PCA has better performance than other dimension reduction methods in the classification of hyperspectral image.

[0004] With the continuous progress of machine learning field, deep learning is gradually applied to the classification of hyperspectral image. Two-dimensional convolutional neural network (2D-CNN) can automatically extract features from hyperspectral image to realize classification, but at the same time, part of the spatial or spectral information will be lost.

[0005] To solve this problem, three-dimensional convolutional neural network (3D-CNN) is introduced into hyperspectral image classification. 3D-CNN has relatively large computational complexity, but can better extract spatial-spectral features in hyperspectral images. Ying et al. used 3D-CNN with relatively small kernel and window size to classify hyperspectral images, reducing the computation time. However, these methods only extract shallow features of hyperspectral images, and the classification effect is not ideal in the case of less samples. Subsequently, Wang et al. used deep residual two-dimensional convolutional neural network (Res-2D-CNN) to realize hyperspectral image classification in the case of small samples, and achieved good classification results. Bing et al. introduced residual connection into 3D-CNN and constructed residual three-dimensional convolutional neural network (Res-3D-CNN), which further improved the classification performance. Roy et al. proposed a 3D and 2D hybrid neural network (HybridSN), which performs joint spatial-spectral feature recognition with 3D convolution, and 2D convolution to provide more spatial data, so that the classification not only has accuracy, but also reduces the computational complexity. Feng et al. proposed a residual hybrid network (R-HybridSN) by reasonably using non-identity residual connection, which also obtained satisfactory classification results in the case of less training samples. However, R-HybridSN does not make good use of shallow features, so the network structure can be further optimized.

[0006] In recent years, attention mechanism has been widely used in network structure, which makes limited resources can be reasonably allocated for processing more important information. Hu et al. proposed squeeze-and-excitation network and introduced attention mechanism into image classification network, winning the championship of 2017 ImageNet Large Visual Recognition Challenge. Li et al. added an attention module after the dense connection module for shallow and intermediate layer feature extraction to refine the effective features of spectral bands and further extract deep features. However, since the spatial attention and channel attention of the attention mechanism are usually separated from each other. Li et al. proposed an attention network (DANet) that can effectively classify hyperspectral images.

[0007] Although existing methods can effectively extract hyperspectral image features, the classification performance is insufficient in the case of limited training samples. Zhao et al. based on DensNet and attention mechanism, proposed a hybrid dense network with attention mechanism (HDDA) for hyperspectral image classification, which enhances the spectral-spatial feature separability, and has good classification effect on three data sets. However, the extraction of features through dense network will increase the number of parameters, and the residual attention module may not work. SUMMARY

[0008] The application aims to solve the problem of insufficient classification performance caused by limited training samples in the prior art, and provides a hyperspectral remote sensing image classification method based on an attention joint network to solve the above problem.

[0009] In order to achieve the above-mentioned purpose, the technical scheme of the application is as follows:

[0010] A hyperspectral remote sensing image classification method based on an attention joint network comprises the following steps:

[0011] Obtaining and preprocessing of training samples: obtaining a hyperspectral image to be trained and preprocessing the same;

[0012] Constructing an attention joint network: based on a three-dimensional and two-dimensional convolutional neural network model, an attention mechanism is introduced to establish an attention joint network with two feature extraction branches;

[0013] Training of the attention joint network: the preprocessed hyperspectral image is input into the attention joint network, a convolutional neural network with attention is used to extract features, then the features are fused and trained for classification;

[0014] Obtaining and preprocessing of classification samples: obtaining a hyperspectral image to be classified and preprocessing the same;

[0015] Obtaining of the classification result of the hyperspectral remote sensing image: the preprocessed hyperspectral image to be classified is input into the trained attention joint network to obtain the classification result of the hyperspectral remote sensing image.

[0016] The obtaining and preprocessing of the training samples comprise the following steps:

[0017] A sample X is obtained in each ground object category sample of the hyperspectral image in proportion as a training sample, and the sample X is represented as:

[0018] X=[x1,x2,x3,...,x B ] T ∈R (M×N)×B ,

[0019] wherein M, N and B respectively represent the width, height and spectral dimension of the hyperspectral remote sensing data, x i =[x 1,i ,x 2,i ,x 3,i ,...,x B,i ] T is the i-th sample of the hyperspectral data;

[0020] Dimension reduction of the sample X through principal component analysis: eigenvalues and corresponding eigenvectors of a covariance matrix E are calculated by using a characteristic decomposition method, and the calculation formula is as follows:

[0021] E = AUA T ,

[0022] wherein, E is a covariance matrix, A is an eigenvector matrix, A T is A transpose, U = diag [λ1, λ2, λ3,..., λ X ] is a diagonal matrix of eigenvalues of the covariance matrix,

[0023] The result of dimensionality reduction is represented as:

[0024] X P = AX,

[0025] wherein, X P is the hyperspectral data after dimensionality reduction, A is a transformation matrix, and X is the original hyperspectral data;

[0026] The hyperspectral image after dimensionality reduction is taken as a sample in a central pixel size w x w field and its corresponding class label to obtain training samples X P and its label Y P , the size of X P is w x w x d, and the size of Y is w x w, wherein w x w respectively represents width and height, and d represents spectral dimension;

[0027] The to-be-trained sample data X P is converted into a two-dimensional matrix X T , and the data size thereof is (w x w, d), each row thereof represents spectral information contained by a sample, and each column thereof represents different spectral dimensions; the label data Y P is converted into Y T , and the data size thereof is (w x w, 1).

[0028] The construction of the attention joint network includes the following steps:

[0029] A three-dimensional convolutional neural network is built:

[0030] The three-dimensional convolutional neural network includes four three-dimensional convolutional blocks, wherein the hyperspectral remote sensing data with X P as input is subjected to spatial spectral feature extraction by using convolution kernels with channel numbers n1, n2, n3 and n4, i.e., (a x a x b, n1), (a x a x c, n2), (a x a x a, n3) and (a x a x a, n4) to input data;

[0031] In the three-dimensional convolutional neural network, the input data is convolved with a three-dimensional kernel function, and then a nonlinearity is induced through an activation function, and a three-dimensional feature map is generated by convolving the extracted spectral band with the three-dimensional kernel function;

[0032] In the three-dimensional convolution process, the mathematical expression of the three-dimensional convolution is represented as:

[0033]

[0034] wherein, represents the result obtained by the i-th layer j-th feature map at the (x, y, z) position, is an activation function, τ represents the number of channels, η, γ and δ represent the dimensions of the length, width and channel direction of the three-dimensional convolution kernel respectively, represents the weight of the three-dimensional convolution kernel at the τ-th feature map (λ, σ, ρ), b i,j is a bias parameter;

[0035] A two-dimensional convolutional neural network is built:

[0036] The two-dimensional convolutional neural network is set to include four two-dimensional convolutional blocks connected, wherein the hyperspectral remote sensing data with X T as input is subjected to spatial feature extraction by using convolution kernels with channel numbers n1, n2, n3 and n4: (a×a, n1), (a×a, n2), (a×a, n3) and (a×a, n4);

[0037] In the two-dimensional convolutional neural network, the input data is subjected to convolution operation with a two-dimensional kernel function to obtain a two-dimensional feature map, and the convolution features are processed by an activation function; in the two-dimensional convolution process, the mathematical expression of the two-dimensional convolution is represented as:

[0038]

[0039] wherein, represents the result obtained by the i-th layer j-th feature map at the (x, y) position, is an activation function, τ represents the number of channels, η and γ represent the length and width of the three-dimensional convolution kernel respectively, represents the weight of the three-dimensional convolution kernel at the τ-th feature map (λ, σ), b i,j is a bias parameter;

[0040] In the four convolutional blocks of the three-dimensional convolutional neural network, an attention module is inserted between every two adjacent convolutional blocks;

[0041] In the four convolutional blocks of the two-dimensional convolutional neural network, an attention module is inserted between every two adjacent convolutional blocks;

[0042] The attention module is set to include a channel attention module and a spatial attention module;

[0043] The attention module is set as follows:

[0044] In the channel attention module, the feature map F∈R w×w×d×nAs input, where w x w is the size, d is the spectral dimension, and n is the number of channels;

[0045] First, generate features by max-pooling and average-pooling operations and

[0046] Second, set the shared network SN to include the upper convolution layer, the activation function, and the lower convolution layer from top to bottom, and input the two features and to the shared network SN;

[0047] Finally, merge the output features using the summation method, and get the channel attention map M C through the sigmoid activation function.

[0048] The channel attention calculation method is as follows:

[0049]

[0050] Where θ and θ' represent the Sigmoid and Relu activation functions respectively, Avgpool and Maxpool represent the global average pooling and global maximum pooling operations respectively, F is the original input feature map, and SN represents the shared network; W0 and W1 are the weights of the shared network SN, and the feature map F' ∈ R w×w×d×n is obtained by matrix multiplication, and the calculation method is as follows:

[0051]

[0052] Where M C is the channel attention feature map, is the matrix multiplication operation, and F is the input feature map.

[0053] Set the spatial attention module, and input the feature map after channel attention processing into the spatial attention module. The spatial attention module is used to extract spatial information in different regions.

[0054] First, generate features by global average pooling and global maximum pooling operations and and the two features have the same dimensions.

[0055] Then, generate a new feature

[0056] Finally, generate the spatial attention map M S through a convolution layer and a sigmoid function; the calculation method of spatial attention is as follows:

[0057]

[0058] wherein, θ is a Sigmoid activation function, f K×K×K represents a convolution operation with a kernel of KxKxK, Avgpool and Maxpool represent global average pooling and global maximum pooling operations respectively, F' is a feature map output by the channel attention module,

[0059] The feature map F'' is obtained by matrix multiplication, and F'' e R w×w×d×n The calculation method is as follows:

[0060]

[0061] wherein, M S is a spatial attention feature map, is a matrix multiplication operation, and F' is a feature map processed by the channel attention;

[0062] A three-dimensional convolutional neural network is used after a three-dimensional convolutional layer and a three-dimensional global average pooling layer to become a one-dimensional array, and a two-dimensional convolutional neural network is used after a two-dimensional convolutional layer and a two-dimensional global average pooling layer to become a one-dimensional array, then the one-dimensional array output by the three-dimensional convolutional neural network is connected with the one-dimensional array output by the two-dimensional convolutional neural network, and is connected into a Flatten layer and a fully connected layer;

[0063] A Relu activation function is used after each convolutional layer in the three-dimensional convolutional neural network and the two-dimensional convolutional neural network to guide nonlinearity and padding is used in the padding layer, and the mathematical expression of the ReLU activation function is as follows:

[0064]

[0065] ReLU is a piecewise linear function, which compares the input data x with the value 0, and outputs the maximum value, that is, all negative values are changed to 0, while the positive values remain unchanged.

[0066] The training of the attention joint network includes the following steps:

[0067] The preprocessed hyperspectral remote sensing image is centered on a neighborhood pixel of each ground object sample to form an attention joint network training data set, and each sample has a size of w x w x d three-dimensional cube;

[0068] The attention joint network training data set is input into the three-dimensional convolutional neural network module with attention, the three-dimensional convolutional layer is used to extract spatial spectral features, and the attention module is used to selectively learn image features to obtain a three-dimensional output feature map F 3D e R w×w×c×n′where w x w is the size of the feature map, c is the spectral dimension, and n' represents the number of channels; then the output features are subjected to 1 x 1 x d convolution and three-dimensional global average pooling to obtain a 1 x n4 spatial-spectral feature two-dimensional vector;

[0069] The three-dimensional hyperspectral training data set is converted into a two-dimensional matrix through Reshape remodeling and input into the two-dimensional convolutional neural network module with attention, the two-dimensional convolutional layer is used for extracting spatial features, and the attention module is used for selectively learning image features, to obtain a two-dimensional output feature map F 2D ∈R w×w×n′ where w x w is the size of the feature map, c is the spectral dimension, and n' represents the number of channels; then the output features are subjected to 1 x 1 x d convolution and three-dimensional global average pooling to obtain a 1 x n4 spatial-spectral feature two-dimensional vector;

[0070] The spatial-spectral features extracted in the three-dimensional and two-dimensional two paths are superimposed, are expanded through a Flatten layer, and are subjected to two Dropout layers to prevent overfitting, and finally a fully connected layer with a Softmax function is used to obtain a classification result.

[0071] In the network training process, the classification cross-entropy loss function is used to update the network parameters, and the expression is as follows:

[0072]

[0073] where ∑ is a summation operation, log is a logarithm operation, represents the error loss between the predicted value and the true value; Y i ={y1,y2,...,y L} represents a true label vector, represents a predicted label vector; L is the number of samples, and s is the number of classifications.

[0074] The Adam optimization algorithm is used to optimize the network, and Adam replaces the first-order optimization algorithm in the traditional stochastic gradient descent process, and iteratively updates the neural network weights based on the training data.

[0075] Advantages

[0076] Compared with the prior art, the spatial-spectral feature and the spatial feature are extracted from the PCA dimension-reduced hyperspectral image, the attention module composed of channel attention and spatial attention is introduced to refine the features, the enhanced spatial-spectral features are obtained by connecting the outputs of the two branches, and finally the Softmax classifier is used for classification, and the classification performance is still good under the condition of limited training samples. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 A method flowchart of the present application;

[0078] Figure 2 A false color image and a real object distribution map of the Indian Pines data set;

[0079] Figure 3 A false color image and a real object distribution map of the Pavia University data set;

[0080] Figure 4 A false color image and a real object distribution map of the Salinas data set;

[0081] Figure 5 The influence of different dimensions d of the method of the present application on classification accuracy;

[0082] Figure 6 Classification result maps of the method of the present application and comparative methods on the Indian Pines data set;

[0083] Figure 7 Classification result maps of the method of the present application and comparative methods on the Pavia University data set;

[0084] Figure 8 Classification result maps of the method of the present application and comparative methods on the Salinas data set;

[0085] Figure 9 The influence of the method of the present application on classification results under different sample proportions. DETAILED DESCRIPTION

[0086] In order to have a further understanding of the structural characteristics and the achieved effects of the present application, the following detailed description is given in conjunction with preferred embodiments and the accompanying drawings, as follows:

[0087] As shown in Figure 1 , the high-spectral remote sensing image classification method based on an attention joint network of the present application comprises the following steps:

[0088] Step 1: Obtain and preprocess the training samples: obtain the high-spectral image to be trained and preprocess it. The obtaining and preprocessing of the training samples comprise the following steps:

[0089] (1) Obtain a sample X in each class of object category sample of the high-spectral image as a training sample in proportion, and the sample X is represented as:

[0090] X=[x1,x2,x3,...,x B ] T ∈R (M×N)×B ,

[0091] wherein M, N, B represent the width, height, and spectral dimension of the hyperspectral remote sensing data respectively, x i = [x 1,i ,x 2,i ,x 3,i ,...,x B,i ] T is the i-th sample of the hyperspectral data.

[0092] (2) Dimensionality reduction of the sample X by principal component analysis: the eigenvalues and corresponding eigenvectors of the covariance matrix E are calculated using the eigenvalue decomposition method, and the calculation formula is as follows:

[0093] E = AUA T ,

[0094] wherein E is the covariance matrix, A is the eigenvector matrix, A T is the transpose of A, U = diag[λ1, λ2, λ3,..., λ X ] is the eigenvalue diagonal matrix of the covariance matrix,

[0095] Then the result of dimensionality reduction is represented as:

[0096] X P = AX,

[0097] wherein X P is the dimensionality-reduced hyperspectral data, A is the transformation matrix, and X is the original hyperspectral data.

[0098] (3) The dimensionality-reduced hyperspectral image is taken as a sample with a center pixel size w x w field and its corresponding class label to obtain the training sample X P and its label Y P , the size of X P is w x w x d, and the size of Y is w x w, wherein w x w represents the width and height respectively, and d represents the spectral dimension.

[0099] (4) The to-be-trained sample data X P is converted into a two-dimensional matrix X T , whose data size is (w x w, d), each row of which represents the spectral information contained by a sample, and each column represents a different spectral dimension; the label data Y P is converted into Y T , which is a one-dimensional vector with a data size of (w x w, 1).

[0100] Second step, constructing an attention joint network: based on three-dimensional and two-dimensional convolutional neural network models, an attention mechanism is introduced to establish an attention joint network with two feature extraction branches.

[0101] The traditional hyperspectral image feature extraction and classification method only extracts the spectral or spatial features of the hyperspectral image, so that the classification accuracy is not ideal. Therefore, the application adopts three-dimensional convolution and two-dimensional convolution, and introduces channel attention and spatial attention to construct an attention joint network with two feature extraction branches. The convolution layer can adaptively learn the semantic features in the hyperspectral image and obtain a large number of identifiable spatial spectral features. In addition, the features extracted by the deep network structure are more suitable for classification.

[0102] However, in the network training process, the weights assigned between the pixel features output by the convolution layer are the same, and the importance between the features cannot be effectively distinguished. The attention module composed of channel attention and spatial attention can differentially treat different feature maps, selectively learn more useful features, and weaken useless features, so that the classification effect is better. In addition, in view of the complex structure of hyperspectral data, the pooling layer and the Dropout layer are introduced to simplify the calculation and reduce the training time, and the Relu activation function is used to prevent the occurrence of training overfitting phenomenon, so that the classification model converges faster.

[0103] The specific operation steps are as follows:

[0104] (1) Build a three-dimensional convolutional neural network:

[0105] The three-dimensional convolutional neural network includes 4 three-dimensional convolutional blocks, wherein X P As input, the hyperspectral remote sensing data adopts convolution kernels with channel numbers n1, n2, n3 and n4: (a x a x b, n1), (a x a x c, n2), (a x a x a, n3) and (a x a x a, n4) to input data for spatial spectral feature extraction;

[0106] In the three-dimensional convolutional neural network, the input data is convolved with the three-dimensional kernel function, then the nonlinearity is induced through the activation function, and the three-dimensional feature map is generated by convolving the extracted spectral band with the three-dimensional kernel function;

[0107] In the three-dimensional convolution process, the mathematical expression of three-dimensional convolution is:

[0108]

[0109] Wherein, represents the result obtained by the i-th layer j-th feature map at (x, y, z) position, is an activation function, τ represents the number of channels, η, γ and δ respectively represent the length, width and channel direction dimension of the three-dimensional convolution kernel, represents the weight of the three-dimensional convolution kernel at the τ-th feature map (λ, σ, ρ), b i,j is a bias parameter.

[0110] (2) Building a two-dimensional convolutional neural network:

[0111] The two-dimensional convolutional neural network is set to include four two-dimensional convolutional blocks connected in series, with X T As the input of the hyperspectral remote sensing data, the convolution kernels with channel numbers n1, n2, n3, and n4 are used to extract spatial features from the input data: (a x a, n1), (a x a, n2), (a x a, n3), and (a x a, n4).

[0112] In the two-dimensional convolutional neural network, the input data is convolved with the two-dimensional kernel function to obtain a two-dimensional feature map, and the convolution features are processed through an activation function. In the two-dimensional convolution process, the mathematical expression of two-dimensional convolution is represented as:

[0113]

[0114] wherein, represents the result obtained by the i-th layer and the j-th feature map at the (x, y) position, is an activation function, τ represents the channel number, η and γ represent the length and width of the three-dimensional convolution kernel, respectively, represents the weight of the two-dimensional convolution kernel at the (λ, σ) position of the τ-th feature map, b i,j is a bias parameter.

[0115] (3) In the four convolutional blocks of the three-dimensional convolutional neural network, an attention module is inserted between every two adjacent convolutional blocks.

[0116] In the four convolutional blocks of the two-dimensional convolutional neural network, an attention module is inserted between every two adjacent convolutional blocks.

[0117] (4) The attention module is set to include a channel attention module and a spatial attention module.

[0118] A1) The attention module is set as follows:

[0119] In the channel attention module, the feature map F ∈ R w×w×d×n is taken as the input, where w x w is the size, d is the spectral dimension, and n is the channel number.

[0120] First, the features and

[0121] Second, the shared network SN includes an up-convolution layer, an activation function, and a down-convolution layer from top to bottom, and the two features and are input into the shared network SN.

[0122] Finally, the output features are combined by summation and the channel attention map M is obtained by a sigmoid activation function C ;

[0123] The channel attention calculation method is as follows:

[0124]

[0125] Where, θ, θ' represent Sigmoid and Relu activation functions respectively, Avgpool, Maxpool represent global average pooling and global maximum pooling operations respectively, F is the original input feature map, SN represents a shared network; W0, W1 are the weights of the shared network SN, and the feature map F' ∈ R w×w×d×n is obtained by matrix multiplication, and the calculation method is as follows:

[0126]

[0127] Where, M C is the channel attention feature map, is the matrix multiplication operation, and F is the input feature map;

[0128] A2) Set the spatial attention module, the feature map is input into the spatial attention module after channel attention processing, and the spatial attention module is used to extract spatial information of different regions;

[0129] First, generate features and by global average pooling and global maximum pooling operations, and the two features have the same dimensions;

[0130] Then, generate a new feature

[0131] Finally, generate the spatial attention map M S by a convolution layer and a sigmoid function; the calculation method of spatial attention is as follows:

[0132]

[0133] Where, θ is a Sigmoid activation function, f K×K×K represents a convolution operation with a convolution kernel of KxKxK, Avgpool, Maxpool represent global average pooling and global maximum pooling operations respectively, F' is the feature map output by the channel attention module,

[0134] and the feature map F'' ∈ R w×w×d×n is obtained by matrix multiplication, and the calculation method is as follows:

[0135]

[0136] wherein M S is a spatial attention feature map, is a matrix multiplication operation, and F' is a feature map after channel attention processing.

[0137] (4) using a three-dimensional convolution layer and a three-dimensional global average pooling layer after the three-dimensional convolutional neural network to become a one-dimensional array, using a two-dimensional convolution layer and a two-dimensional global average pooling layer after the two-dimensional convolutional neural network to become a one-dimensional array, then connecting the one-dimensional array output by the three-dimensional convolutional neural network and the one-dimensional array output by the two-dimensional convolutional neural network, and connecting into the Flatten layer and the fully connected layer;

[0138] A Relu activation function is used after each convolution layer in the three-dimensional convolutional neural network and the two-dimensional convolutional neural network to guide nonlinearity and padding is used, and the mathematical expression of the Relu activation function is as follows:

[0139]

[0140] ReLU is a piecewise linear function, which compares the input data x with the value 0, and outputs the maximum value, that is, all negative values are changed to 0, while the positive values remain unchanged.

[0141] Third step, training of the attention joint network: inputting the preprocessed hyperspectral image into the attention joint network, using the convolutional neural network with attention to extract features, then fusing the features and training and classifying.

[0142] The training of the attention joint network includes the following steps:

[0143] (1) the preprocessed hyperspectral remote sensing image is centered on the neighborhood pixels of each ground object sample to form an attention joint network training data set, and each sample size is a three-dimensional cube of w x w x d.

[0144] (2) inputting the attention joint network training data set into the three-dimensional convolutional neural network module with attention, using a three-dimensional convolution layer to extract spatial spectral features, and using an attention module to selectively learn image features to obtain a three-dimensional output feature map F 3D ∈R w×w×c×n′ wherein w x w is the size of the feature map, c is the spectral dimension, and n' represents the number of channels; then using a 1 x 1 x d convolution and a three-dimensional global average pooling operation to obtain a 1 x n4 spatial feature two-dimensional vector;

[0145] The three-dimensional hyperspectral training data set is converted into a two-dimensional matrix by Reshape remodeling and input into the two-dimensional convolutional neural network module with attention, the two-dimensional convolutional layer is used for extracting spatial features, and the attention module is used for selectively learning image features, to obtain a two-dimensional output feature map F 2D ∈R w×w×n′ , wherein w x w is the size of the feature map, c is the spectral dimension, and n' represents the number of channels, and then the output features are subjected to 1 x 1 convolution and two-dimensional global average pooling operation to obtain a 1 x n4 spectral feature two-dimensional vector;

[0146] The spatial spectral features extracted in the two paths of three-dimensional and two-dimensional are superimposed, are expanded through a Flatten layer, are prevented from overfitting by adopting two layers of Dropout layers, and finally are subjected to a fully connected layer with a Softmax function to obtain a classification result.

[0147] (3) In the network training process, a classification cross-entropy loss function is used to update the network parameters, and the expression is as follows:

[0148]

[0149] , wherein ∑ is a summation operation, log is a logarithm operation, represents the error loss between the predicted value and the true value; Y i ={y1,y2,...,y L} represents a true label vector, represents a predicted label vector; L is the number of samples, and s is the number of classifications;

[0150] The Adam optimization algorithm is used to optimize the network, the Adam replaces the first-order optimization algorithm in the traditional stochastic gradient descent process, and iteratively updates the neural network weight based on the training data. The preprocessed hyperspectral training sample is input into the attention-based joint network model for training, the corresponding weights are given in the spatial dimension and the channel dimension of the attention, the spatial spectral features in the hyperspectral image are selectively learned, different weights are allocated to different features, and the feature extraction capability of the network is further improved, to obtain the trained attention joint network model.

[0151] Fourth step, acquisition and preprocessing of classification samples: acquiring a hyperspectral image to be classified and performing preprocessing.

[0152] Fifth step, obtaining of a hyperspectral remote sensing image classification result: inputting the preprocessed hyperspectral image to be classified into the trained attention joint network to obtain a hyperspectral remote sensing image classification result.

[0153] The effect of the present application is further illustrated in combination with a simulation experiment:

[0154] 1. Simulation experiment conditions: The computer hardware environment of the experiment is AMD Ryzen7-4800H, GTX1650Ti, RAM 16GB, the software environment is Windows10 (64) bit operating system, the compiling environment is Spyder, and the deep learning framework adopts Keras. In order to verify the classification performance of the AUN method proposed in the application, the application verifies on Indiana Pines (IP) data set, Pavia University (UP) data set and Salinas (SA) data set. The detailed information of the three data sets is shown in Table 1, Figure 2 a, 2b is the false color image and the real classification map of the ground object of the IP data set, Figure 3 a, 3b is the false color image and the real classification map of the ground object of the PU data set, Figure 4 a, 4b is the false color image and the real classification map of the ground object of the SA data set.

[0155] Table 1 Detailed information of different data sets

[0156]

[0157] In addition, the application adopts overall accuracy (Overall Accuracy, OA), average accuracy (Average Accuracy, AA) and Kappa coefficient based on confusion matrix as evaluation indexes.

[0158] 2. Simulation experiment content and result analysis:

[0159] Example 1: The main parameters affecting the classification effect of the method include dimension d and spatial size w. 5%, 1%, 1% of the samples in the IP, PU and SA data sets are randomly extracted for training of the AUN, and the remaining samples are used for testing. In addition, the learning rate is set to 0.001, the dropout value is 0.4, the cross-entropy loss function and Adam optimization algorithm are used to train the AUN, the batch size is set to 64, and the training times are 150. Figure 5 The influence of the AUN method of the application on the classification accuracy under different dimensions d. For the IP data set, when the dimension is 25, the OA reaches the maximum value; with the continuous increase of d, the OA slightly decreases, because with the increase of d, although the spectral information of the image is increased, the redundancy of the spectrum is also increased. For the PU data set, when d increases to 20, the OA reaches the maximum value, and when d is greater than 20, the OA slightly decreases; for the SA data set, when the dimension increases to 20, with the continuous increase of d, the OA presents a stable trend. Therefore, for IP, PU and SA, the dimensions are respectively taken as 21, 19 and 19, so that the calculation complexity is reduced while the ideal classification accuracy is obtained.

[0160] Table 2 is the influence of different data sets on the classification accuracy under different w. In the three data sets, the size of w is set to 13, 15, 17, 19, 21, 23, and 25, respectively, and the OA value is observed to analyze the influence of w on the experimental results. For the IP data set, as w increases, the OA value slowly rises, and when w = 21, the classification effect is the best; when w continues to increase, OA does not rise but slightly decreases, indicating that when w is too large, similar categories may have a negative impact on the classification results. If the w value is too small, the extracted features may not be representative enough. Therefore, for the IP data set, the w value is set to 21. Similarly, the w values of the PU and SA data sets are both 19.

[0161] Table 2 Influence of different data sets on OA under different w

[0162]

[0163] Example 2: In order to further verify the effectiveness of the algorithm of the present application, 5%, 1%, and 1% of each type of feature in the IP, UP, and SA data sets are randomly selected as the training sample set, and the rest is used as the test sample set. Four kinds of hyperspectral image classification methods, Res-2D-CNN, Res-3D-CNN, HybridSN, and R-HybridSN, are used as comparative experimental objects, the classification results are taken as the average of ten experimental results, and the standard deviation is recorded, so as to verify the classification performance of the AUN method.

[0164] Table 3 Classification accuracy of different classification methods on the IP data set

[0165]

[0166] Table 4 Classification accuracy of different classification methods on the PU data set

[0167]

[0168]

[0169] Table 5 Classification accuracy of different classification methods on the PU data set

[0170]

[0171] Tables 3, 4, and 5 show the classification accuracy of different methods on three data sets, Figure 6 , Figure 7 , Figure 8 respectively show the real feature distribution of the IP, PU, and SA data sets and the classification result graphs of the five methods, Res-2D-CNN, Res-3D-CNN, HybridSN, R-HybridSN, and AUN, on the three data sets.

[0172] From the data and classification result figures, it can be seen that the AUN has the highest overall classification accuracy in the classification results of IP, PU, and SA data sets. Among the comparative models, the OA of Res-2D-CNN on the three data sets is lower than that of other comparative models, indicating that the Res-2D-CNN model is not suitable for small sample hyperspectral classification. Secondly, the classification result of Res-3D-CNN is higher than that of Res-2D-CNN, indicating that effectively exploring the spectral features of the training samples can significantly improve the classification accuracy. At the same time, AUN, R-HybridSN and HybridSN are all better than Res-3D-CNN in classification effect, because these three models also make full use of the large amount of spatial information contained in the hyperspectral image when extracting spectral features, enhancing the discriminant ability of the algorithm and improving the classification accuracy. This also proves to some extent that compared with using 3D or 2D convolution layer alone, combining 3D and 2D convolution layer is more suitable for classification under small sample condition.

[0173] In addition, the AUN method effectively solves the problem of low classification accuracy of Alfalfa in the IP data set. And in some classes with few training samples and easy to misclassify, such as Grass-pasture-mowed, Oats, Stone-Steel-Towers, the OA value still reaches more than 95%. Among the three joint 3D and 2D convolution layer models, the classification accuracy of AUN in the three data sets is relatively balanced, further proving the necessity of joint feature extraction and attention module.

[0174] Example Three: In order to further verify the classification performance of AUN when the training sample is limited, different proportions of IP, PU, and SA data sets are taken for experiment. For the IP data set, the proportions taken are: 1%, 3%, 5%, 7%, and 10%; for the PU and SA data sets, the proportions taken are: 0.1%, 0.5%, 1%, 3%, and 5%, and the experimental results are as shown in Figure 9 Figure 9 a is the classification result figure of different methods under different IP data set sample proportions, Figure 9 b is the classification result figure of different methods under different PU data set sample proportions, Figure 9 c is the classification result figure of different methods under different SA data set sample proportions, from Figure 9 a, it can be seen that due to the small sample size of the IP data set and the chaotic class, the classification effect of the CNN algorithm is not ideal. However, the classification accuracy of AUN is still the highest. Figure 9 From b, it can be seen that for the PU data set, even in the case of fewer training samples and higher class complexity, AUN can still maintain the highest classification accuracy by extracting deeper features. Figure 9 ​It can be seen from c that for the SA data set, the AUN classification accuracy is the highest and the first to reach stability when the training sample amount is only 1%. It is shown comprehensively that in view of the problems of insufficient performance of the current neural network classification and limited training samples in hyperspectral image classification, the application proposes a hyperspectral image classification method based on an attention joint network from the perspective of network optimization. The network is composed of two branches of 3D-CNN and 2D-CNN. And the attention mechanism is introduced in the two feature extraction branches, which is used for selective learning of hyperspectral image features. Further improve the feature extraction capability of the network. The IP, PU and SA three data sets are used to test the network, and the experimental results show that compared with the comparison method, the method has better classification performance.

[0175] The above shows and describes the basic principles, main features and advantages of the application. Those skilled in the art should understand that the application is not limited to the above examples, and the above examples and descriptions in the specification are only the principles of the application. Without departing from the spirit and scope of the application, various changes and improvements can be made to the application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection claimed by the application is defined by the appended claims and their equivalents.

Claims

1. A hyperspectral remote sensing image classification method based on an attention joint network, characterized in that, The method comprises the following steps: 11) obtaining and preprocessing of training samples: obtaining hyperspectral images to be trained and preprocessing; 12) constructing an attention joint network: based on a three-dimensional and two-dimensional convolutional neural network model, an attention mechanism is introduced to establish an attention joint network with two feature extraction branches; The constructing an attention joint network comprises the following steps: 121) building a three-dimensional convolutional neural network: The three-dimensional convolutional neural network comprises four three-dimensional convolutional blocks connected in X P As the input of the hyperspectral remote sensing data, the convolution kernels with the channel numbers of n1, n2, n3 and n4 are respectively used to input the data (a×a×b, n1), (a×a×c, n2), (a×a×a, n3) and (a×a×a, n4) to extract the spatial spectral features. In the three-dimensional convolutional neural network, the input data is convolved with the three-dimensional kernel function, then the nonlinearity is induced through the activation function, and the three-dimensional feature map is generated by convolving the extracted spectral band with the three-dimensional kernel function; 122) building a two-dimensional convolutional neural network: The setting two-dimensional convolutional neural network comprises four two-dimensional convolutional blocks connected in X T As the input of the hyperspectral remote sensing data, the convolution kernels with the channel numbers n1, n2, n3 and n4 are respectively used to extract the spatial features of the input data, that is, (a x a, n1), (a x a, n2), (a x a, n3) and (a x a, n4). In the two-dimensional convolutional neural network, the input data is convolved with the two-dimensional kernel function to obtain a two-dimensional feature map, and the convolution features are processed through the activation function; 123) for each two adjacent convolution blocks in the four convolution blocks of the three-dimensional convolutional neural network, an attention module is inserted; for each two adjacent convolution blocks in the four convolution blocks of the two-dimensional convolutional neural network, an attention module is inserted; 124) a three-dimensional convolution layer and a three-dimensional global average pooling layer are used after the three-dimensional convolutional neural network to become a one-dimensional array, and a two-dimensional convolution layer and a two-dimensional global average pooling layer are used after the two-dimensional convolutional neural network to become a one-dimensional array, then the one-dimensional array output by the three-dimensional convolutional neural network is connected with the one-dimensional array output by the two-dimensional convolutional neural network, and connected to the Flatten layer and the fully connected layer; A Relu activation function is used after each convolution layer in the three-dimensional and two-dimensional convolutional neural networks to guide nonlinearity and padding is used; 13) training of the attention joint network: input the preprocessed hyperspectral image into the attention joint network, extract features using the convolutional neural network with attention, then fuse the features and perform training and classification; 14) obtaining and preprocessing of classification samples: obtaining hyperspectral images to be classified and preprocessing; 15) obtaining of hyperspectral remote sensing image classification results: inputting the preprocessed hyperspectral image to be classified into the trained attention joint network to obtain the hyperspectral remote sensing image classification results.

2. The hyperspectral remote sensing image classification method based on an attention joint network according to claim 1, characterized in that, The obtaining and preprocessing of training samples comprises the following steps: 21) obtaining sample X in each class of ground object class sample in the hyperspectral image as a training sample, and sample X is represented as: ， Wherein, M, N, B respectively represent the width, height, and spectral dimension of the hyperspectral remote sensing data, is the i-th sample of the hyperspectral data; 22) dimension reduction of sample X by principal component analysis: using eigenvalue decomposition method to solve the eigenvalue and corresponding eigenvector of the covariance matrix E, and the calculation formula is as follows: ， where E is a covariance matrix, A is a matrix of eigenvectors, is the transpose of A, is a diagonal matrix of eigenvalues of the covariance matrix, The result after dimension reduction is represented as: ， wherein X P is the reduced hyperspectral data, A is the transformation matrix, and X is the original hyperspectral data; 23) The high-dimensional hyperspectral image is sampled with the center pixel size w x w field and its corresponding class label to obtain training samples X P and its label Y P The size of X P is w x w x d, and the size of Y is w x w, where w x w represents width and height respectively, and d represents the spectral dimension; 24) Transform the training sample data X P into a two-dimensional matrix X T with data size (w x w, d), where each row represents the spectral information contained in a sample, and each column represents a different spectral dimension. Transform the label data Y P into Y T with data size (w x w, 1). 3.The hyperspectral remote sensing image classification method based on attention joint network according to claim 1, characterized in that, The constructing an attention joint network comprises the following steps: 31) in the three-dimensional convolution process in the step of building a three-dimensional convolutional neural network, the mathematical expression of three-dimensional convolution is represented as: ， wherein, represents the result obtained by the jth feature map of the ith layer at the (x, y, z) position, is an activation function, represents the number of channels, , and respectively represent the dimensions of the length, width and channel direction of the three-dimensional convolution kernel, represents the weight of the three-dimensional convolution kernel at the jth feature map of the ith layer, is a bias parameter;​​ 32) in the two-dimensional convolution process in the step of building a two-dimensional convolutional neural network, the mathematical expression of two-dimensional convolution is represented as: ， wherein, represents the result obtained by the jth feature map of the ith layer at the (x, y) position, is an activation function, represents the number of channels, and respectively represent the length and width of the three-dimensional convolution kernel, represents the weight of the two-dimensional convolution kernel at the jth feature map of the ith layer, is a bias parameter;​​ 33) setting the attention module includes a channel attention module and a spatial attention module; 331) the attention module is set as follows: In the channel attention module, the feature map is taken as input, where w x w is the size, d is the spectral dimension, and n is the number of channels. First, features are generated by max-pooling and average-pooling operations and ; Secondly, the shared network SN comprises an upper convolutional layer, an activation function, and a lower convolutional layer from top to bottom, and the two features and are input into the shared network SN; Finally, the output features are combined using summation and passed through a sigmoid activation function to obtain the channel attention map ; The channel attention calculation method is as follows: ， wherein, , denote Sigmoid and Relu activation functions, Avgpool, Maxpool denote global average and maximum pooling operations, respectively, is the original input feature map, SN denotes a shared network; , is the weight of the shared network SN, and the feature map is obtained by matrix multiplication, and the calculation method is as follows: ， wherein, is a channel attention feature map, is a matrix multiplication operation, is an input feature map; 332) Set the spatial attention module, the feature map is input into the spatial attention module after channel attention processing, and the spatial attention module is used for extracting spatial information of different regions; First, the features are generated by global average pooling and global max pooling operations and and the two features have the same dimension; Then, new features are generated by joint operations ; Finally, the spatial attention map is generated by a convolutional layer and a sigmoid function The method for calculating the spatial attention is as follows: ， wherein, is a Sigmoid activation function, represents a convolution operation with a convolution kernel Avgpool, Maxpool represent global average pooling and global maximum pooling operations, respectively, is a feature map output by the channel attention module, Feature maps are obtained using matrix multiplication The calculation method is as follows: ， wherein, is a spatial attention feature map, is a matrix multiplication operation, is a feature map processed by channel attention. 34) In step 124), the mathematical expression of the Relu activation function is as follows: 。 4. Relu is a piecewise linear function, which compares the input data x with the value 0, and outputs the maximum value, that is, all negative values are changed to 0, while the positive values remain unchanged. 5.The hyperspectral remote sensing image classification method based on attention joint network according to claim 1, characterized in that, The training of the attention joint network includes the following steps: 41) The preprocessed hyperspectral remote sensing image is composed of attention joint network training data set with each land cover sample as the center of the neighborhood pixel, and each sample size is a three-dimensional cube of w x w x d; 42) The attention joint network training data set is input into the three-dimensional convolutional neural network module with attention, the three-dimensional convolutional layer is used for extracting spatial spectral features, and the attention module is used for selectively learning image features to obtain a three-dimensional output feature map wherein is the size of the feature map, c is the spectral dimension, represents the number of channels; then the output features are subjected to a 1x1xd convolution and a three-dimensional global average pooling operation to obtain a 1xn4 spatial-spectral feature two-dimensional vector; The two-dimensional matrix converted by reshaping and remodeling the three-dimensional hyperspectral training data set is input into the two-dimensional convolutional neural network module with attention, the two-dimensional convolutional layer is used for extracting spatial features, and the attention module is used for selectively learning image features to obtain a two-dimensional output feature map wherein is the size of the feature map, c is the spectral dimension, represents the number of channels, and then the output features are subjected to 1×1 convolution and two-dimensional global average pooling operation to obtain a 1×n4 spectral feature two-dimensional vector; The spatial spectral features extracted in the three-dimensional and two-dimensional two paths are superimposed, expanded through the Flatten layer, and prevented from overfitting through two layers of Dropout layer, and finally a fully connected layer with Softmax function is used to obtain the classification result; 43) In the network training process, the classification cross-entropy loss function is used to update the network parameters, and the expression is as follows: ， where ∑ is a summation operation, and log is a logarithm operation, denotes an error loss between the predicted value and the true value; denotes a true label vector, denotes a predicted label vector; L is the number of samples, and s is the number of classifications. The network is optimized using the Adam optimization algorithm. Adam replaces the first-order optimization algorithm in the traditional stochastic gradient descent process, and iteratively updates the neural network weights based on the training data.

Images