Hyperspectral image classification method based on attention mechanism and space-spectrum joint residual network

Abstract

The application discloses a hyperspectral image classification method based on an attention mechanism and a space-spectrum combined residual network, and has the characteristics that the method comprises the following steps: constructing a training pixel set and a test pixel set in proportion, and constructing corresponding stereoscopic blocks with each pixel in the training pixel set and the test pixel set as the center to obtain a training sample set and a test sample set; constructing a neural network; performing forward propagation on the neural network through the training sample set and the test sample set, determining a loss by using a loss function according to a real label, and then performing backward propagation on the neural network through the loss function to obtain a trained neural network; and performing classification test on the test sample set through the trained model, and taking an overall accuracy OA, an average accuracy AA and a Kappa coefficient as evaluation indexes of classification performance.

Description

Technical Field

[0001] This invention belongs to the fields of remote sensing and image processing, and specifically relates to a hyperspectral image classification method based on an attention mechanism and a spatial-spectral joint residual network. Background Technology

[0002] Hyperspectral remote sensing originated in the early 1980s, evolving from traditional multispectral remote sensing technology. It is a novel remote sensing technology that combines imaging and spectral measurement. Hyperspectral remote sensing utilizes imaging spectrometers to image ground features across hundreds of continuous and narrow spectral bands, acquiring approximately continuous spectral curves that reflect the characteristics of the ground features. Hyperspectral remote sensing images organically combine two-dimensional spatial information reflecting the spatial, geometric, and textural characteristics of ground features with one-dimensional spectral information reflecting their reflectance, radiation, or scattering characteristics, achieving a unified image and spectrum. Each pixel in a hyperspectral image has a unique spectral curve, reflecting the characteristics of the corresponding landmass.

[0003] Compared to traditional remote sensing images, hyperspectral remote sensing images offer advantages such as high spectral resolution, a wide spectral range, multiple and continuous spectral bands, and integrated image and spectrum representation. Hyperspectral images not only provide spatial information about ground targets but also offer rich spectral information. This integrated image and spectrum representation allows hyperspectral remote sensing images to contain more comprehensive data, facilitating more effective target detection and identification.

[0004] In the field of hyperspectral remote sensing, hyperspectral image classification is one of the research hotspots. Hyperspectral image classification refers to assigning a category label to each pixel in an image based on the rich data information contained in the hyperspectral image, and it is widely used in urban road recognition, forest tree species identification, land use surveying, and other fields. Therefore, hyperspectral image classification has important value for the national economy and national defense, and the research on hyperspectral image classification algorithms is of great significance.

[0005] Over the past two decades, scholars both domestically and internationally have conducted extensive research on hyperspectral image classification and proposed a series of hyperspectral image classification methods. Classic algorithms include the k-Nearest Neighbor (k-NN) algorithm, Support Vector Machine (SVM) algorithm, and Extreme Learning Machine (ELM) algorithm.

[0006] Since the emergence of deep learning, due to its superior feature extraction capabilities, scholars have proposed numerous hyperspectral image classification methods based on deep learning. Among them, convolutional neural networks (CNNs) are one of the most representative deep learning methods.

[0007] However, due to the unique imaging mechanism of hyperspectral images, hyperspectral image classification tasks face challenges such as the problem of different spectra for the same object and different objects with the same spectrum. Therefore, it is difficult to achieve good classification results using only spectral information. In addition, when extracting features from hyperspectral images, CNNs inevitably extract many redundant features. Since CNNs treat all extracted features equally, these redundant features will affect the classification performance of CNNs. Summary of the Invention

[0008] In view of this, the main objective of the present invention is to provide a hyperspectral image classification method based on an attention mechanism and a spatial-spectral joint residual network.

[0009] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0010] This invention provides a hyperspectral image classification method based on an attention mechanism and a spatial-spectral joint residual network. The method includes the following steps:

[0011] Construct training and testing pixel sets proportionally, and construct corresponding 3D blocks centered on each pixel in the training and testing pixel sets to obtain training and testing sample sets.

[0012] Constructing a neural network;

[0013] The neural network is forward-propagated using the training and testing sample sets, and the loss is determined using the loss function based on the real labels. Then, the neural network is back-propagated using the loss function to obtain the trained neural network.

[0014] The trained model was used to classify the test sample set, and the overall accuracy (OA), average accuracy (AA), and Kappa coefficient were used as evaluation metrics for classification performance.

[0015] In the above scheme, the step of constructing a training pixel set and a test pixel set proportionally, and constructing corresponding 3D blocks centered on each pixel in the training pixel set and the test pixel set to obtain the training sample set and the test sample set, specifically involves:

[0016] Step 1.1: A hyperspectral remote sensing image with n rows, m columns, and d spectral bands. If the maximum element of tensor H0 is max(H0) and the minimum element is min(H0), then the normalized image expression is: To normalize all element values ​​in an image to between 0 and 1;

[0017] Step 1.2: Introduce the PCA function and perform PCA dimensionality reduction by extracting principal components to transform the hyperspectral image H1 with n rows, m columns and d spectral bands into a hyperspectral image H with n rows, m columns and C spectral bands.

[0018] Step 1.3: Randomly select r% of the available labeled pixels from each category for training, and use the remaining t% of the available labeled pixels for testing, resulting in a total of R training pixels and T test pixels; where t% = 1 - r%.

[0019] Step 1.4: Each sample consists of a 3D block S with H rows, W columns, and C spectral bands centered on the location of the center pixel. The R training pixels and T test pixels constitute R training samples and T test samples, respectively. The R training samples together constitute the training sample set Training_Set, and the T test samples together constitute the test sample set Test_Set.

[0020] In the above scheme, the construction of the neural network specifically refers to:

[0021] Step 2.1: Construct a 2-D residual block. The input data of the 2-D residual block is processed by one 2-D convolution, one ReLU operation, and one 2-D convolution. The calculation result is then added to the input data. Finally, the sum is processed by one ReLU operation and output.

[0022] Step 2.2: Construct a 3D residual block. The input data of the 3D residual block is processed by a 3D convolution, a ReLU operation, and a 3D convolution. The calculation result is then added to the input data. Finally, the sum is processed by a ReLU operation and output.

[0023] Step 2.3: Construct a channel attention module, perform global max pooling and global average pooling on the input feature tensor in the spatial direction to obtain the max pooling vector and the average pooling vector, and pass them through a multilayer perceptron in the form of two branches. Add the output vectors of the two branches, process them through the Sigmoid function, and output the result. The output result suppresses or enhances the channels of the spectral dimension of the input feature tensor accordingly.

[0024] Step 2.4: Construct a spatial attention module. Calculate the maximum and average values ​​of the input feature tensor along the spectral direction to obtain the max pooling matrix and average pooling matrix. Concatenate the two two-dimensional matrices to obtain a three-dimensional tensor with two channels. Perform a 2-D convolution on this tensor with a kernel of 1, and then process it with the Sigmoid function before outputting the result. This output suppresses or enhances the spatial dimension of the input feature tensor accordingly.

[0025] Step 2.5: Construct an attention-assisted 3D residual network. The front end is a 3D convolutional block, followed by three cascaded 3D residual blocks. Each residual block has two 3D convolution processes. The spectral dimension and feature map dimension of the output data of the last 3D residual block are merged and concatenated on the spectral dimension to form a feature map. Finally, a channel attention module and a spatial attention module are used.

[0026] Step 2.6: Construct an attention-assisted 2D residual network. The front end is a 2D residual block, which has two 2D convolution processes. The 2D residual block is followed by a 2D convolution block, and finally a spatial attention module is used.

[0027] Step 2.7: Construct a fully connected module. After vectorizing the output of the attention-assisted 2-D residual network, perform three fully connected operations, with the first two fully connected operations followed by the Dropout function, and finally output the classification category.

[0028] In the above scheme, the forward propagation of the neural network using the training sample set and the test sample set, the determination of the loss using the loss function based on the true labels, and the back propagation of the neural network using the loss function to obtain the trained neural network are specifically as follows:

[0029] Step 3.1: Define the number of training set iterations (Epoch), the batch size (Batch_Size), and the total number of training samples (S). Then, define the number of batches trained in each iteration cycle.

[0030] Step 3.2: Define the learning rate lr, and the total number of categories as Class;

[0031] Step 3.3: Set the loss function for the forward propagation of the neural network. Use CrossEntropyLoss, a commonly used loss function in neural networks, as the loss function. The network output is y, and the true label is... Then, the loss is calculated using cross-entropy based on the true labels.

[0032] Step 3.4: Optimize all parameters of the neural network using the Adam optimizer. The learning rate in the Adam optimizer is lr, and the loss function is... Differentiate all parameters in the network and perform backpropagation;

[0033] Step 3.5: Determine if the total number of training iterations is equal to the number of Epochs. If it is, end the training; otherwise, continue training and return to step 3.3.

[0034] Compared with existing technologies, this invention improves the feature extraction performance of the model by simultaneously extracting spatial and spectral features through a spatial-spectral joint residual network and suppressing redundant features through an attention mechanism, thereby achieving high-accuracy hyperspectral image classification. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and, together with their descriptions, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0036] Figure 1 This is a schematic diagram of the residual block structure in an embodiment of the present invention;

[0037] Figure 2 This is a network structure diagram provided for an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, article, or apparatus that includes that element.

[0040] This invention provides a hyperspectral image classification method based on an attention mechanism and a spatial-spectral joint residual network, such as... Figure 1 , Figure 2 As shown, it includes the following steps:

[0041] Step 1: Perform dataset preprocessing, including pixel normalization and PCA dimensionality reduction. Construct training and testing pixel sets proportionally, and then construct corresponding 3D blocks centered on each pixel in the training and testing pixel sets to obtain the training and testing sample sets.

[0042] Step 1.1: The University of Pavia dataset is one of the most commonly used datasets in the field of hyperspectral image classification. Therefore, this invention uses the University of Pavia dataset as the dataset for the hyperspectral image classification task. The University of Pavia dataset was obtained by the ROSIS imaging spectrometer over the city of Pavia in northern Italy. It contains 610×340 pixels with a spatial resolution of 1.3m. This dataset contains a total of 9 pixel categories (N=9), with 103 spectral bands covering a wavelength range of 0.43–0.86 μm. This hyperspectral remote sensing image... If the maximum value of the tensor H0 is max(H0) = 8000 and the minimum value of the tensor H0 is min(H0) = 0, then the normalized image expression is:

[0043]

[0044] This achieves the normalization of all element values ​​in the image to between 0 and 1.

[0045] Step 1.2: Introduce the PCA function, with the number of principal components pca_components = 30. By extracting the principal components, perform PCA dimensionality reduction to convert the 610-row, 340-column hyperspectral image H1 with 103 spectral bands into a 610-row, 340-column hyperspectral image H with 30 spectral bands.

[0046] Step 1.3: The hyperspectral image H contains num_pixel0 = 610 × 340 = 207400 pixels. The sample 3D block size is 25 × 25 × 30. The available labeled pixels are those labeled pixels that can be used as the center of the sample 3D block, that is, the non-zero labeled pixels remaining after removing the outermost 12 layers of pixels, totaling num_pixel = 42776 pixels. For each of the N = 9 classes of available labeled pixels, 10% are randomly selected for training, and the remaining 90% are used for testing. A total of 4277 training pixels and 38499 test pixels are obtained.

[0047] Step 1.4: Each sample consists of a 25×25×30 3D block S centered on the location of the center pixel. The 4277 training pixels and 38499 test pixels from Step 1.3 constitute the 4277 training samples and the 38499 test samples, respectively. The 4277 training samples together form the training sample set Training_Set, and the 38499 test samples together form the test sample set Test_Set.

[0048] Step 2: Construct the various sub-modules and sub-networks of the neural network:

[0049] Step 2.1: Construct the 2-D residual block. The input to the 2-D residual block is h. l Let the process of "one 2D convolution, one ReLU operation, and one 2D convolution" be denoted by the function F(·), where the kernel size of the 2D convolution is 3×3. Then h l After a process involving one 2D convolution, one ReLU operation, and one more 2D convolution within the 2D residual block, F(h) is obtained. l h l Using Shortcut and F(h) l The two numbers are added together, and finally, after a ReLU operation, the output h is obtained. l+2 This process can be expressed by the formula as follows:

[0050]

[0051] Step 2.2: Construct the 3-D residual block. The input to the 3-D residual block is h. l Let the process of “one 3D convolution, one ReLU operation, and one 3D convolution” be denoted as function F. 3-D (·). The kernel size for the first convolution process is 5×3×3 (3×3 in spatial dimension and 5 in spectral dimension), and the kernel size for the second convolution process is 3×3×3. Then h l After undergoing one 3D convolution, one ReLU operation, and one 2D convolution within the 3D residual block, F is obtained. 3-D (h l h l via Shortcut and F 3-D (h l The two numbers are added together, and finally, after a ReLU operation, the output h is obtained. l+2 This process can be expressed by the formula as follows:

[0052]

[0053] Step 2.3: Construct the channel attention module. The input feature tensor... Global max pooling and global average pooling are performed separately in the spatial direction to obtain the max pooling vector. and average pooling vector The value of the input feature tensor F at the (i, j)th position in the spatial direction and the cth element in the spectral direction is F. c (i, j), then the max pooling vector v max The value of the c-th element can be represented as

[0054] v max (c)=F c (4)

[0055] Average pooling vector vavg The value of the c-th element can be represented as

[0056]

[0057] v is presented in a two-branch form max and v avg The corresponding vectors v′ are obtained by using a multilayer perceptron (MLP). max and v′ avg The MLP is designed as a bottleneck structure, first reducing dimensionality and then increasing it. The weights for the dimensionality reduction process are W1, and the weights for the dimensionality increase process are W2. The bottleneck structure reduces model complexity. Furthermore, the max-pooling vector v... max and average pooling vector v avg When passing through the MLP, it shares the weights W1 and W2 of the MLP. Therefore, v′ max It can be represented as

[0058] v′ max =W2σ(W1v) max (6)

[0059] v′ avg It can be represented as

[0060]

[0061] Where σ(·) represents the ReLU function. The output vectors v′ of the two branches are... max and v′ avg The sum is processed by the Sigmoid function, and the output is v. out ,Right now

[0062]

[0063] Where τ(·) represents the ReLU function. This output result is applied to the input feature tensor spectral dimension channel F. c To perform corresponding suppression or enhancement, denoted as

[0064]

[0065] Step 2.4: Construct the spatial attention module. The input feature tensor... The maximum and average values ​​are calculated along the spectral direction to obtain the max-pooling matrix. and average pooling matrix The value of the input feature tensor F at the (i, j)th position in the spatial direction and the cth element in the spectral direction is F. c (i, j), max pooling matrix M max The value of the (i, j)th element is M. max(i, j), average pooling matrix M avg The value of the (i, j)th element is M. avg (i, j). Then the maximum pooling matrix M max The value of the (i, j)th element can be represented as

[0066]

[0067] Average pooling matrix M avg The value of the (i, j)th element can be represented as

[0068]

[0069] Two 2D matrices are concatenated to obtain a 3D tensor with two channels. This tensor is then subjected to a 2D convolution with a single kernel, and finally processed by the Sigmoid function before output. The final output M is... out It is a two-dimensional matrix representing the weight coefficients of each pixel in the spatial direction, which can be represented as:

[0070] M out =τ([M max M avg ]*W) (12)

[0071] Where * represents the convolution operation, and W represents the parameters of the convolutional layer. M out By performing corresponding suppression or enhancement on the position of the input feature tensor F in space dimension, we obtain This process can be represented as

[0072] F′(i,j)=M out (i, j)F(i, j) (13)

[0073] Step 2.5: Construct an attention-assisted 3D residual network. The front end is a 3D convolutional block Conv3d_1 with 8 kernels and a size of 7×7×7 (7×7 spatial dimension, 7 spectral dimension). This is followed by three cascaded 3D residual blocks Res3d_1, Res3d_2, and Res3d_3. Each 3D residual block has the same structure, and the size and number of kernels at corresponding positions are also the same. Each residual block undergoes two 3D convolution processes: the first convolution process has a kernel size of 5×3×3 (3×3 spatial dimension, 5 spectral dimension), and the second convolution process has a kernel size of 3×3×3. The number of kernels in both convolution processes within the residual block is set to 1, thus maintaining a feature map size of 8 during the residual convolution process. The spectral dimension of the output data from the last 3D residual block is merged with the feature map dimension, meaning the 8 feature maps are concatenated into a single feature map along the spectral dimension. Finally, a Channel_Attention_1 module and a Spatial_Attention_1 module are used. In the Channel_Attention module, Bottleneck first reduces the number of channels in the max-pooling and average-pooling vectors to 1 / 16 of their original value, then increases them back to their original value. In the Spatial_Attention module, the convolutional kernel size is 7×7, and the number of kernels is 1.

[0074] Step 2.6: Construct an attention-assisted 2D residual network. The front end is a 2D residual block Res2d_1, which undergoes two 2D convolution processes. Following Res2d_1 is a 2D convolutional block Conv2d_1. The kernel size of both the 2D convolutional block and the two 2D convolution processes within the 2D residual block is 3×3. To reduce the complexity of subsequent fully connected layers and ensure the quality of extracted features, the number of kernels in the 2D convolutional block is set to 1 / 3 of the spectral depth of the input data. Finally, a spatial attention module, Spatial_Attention_2, is used. The spatial attention module has a kernel size of 7×7 and a kernel size of 1.

[0075] Step 2.7: Construct the fully connected module. Vectorize the output of Step 2.6 and connect it to 256 neurons, then to 128 neurons, and finally to N=9 neurons. After the first two fully connected operations, a Dropout function is applied. Finally, a Softmax operation is performed to output the classification category.

[0076] Step 3: Perform forward propagation of the neural network using the training dataset, calculate the loss using the loss function based on the true labels, and then perform backpropagation of the neural network model parameters using the loss function.

[0077] Step 3.1: Given that the training set iteration count is 100, the batch size is 128, and the total number of training samples is S = 4277, then the number of batches trained in each iteration period is...

[0078] Step 3.2: Set the learning rate lr = 0.001 and the total number of categories to Class = 9.

[0079] Step 3.3: Use cross-entropy as the loss function, the network output is y, and the true label is... The loss is then calculated using the loss function.

[0080]

[0081] Step 3.4: Optimize all parameters of the neural network using the Adam optimizer. The learning rate in the Adam optimizer is lr, and the loss function is... Differentiate all parameters in the network and perform backpropagation;

[0082] Step 3.5: Determine if the total number of training iterations is equal to the number of Epochs. If it is, end the training; otherwise, continue training and return to step 3.3.

[0083] Step 4: Use the trained model to perform classification tests on the test sample set, and use OA, AA, and Kappa coefficients as evaluation metrics for classification performance.

[0084] Step 4.1: The batch size for the test is also Batch_Size = 128, and the total number of test samples is T = 38499. Then, after... The batch operation yields the classification result for each test sample.

[0085] Step 4.2: Organize the classification results of all test samples to obtain a 9×9 confusion matrix C, where the (i, j)th element represents the number of samples of class j classified as class i.

[0086] Step 4.3: Based on the confusion matrix in Step 4.2, the ratio of each element value on the diagonal to the number of test samples corresponding to that class is the classification accuracy for each land cover class. Accuracy is calculated using OA, AA, and Kappa methods, all of which represent the classification accuracy of the entire dataset.

[0087] The overall accuracy (OA) is defined as follows:

[0088]

[0089] Among them, C iiIt is the value of the i-th element on the diagonal of the confusion matrix C, representing the number of test samples of class i that are correctly classified.

[0090] The average accuracy (AA) is defined as follows:

[0091]

[0092] Among them, Recall i This represents the ratio of the number of correctly predicted samples to the total number of samples in the i-th class of tests.

[0093] The Kappa coefficient is defined as follows:

[0094]

[0095] Among them, C +i It is the summation of the elements in the i-th row of the confusion matrix C, where C is the summation of the elements in the i-th row of the confusion matrix C. i+ It is the summation of the elements in the i-th column of the confusion matrix C.

[0096] To demonstrate the practicality of the method in this invention, SSRN, FDSSC, DFFN, and HPDM-SPRN were selected as comparison methods, and OA, AA, and Kappa coefficients were used as evaluation metrics for the algorithm's classification accuracy. By comparing the OA, AA, and Kappa coefficients and the classification accuracy of each land cover category with the comparison methods on the University of Pavia dataset, the practicality of the proposed classification method was objectively evaluated. The final results are shown in Table 1.

[0097] Table 1: Results of the comparative experiment

[0098]

[0099] Table 1 shows that the proposed hyperspectral image classification method based on attention mechanism and spatial-spectral joint residual network achieves an overall accuracy (OA) of 99.68% on the University of Pavia dataset. This is 2.58% higher than the comparative methods SSRN (OA: 97.10%), 1.32% higher than FDSSC (OA: 98.36%), 1.21% higher than DFFN (OA: 98.47%), and 0.69% higher than HPDM-SPRN (OA: 98.99%). Therefore, the OA is higher than all the comparative algorithms. The average accuracy (AA) reaches 99.08%, and the Kappa coefficient reaches 0.9958, both of which are higher than all the comparative algorithms.

[0100] The above results show that the hyperspectral image classification method based on attention mechanism and spatial-spectral joint residual network proposed in this invention has higher classification accuracy and its classification performance is more suitable for the classification needs of hyperspectral images.

[0101] This invention utilizes a 3-D residual network and a small segment of a 2-D residual network to simultaneously and precisely extract the spatial and spectral features of hyperspectral images, overcoming the problems of different spectra for the same object and different objects for the same spectrum, and effectively preventing the gradient vanishing phenomenon in deep neural networks.

[0102] This invention introduces spatial attention and channel attention mechanisms at the end of two residual networks to suppress useless spatial spectral features extracted by CNNs and enhance useful features, thereby improving classification performance. The scheme was ultimately tested on the University of Pavia dataset to achieve hyperspectral image classification accuracy of 99.68%, 99.08%, and 0.9958 for OA, AA, and Kappa coefficients, respectively.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A hyperspectral image classification method based on an attention mechanism and a spatial-spectral joint residual network, characterized in that, The method includes the following steps: Construct training and testing pixel sets proportionally, and construct corresponding 3D blocks centered on each pixel in the training and testing pixel sets to obtain training and testing sample sets. Constructing a neural network; The neural network is forward-propagated using the training and testing sample sets, and the loss is determined using the loss function based on the real labels. Then, the neural network is back-propagated using the loss function to obtain the trained neural network. The trained model was used to classify the test sample set, and the overall accuracy (OA), average accuracy (AA), and Kappa coefficient were used as evaluation metrics for classification performance. The construction of the neural network specifically involves: Step 2.1: Construct a 2-D residual block. The input to the 2-D residual block is... The process of "one 2D convolution, one ReLU operation, and one 2D convolution" is denoted as a function. The kernel size of the 2D convolution is 1. ,but After undergoing one 2D convolution, one ReLU operation, and one 2D convolution within the 2D residual block, we obtain... , via Shortcut and The ingredients are added together, and finally the result is output after a ReLU operation. This process can be expressed by the formula as follows: (2) Step 2.2: Construct 3D residual blocks. The input to the 3D residual blocks is... The process of "one 3D convolution, one ReLU operation, and one 3D convolution" is denoted as a function. The kernel size in the first convolution process is In terms of spatial dimension The spectral dimension is 5, and the kernel size in the second convolution process is... ,but After undergoing one 3D convolution, one ReLU operation, and one 2D convolution in the 3D residual block, we obtain... , via Shortcut and The ingredients are added together, and finally the result is output after a ReLU operation. This process can be expressed by the formula as follows: (3) Step 2.3: Construct a channel attention module to process the input feature tensor. Global max pooling and global average pooling are performed separately in the spatial direction to obtain the max pooling vector. and average pooling vector The input feature tensor In the spatial direction Position, spectral direction The value of the element Then the max pooling vector In the The value of an element can be represented as (4) Average pooling vector In the The value of an element can be represented as (5) and in the form of two branches and The corresponding vectors are obtained by using a multilayer perceptron (MLP). and The MLP is designed as a structure called Bottleneck, which first reduces dimensionality and then increases it. The weights for the dimensionality reduction process are... The weight of the dimensionality upgrade process is The Bottleneck structure can reduce model complexity and max pooling vectors. and average pooling vector The weights of the MLP are shared when passing through it. and ,then, It can be represented as (6) It can be represented as (7) in, Represents the ReLU function, which converts the output vectors of the two branches. and The sum is processed by the Sigmoid function and then output. ,Right now (8) in, Represents the ReLU function, and the output is obtained by considering the spectral dimension of the input feature tensor. To perform corresponding suppression or enhancement, denoted as (9) Step 2.4: Construct a spatial attention module to process the input feature tensor. The maximum and average values ​​are calculated along the spectral direction to obtain the max-pooling matrix. and average pooling matrix The input feature tensor In the spatial direction Position, spectral direction The value of the element Max pooling matrix In the The value of the element Average pooling matrix In the The value of the element Max pooling matrix In the The value of an element can be represented as (10) Average pooling matrix In the The value of an element can be represented as (11) Concatenate two 2D matrices to obtain a 3D tensor with two channels. Perform a 2D convolution on this tensor with a single kernel, then process it using the Sigmoid function before outputting the final output. It is a two-dimensional matrix representing the weight coefficients of each pixel in the spatial direction, which can be represented as: (12) in For convolution operations, These are the parameters of the convolutional layer. For the input feature tensor By suppressing or enhancing the position in the spatial dimension accordingly, we obtain... This process can be represented as (13) Step 2.5: Construct an attention-assisted 3D residual network, with the front end being a 3D convolutional block Conv3d_1 containing 8 kernels of a size of [size missing]. In terms of spatial dimension The spectral dimension is 7, followed by three cascaded 3-D residual blocks Res3d_1, Res3d_2, and Res3d_3. Each 3-D residual block has an identical structure, and the size and number of convolutional kernels at corresponding positions are also the same. Each residual block contains two 3-D convolution processes. The kernel size in the first convolution process is... In terms of spatial dimension The spectral dimension is 5, and the kernel size in the second convolution process is... In the residual block, the number of convolution kernels is set to 1 in both convolution processes, so the number of feature maps remains at 8 during the residual convolution process. The spectral dimension of the output data of the last 3-D residual block is merged with the feature map dimension, that is, the 8 feature maps are concatenated into one feature map on the spectral dimension. At the end, a channel attention module (Channel_Attention_1) and a spatial attention module (Spatial_Attention_1) are used. In the channel attention module, Bottleneck first reduces the number of channels of the max pooling vector and the average pooling vector to 1 / 16 of the original number, and then increases it back to the original number. The size of the convolution kernel in the spatial attention module is... The quantity is 1; Step 2.6: Construct an attention-assisted 2D residual network. The front end is a 2D residual block Res2d_1, which has two 2D convolution processes. Res2d_1 is followed by a 2D convolution block Conv2d_1. The kernel size of the 2D convolution block and the two 2D convolution processes in the 2D residual block is 2D. The number of kernels in the 2D convolutional blocks is set to 1 / 3 of the spectral depth of the input data. Finally, a spatial attention module, Spatial_Attention_2, is used, with a kernel size of [missing value]. The quantity is 1; Step 2.7: Construct a fully connected module, vectorize the output of Step 2.6, connect it to 256 neurons, then connect it to 128 neurons, and finally connect it to... The system has 10 neurons, and after the first two fully connected operations, it is followed by the Dropout function. Finally, after the Softmax operation, it outputs the classified category.

2. The hyperspectral image classification method based on attention mechanism and spatial-spectral joint residual network according to claim 1, characterized in that, The process of constructing training and testing pixel sets proportionally, and then constructing corresponding 3D blocks centered on each pixel in the training and testing pixel sets to obtain training and testing sample sets, specifically involves: Step 1.1: One OK, List, have Hyperspectral remote sensing images in spectral bands tensor The maximum value of the element is The minimum element value is The normalized image expression is then: This ensures that all element values ​​in the image are normalized to between 0 and 1. Step 1.2: Introduce the PCA function and perform dimensionality reduction by extracting principal components. OK, List, have Hyperspectral images in spectral bands Convert to OK, List, have Hyperspectral images in spectral bands ; Step 1.3: Randomly select pixels from each category. % of the available labeled pixels are used for training, the remainder % of the available labeled pixels were used for testing, and a total of % were obtained. One training pixel, One test pixel; among which... % % Step 1.4: Each sample is centered on the location of the center pixel. OK, List, have A three-dimensional block of spectral bands Composition, the training pixels and Each test pixel constitutes training samples, One test sample; The training samples together constitute the training sample set, Training_Set. The test samples together constitute the test sample set Test_Set.

3. The hyperspectral image classification method based on attention mechanism and spatial-spectral joint residual network according to claim 2, characterized in that, The process involves forward propagation of the neural network using the training and testing sample sets, determining the loss based on the true labels using a loss function, and then backpropagating the neural network using the loss function to obtain a trained neural network. Specifically: Step 3.1: Define the number of training set iterations Training batch size The total number of training samples is The number of batches trained in each iteration period ; Step 3.2: Define the learning rate The total number of categories is ; Step 3.3: Set the loss function for the forward propagation of the neural network. Use CrossEntropyLoss, a commonly used loss function in neural networks, as the loss function. The network output is... The real label is Then, the loss is calculated using cross-entropy based on the true labels. ; Step 3.4: Optimize all parameters of the neural network using the Adam optimizer. The learning rate in the Adam optimizer is... , loss function Differentiate all parameters in the network and perform backpropagation; Step 3.5: Determine if the total number of training iterations is equal to... If the condition is met, end the training; otherwise, continue training and return to step 3.3.

Images