A hyperspectral image classification method

Abstract

The application discloses a hyperspectral image classification method, and relates to the technical field of image processing, and comprises the following steps: preparing and preprocessing a data set; constructing a network model; selecting a loss function and an evaluation index; and training the network model; the spectral feature extraction module and the spatial feature extraction module are processed in parallel, different receptive field spectral and spatial features are acquired by adopting a multi-scale feature extraction mode, and a dynamic feature fusion module is adopted to adaptively fuse the spectral and spatial features, so that the classification precision, the training efficiency and the generalization of the model can be effectively improved, and good flexibility and expansibility are achieved; the dynamic feature fusion module can adaptively perform dynamic cross-feature interaction based on a cross-attention module, the cross-attention module can realize efficient interaction between spectral and spatial features based on a dynamic mechanism, and the generalization capability of the model for complex scenes is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a hyperspectral image classification method. BACKGROUND

[0002] Hyperspectral image classification aims to classify ground objects by analyzing rich spectral information and spatial structure information in pixels, and has very wide application prospects in many fields such as crop detection, geological exploration, environmental monitoring and military reconnaissance. The networks that can obtain good performance in hyperspectral image classification mainly include artificial neural networks, deep belief networks, graph neural networks and convolutional neural networks. Since the convolutional neural network has strong automatic feature extraction capability, it can effectively extract spectral and spatial information, so it has been widely applied in hyperspectral image classification methods. However, the classification method of the convolutional neural network pays more attention to local feature extraction in the local area due to the limitation of the local receptive field, and ignores the global context information. Moreover, the traditional convolutional neural network model needs a large number of labeled samples, has a large number of network parameters, and is prone to overfitting and poor generalization. Therefore, it has great practical significance to ensure the classification accuracy while reducing the parameter quantity and improving the training efficiency in the hyperspectral image classification.

[0003] The Chinese patent with the publication number CN116563606A and the name of "A hyperspectral image classification method based on a double-branch space-spectrum global feature extraction network" first pre-processes the data and labels the sample index. Then, the pre-processed data is input into the encoding-decoding structure of the spatial sub-network for spatial feature extraction, the output pixels of the spatial sub-network are recorded according to the index value, and then the pixels selected by the training index are input into the progressive feature learning of the spectral sub-network for spectral feature extraction. Finally, the extracted spatial and spectral features are classified by adaptive weighting. This method has the problems of low accuracy, poor generalization performance, insufficient feature extraction and poor feature fusion effect in the hyperspectral image classification. SUMMARY

[0004] The main purpose of the embodiment of the present application is to provide a hyperspectral image classification method, which comprises the following steps:

[0005] Step 1, preparing and pre-processing the data set: preparing four kinds of hyperspectral data sets, and pre-processing each data set to divide the training set and the test set;

[0006] Step 2, constructing a network model: constructing a network model comprising a spectral calibration block, a spectral feature extraction module, a spatial feature extraction module, a dynamic feature fusion module, and an M-type classifier; wherein the spectral feature extraction module comprises a lightweight transformer encoder module and a multi-scale spectral feature extraction module for extracting global spectral features and multi-scale local spectral features; the spatial feature extraction module comprises a convolutional nonlinear module, a similarity self-attention module, a multi-scale spatial feature extraction module, and a convolutional layer for extracting global spatial features and multi-scale local spatial features; the dynamic feature fusion module adaptively fuses spectral and spatial features based on a cross-attention module;

[0007] Step 3, selecting a loss function and evaluation index: using a cross-entropy loss function as the loss function, and selecting overall classification accuracy, average classification accuracy, kappa coefficient, and confusion matrix as the evaluation index;

[0008] Step 4, training the network model: setting the maximum number of training rounds, selecting the Adam optimizer and using the warm-up strategy to gradually increase the learning rate, training the network model until the value of the loss function is less than the set threshold.

[0009] Further, in step 1, the four hyperspectral data sets include the Indian Pine Tree data set, the Salinas Valley data set, the University of Pavia data set, and the Houston 2013 data set.

[0010] The step of preprocessing each data set includes standardizing the hyperspectral image data to eliminate differences between different spectral bands, and using a sliding window method to divide the standardized hyperspectral image into fixed-size cubic blocks.

[0011] Further, in step 2, the lightweight transformer encoder module comprises layer normalization, lightweight multi-head self-attention, a feedforward network, and residual connection.

[0012] The steps of the lightweight multi-head self-attention include: linearly mapping the input into Q, K, and V matrices, using a window size of 2 and a stride of 2 maximum pooling operation on matrices K and V before attention operation, dimension transformation on matrices Q, K, and V, concatenating the transformed K and V, and then Q, K, and V passing through a linear layer and entering the multi-head self-attention;

[0013] The steps of the feedforward network include: channel expansion through 1x1x1 convolution and R-type activation function, using channel segmentation technology to divide the features into two groups W1 and W2, wherein one group W1 uses 1x1x7 convolution and R-type activation function to encode the local context, then the processed W1 and W2 are spliced and input into 1x1x1 convolution for channel adjustment.

[0014] Further, in step 2, the multi-scale spectral feature extraction module includes a grouping convolution, a convolution block one, a convolution block two, a convolution block three and a convolution block four.

[0015] Further, in step 2, the convolution nonlinear module includes reshaping, a two-dimensional convolution layer, a batch normalization layer and an M-type activation function.

[0016] The similarity self-attention module includes a cosine similarity, a Gaussian Euclidean similarity, an S-type function and a residual connection.

[0017] The multi-scale spatial feature extraction module includes six convolution blocks, a global average pooling, a full connection layer, a batch normalization layer and an R-type activation function.

[0018] Further, in step 2, the dynamic feature fusion module includes a two-dimensional adaptive average pooling layer, a three-dimensional adaptive average pooling layer, a flattening layer and a cross-attention module.

[0019] Further, in step 2, the spectral calibration block uses a three-dimensional convolution layer to reduce the spectral dimension.

[0020] Further, the M-type classifier adopts a multi-layer perceptron to process the complex nonlinear relationship in the spectral information, and closely connects the features and the classification output.

[0021] Further, four activation functions are used, which are R-type activation function, L-type function, S-type function and M-type function, to increase the nonlinear factor of the network and fully extract the image features.

[0022] Further, the hyperspectral image classification method further includes: step 5, saving the model.

[0023] Compared with the prior art, the present application has the following beneficial effects:

[0024] 1. The present application provides a dual-branch multi-scale feature extraction classification method, which first designs a spectral feature extraction module including a lightweight transformer encoder module and a multi-scale spectral feature extraction module, then designs a spatial feature extraction module including a convolution nonlinear module, a similarity self-attention module, a multi-scale spatial feature extraction module and a convolution layer, and further designs a dynamic feature fusion module to fully fuse the extracted spectral and spatial features. The method can capture more comprehensive spatial and spectral features, thereby effectively improving the classification accuracy and generalization, and has good flexibility, scalability and computational efficiency.

[0025] 2、The dynamic feature fusion module disclosed by the present application can adaptively perform dynamic cross-feature interaction based on the cross-attention interaction module, the cross-attention interaction module can realize efficient interaction between spectral and spatial features based on a dynamic mechanism, and meanwhile, the generalization ability of the model to a complex scene is improved, the extracted spectral features and spatial features can be fully fused by adaptively adjusting the weight distribution, and thus the accuracy, robustness and generalization of classification are effectively improved.

[0026] 3、The similarity self-attention module is proposed in the spatial feature extraction module, global spatial features are extracted by exploring the relationship between pixels, a multi-scale spatial feature extraction module is also proposed, multi-scale local spatial features are extracted through different scales of receptive fields, and adaptive adjustment is performed by using spectral information, so that the feature representation is more accurate and effective, that is, the module can comprehensively capture spatial features, adapt to complex image patterns, and improve classification accuracy.

[0027] 4、The lightweight transformer encoder module is proposed in the spectral feature extraction module, global spectral features are extracted by constructing long-range dependency in the spectral dimension, and a multi-scale spectral feature extraction module is also proposed, multi-scale local spectral features are extracted by applying different scales of receptive fields on grouped convolution, that is, the module reduces the model calculation complexity while extracting rich spectral features, and improves the training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only show some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the drawings shown.

[0029] Figure 1 The step flow chart of the hyperspectral image classification method provided by the present application is provided.

[0030] Figure 2 The overall framework diagram of the network model provided by the present application is provided.

[0031] Figure 3 The framework diagram of the lightweight transformer encoder module provided by the present application is provided.

[0032] Figure 4 The framework diagram of the lightweight multi-head self-attention provided by the present application is provided.

[0033] Figure 5 The framework diagram of the feedforward network provided by the present application is provided.

[0034] Figure 6 A framework diagram of a multi-scale spectral feature extraction module provided by the present application is shown in FIG. 1.

[0035] Figure 7 A framework diagram of a convolutional nonlinear module provided by the present application is shown in FIG. 2.

[0036] Figure 8 A framework diagram of a similarity self-attention module provided by the present application is shown in FIG. 3.

[0037] Figure 9 A framework diagram of a multi-scale spatial feature extraction module provided by the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0038] The present application provides a hyperspectral image classification method, aiming to solve the problems of insufficient feature extraction, poor fusion effect and low classification accuracy caused by low spatial resolution, large data volume and intra-class spectral heterogeneity in existing hyperspectral image classification methods.

[0039] The hyperspectral image classification method proposed by the present application will be described below in specific embodiments:

[0040] Embodiment 1

[0041] A hyperspectral image classification method, comprising the following steps:

[0042] Step 1, preparing and preprocessing the data set: preparing four kinds of hyperspectral data sets, and preprocessing each data set to divide the training set and the test set;

[0043] Step 2, constructing a network model: constructing a network model containing a spectral calibration block, a spectral feature extraction module, a spatial feature extraction module, a dynamic feature fusion module and an M-type classifier; wherein the spectral feature extraction module includes a lightweight transformer encoder module and a multi-scale spectral feature extraction module, which are used to extract global spectral features and multi-scale local spectral features; the spatial feature extraction module includes a convolutional nonlinear module, a similarity self-attention module, a multi-scale spatial feature extraction module and a convolutional layer, which are used to extract global spatial features and multi-scale local spatial features; the dynamic feature fusion module adaptively fuses spectral and spatial features based on a cross-attention module;

[0044] Specifically, the spectral calibration block uses a three-dimensional convolutional layer with a size of (1, 1, C) to reduce the dimension of the spectral dimension while keeping the spatial dimension unchanged;

[0045] The spectral feature extraction module is divided into a lightweight transformer encoder module and a multi-scale spectral feature extraction module, which are respectively used to extract global spectral features and multi-scale local spectral features; the lightweight transformer encoder module is composed of layer normalization, a lightweight multi-head self-attention, a feedforward network and a residual connection, which extracts global spectral features by constructing long-range dependencies in the spectral dimension; the multi-scale spectral feature extraction module is composed of a grouped convolution and four convolution blocks with different kernel sizes, which extracts multi-scale local spectral features by applying different receptive fields to different channel dimensions;

[0046] The spatial feature extraction module is composed of a convolution nonlinear module, a similarity self-attention module, a multi-scale spatial feature extraction module and a convolution layer; the convolution nonlinear module is composed of reshaping, a two-dimensional convolution layer, a batch normalization layer and an M-type activation function, which is used to obtain sufficient input information; the similarity self-attention module is composed of cosine similarity, Gaussian Euclidean similarity, an S-type function and a residual connection, which extracts global spatial features by exploring the relationship between the center pixel and the adjacent pixels; the multi-scale spatial feature extraction module is composed of six convolution blocks, a global average pooling layer, a fully connected layer, a batch normalization layer and an R-type activation function, which extracts multi-scale local spatial features through different scales of receptive fields, and simultaneously uses spectral information for adaptive adjustment to make the feature representation more accurate and effective; the convolution layer integrates global spatial features and multi-scale local spatial features;

[0047] The dynamic feature fusion module includes a two-dimensional adaptive average pooling layer, a three-dimensional adaptive average pooling layer, a flattening layer and a cross-attention module, which can fully fuse the extracted spectral features and spatial features through a dynamic mechanism to adaptively adjust the weight distribution, and is helpful to achieve better classification accuracy.

[0048] Step 3, selecting a loss function and evaluation indicators: using a cross-entropy loss function as the loss function, and selecting overall classification accuracy, average classification accuracy, kappa coefficient and confusion matrix as the evaluation indicators;

[0049] Step 4, training the network model: setting the maximum number of training rounds, selecting the Adam optimizer and using the warm-up strategy to gradually increase the learning rate, training the network model until the value of the loss function is less than the set threshold;

[0050] Step 5, saving the model: solidifying the model parameters according to the best performance of the evaluation indicators to determine the final hyperspectral image classification model.

[0051] Further, in step 1, the four kinds of hyperspectral data sets include Indian Pine Tree data set, Salinas Valley data set, University of Pavia data set and Houston 2013 data set;

[0052] The preprocessing steps for each dataset include standardization of hyperspectral image data to eliminate differences between different spectral bands, and division of the standardized hyperspectral image into fixed-size cubic blocks using a sliding window method.

[0053] Further, in step 2, the lightweight transformer encoder module includes layer normalization, lightweight multi-head self-attention, a feedforward network, and a residual connection.

[0054] The steps of the lightweight multi-head self-attention include linearly mapping the input into three matrices Q, K, and V, using a window size of 2 and a stride of 2 for the maximum pooling operation on matrices K and V before the attention operation, performing dimension transformation on matrices Q, K, and V, concatenating the transformed K and V, and then passing Q, K, and V through a linear layer and into multi-head self-attention.

[0055] The steps of the feedforward network include channel expansion through 1x1x1 convolution and R-type activation function, splitting the features into two groups W1 and W2 using channel splitting technology, encoding the local context using 1x1x7 convolution and R-type activation function for one group W1, concatenating the processed W1 and W2, and inputting 1x1x1 convolution for channel adjustment.

[0056] Further, in step 2, the multi-scale spectral feature extraction module includes grouped convolution, convolution block one, convolution block two, convolution block three, and convolution block four.

[0057] Further, in step 2, the convolution nonlinear module includes reshaping, a two-dimensional convolution layer, a batch normalization layer, and an M-type activation function.

[0058] The similarity self-attention module includes cosine similarity, Gaussian Euclidean similarity, S-type function, and residual connection.

[0059] The multi-scale spatial feature extraction module includes six convolution blocks, global average pooling, a fully connected layer, a batch normalization layer, and an R-type activation function.

[0060] Further, in step 2, the dynamic feature fusion module includes a two-dimensional adaptive average pooling layer, a three-dimensional adaptive average pooling layer, a flattening layer, and a cross-attention module.

[0061] Further, in step 2, the spectral calibration block uses a three-dimensional convolution layer to reduce the spectral dimension.

[0062] Further, the M-type classifier uses a multi-layer perceptron to handle complex nonlinear relationships in spectral information, closely linking features and classification outputs.

[0063] Further, four activation functions are used, respectively R-type activation function, L-type function, S-type function and M-type function, to increase the nonlinear factor of the network and fully extract image features.

[0064] Embodiment 2

[0065] A hyperspectral image classification method, specifically comprising the following steps:

[0066] Step 1, preparing a data set:

[0067] Four kinds of hyperspectral public data sets are prepared, respectively Indian Pine Tree data set, Salinas Valley data set, University of Pavia data set and Houston 2013 data set; the Indian Pine Tree data set is obtained by using an infrared imaging spectrometer sensor in an Indian Pine agricultural test field, the wavelength range of the data set is 0.4-2.5um, the spatial resolution is 20m, 220 bands are contained, 20 bands are removed, 200 bands are retained, the spatial dimension is 145x145, the bands are divided into 16 land cover categories, and a total of 10249 labeled pixels are contained; the Salinas Valley data set is obtained by using an infrared imaging spectrometer sensor above the Salinas Valley in California, USA, the wavelength range of the data set is 0.4-2.5um, the spatial resolution is 3.7m, 224 bands are contained, 20 bands are removed, 204 bands are retained, the size is 512x217, and the bands are divided into 16 land cover categories; the University of Pavia data set is obtained by using a reflective optical system imaging spectrometer sensor above the University of Pavia in Italy, the wavelength range of the data set is 0.43-0.86um, the spatial resolution is 1.3m, 115 bands are contained, 12 bands are removed, 103 bands are retained, the size is 610x340, and the bands are divided into 9 land cover categories, and a total of 42776 labeled pixels are contained; the Houston 2013 data set is obtained by using an airborne laser mapping center mapping by a hyperspectral image analysis group and the University of Houston in the United States, the wavelength range of the data set is 0.38-1.05um, the spatial resolution is 2.5m, 144 bands are contained, 48 bands are removed, 96 bands are retained, the size is 349x1905, and the bands are divided into 15 land cover categories, and a total of 14278 labeled pixels are contained; wherein 3% of the Indian Pine Tree data set is used as a training set, 0.5% of the Salinas data set and the University of Pavia data set is used as a training set, and 5% of the sample data of the Houston 2013 data set is used as a training set.

[0068] Data preprocessing:

[0069] The standardization aims to eliminate the differences between different spectral bands and keep the data distribution of each band consistent, thereby improving the classification performance of the model. The standardization operation needs to read the original hyperspectral image data first, calculate the normalized value of each band pixel value, and then use the sliding window method to divide the standardized hyperspectral image into fixed-size cubic blocks. The label of each cubic block is determined by the label of the center pixel. Taking the Ponderosa Pine dataset with a size of (145, 145, 200) as an example, the size of the input image block is fixed at (9, 9, 200), and the standardization expression is as follows:

[0070]

[0071] where x i,j represents the original value of the i-th pixel in the j-th band, x' i,j represents the standardized value of the i-th pixel in the j-th band, and N is the total number of pixels in the band.

[0072] Step 2, construct the network model: the network model is shown in Figure 2 , which specifically includes a spectral calibration block, a spectral feature extraction module, a spatial feature extraction module, a dynamic feature fusion module, and an M-type classifier.

[0073] The spectral calibration block uses a three-dimensional convolution layer with a kernel size of 1x1x7, a stride of 1x1x2, no padding, and a kernel number of 1, which is used to reduce the spectral dimension.

[0074] The spectral feature extraction module consists of a lightweight transformer encoder module and a multi-scale spectral feature extraction module, which are used to fully extract global spectral features and multi-scale local spectral features, respectively.

[0075] The lightweight transformer encoder module is specifically composed of layer normalization, lightweight multi-head self-attention, feedforward network, and residual connection, as shown in Figure 3 . This module extracts global spectral features by constructing long-range dependencies in the spectral dimension. The input applies layer normalization before the lightweight multi-head self-attention and feedforward network, and applies residual connection after them. Layer normalization is used to reduce the difference between features, and residual connection is used to alleviate the gradient vanishing problem.

[0076] The lightweight multi-head self-attention is specifically composed of Figure 4As shown, the input is first linearly mapped into three matrices Q, K and V. In order to reduce the channel dimension and computational complexity, a max-pooling operation with a window size of 2 and a stride of 2 is used on the matrices K and V before the attention operation. Then, the dimensions of the matrices Q, K and V are transformed, the transformed K and V are spliced, and then Q, K and V pass through a linear layer and enter the multi-head self-attention.

[0077] The specific composition of the feedforward network is as shown in the following table: Figure 5 As shown, the input first passes through a 1x1x1 convolution and an R-type activation function for channel expansion. Then, the channel segmentation technique is used to divide the features into two groups W1 and W2. One group W1 uses a 1x1x7 convolution and an R-type activation function to encode the local context. Then, the processed W1 and W2 are spliced and input into a 1x1x1 convolution for channel adjustment.

[0078] The specific composition of the multi-scale spectral feature extraction module is as shown in the following table: Figure 6 As shown, it is composed of grouped convolution, convolution block one, convolution block two, convolution block three and convolution block four. This module extracts multi-scale local spectral features through different receptive fields of grouped convolution. Each convolution block is composed of a three-dimensional convolution layer, a batch normalization layer and an L-type activation function. The input size of this module is 9x9x48, and the channel dimension is 32. First, along the channel dimension, there are four groups, each group has an image size of 9x9x48 and a channel dimension of 8. Each group passes through different convolution blocks and obtains an output with the same size. The first group is input into a convolution with a size of 1x1x1, a step of 1x1x1 and a padding of 0x0x0. The second group is input into a convolution with a size of 1x1x3, a step of 1x1x1 and a padding of 0x0x1. The third group is input into a convolution with a size of 1x1x7, a step of 1x1x1 and a padding of 0x0x3. The fourth group is input into a convolution with a size of 1x1x11, a step of 1x1x1 and a padding of 0x0x5. The output images of each group with a size of 9x9x48 and a channel dimension of 8 are spliced to obtain the final output with the same size as the input.

[0079] The spatial feature extraction module includes a convolution nonlinear module, a similarity self-attention module, a multi-scale spatial feature extraction module and a convolution layer. The size of the convolution layer is 1x1, which is used to adjust the feature channel and increase the information interaction. The spatial feature extraction module is connected through a residual connection between the output of the convolution nonlinear module and the output of the convolution layer to prevent overfitting.

[0080] The specific composition of the convolution nonlinear module is as shown in the following table: Figure 7As shown, it is composed of reshaping, two-dimensional convolution layer, batch normalization layer and M-type activation function, which is used to fully extract the information in the input data, wherein the reshaping is used to convert the 3D image block into a 2D image block, the convolution kernel size of the two-dimensional convolution layer is 3x3, the padding is 1x1, the batch normalization layer is used to standardize the results, and the M-type activation function is used to enhance the nonlinear expression ability;

[0081] The similarity self-attention module is specifically composed of Figure 8 As shown, it is composed of cosine similarity, Gaussian Euclidean similarity, S-type function and residual connection, which is used to explore the relationship between the center pixel and the adjacent pixel to extract global spatial features; wherein the size of the input X is 9x9x48, the center pixel X i is 1x1x48, and the adjacent pixel is X i,t =[X i,1 ,X i,2 ,X i,3 ,...,X i,9 ], the expressions of the cosine similarity and the Gaussian Euclidean similarity between the center pixel and the adjacent pixel are as follows:

[0082]

[0083] Wherein, the cosine similarity is represented as C i,t , the Gaussian Euclidean similarity is represented as G i,t , and σ represents the decay rate.

[0084] Next, the S-type function is used to normalize the similarity, and the cosine similarity self-attention map and the Gaussian Euclidean similarity self-attention map are obtained from the similarity matrix in turn. The two similarity self-attention maps are added through adaptive weight to obtain the fusion similarity self-attention map. After the fusion similarity self-attention map and the input data X are multiplied element by element, the residual connection is performed. The expression of the fusion similarity self-attention map and the final output is as follows:

[0085] Weighted=λGaEd+(1-λ)Cos;

[0086]

[0087] Wherein, Cos represents the cosine similarity self-attention map, GaEd represents the Gaussian Euclidean similarity self-attention map, Weighted is the fusion similarity self-attention map, Y is the output, λ is the weight parameter, and its value range is from 0 to 1, and the initial value is 0.5.

[0088] The specific composition of the multi-scale spatial feature extraction module is as follows Figure 9As shown, the module is composed of six convolutional blocks, global average pooling, a fully connected layer, a batch normalization layer, and an R-type activation function, which is used to capture multi-scale local spatial features adaptively guided by spectral information. Each convolutional block is composed of a two-dimensional convolutional layer, a batch normalization layer, and an L-type activation function. The input data X first passes through the three parallel branch convolutional blocks to obtain feature maps X1, X2, and X3 in different receptive fields, respectively. The convolutional kernel size of the fifth convolutional block is 3x3, the convolutional kernel size of the sixth convolutional block is 5x5, and the convolutional kernel size of the seventh convolutional block is 7x7. Then, the feature maps obtained by adding X2 and X3 are sequentially subjected to global average pooling, a fully connected layer, a batch normalization layer, an R-type activation function, and a fully connected layer to obtain a new feature map. The new feature map is subjected to an R-type activation function to obtain a channel attention map a and wherein and the ratio of a and represents the importance of each element channel information in the ratio of feature maps X2 and X3, and the channel attention map a and the feature map X4 are expressed as follows: The channel attention map a and the feature map X4 are expressed as follows:

[0089]

[0090] X4 = aX2 + (1-a)X3;

[0091] wherein a represents a channel attention map, σ represents an activation function, W BN represents a batch normalization layer, W FC represents a fully connected layer, W GAP represents a global average pooling layer, represents an element addition operation, and X4 represents an output feature map.

[0092] In order to refine the target position information, the feature maps X1 and X4 need to be added after passing through the eighth convolutional block with a convolutional kernel size of 1x1 to obtain a feature map X5. In order to fully extract multi-scale local spatial features adaptively guided by spectral information, the feature maps X1 and X5 need to be added after passing through the eighth convolutional block to obtain a feature map Y.

[0093] The dynamic feature fusion module is composed of a two-dimensional adaptive average pooling layer, a three-dimensional adaptive average pooling layer, a flattening layer and a cross-attention module, and the module adaptively fuses spectral and spatial features based on the cross-attention module; wherein the spectral branch sequentially passes through the three-dimensional adaptive average pooling layer and the flattening layer, and the spatial branch sequentially passes through the two-dimensional adaptive average pooling layer and the flattening layer, so that the dimensions are consistent for subsequent processing, then the spectral features are mapped into a query matrix Q1, a value matrix V1 and a key matrix K1, the spatial features are mapped into a query matrix Q2, a value matrix V2 and a key matrix K2, and the key matrix K1 and the key matrix K2 are spliced to obtain a matrix which is subjected to a full connection layer and an activation function, so as to obtain a key matrix dynamically adjusted by a weight coefficient; in the spectral branch, the query matrix Q1 containing spectral information is multiplied by the dynamically adjusted key matrix containing spectral and spatial information to obtain a cross-feature attention score, the score is subjected to an S-shaped function to obtain a score weight, and then the score weight is multiplied by the value matrix V1 containing spectral features, and further dynamic cross-feature interaction fusion is performed; in the spatial branch, the specific process of interaction fusion with the spectral features is similar to the above steps, and the specific process of the whole process is as follows:

[0094]

[0095] M=M1+M2;

[0096] Wherein, || represents a splicing operation, K spa∥spe represents a key matrix fusing spectral and spatial information, sigma represents an activation function, W FC represents a full connection layer, X spe represents an input of the spectral branch, X spa represents an input of the spatial branch, M1 represents spectral features embedded with spatial information, M2 represents spatial features embedded with spectral information, and M represents fused features of spectral and spatial information.

[0097] The M-type classifier adopts a multilayer perceptron, which can effectively process complex nonlinear relationships in spectral information, has a relatively simple structure and is easy to implement.

[0098] In order to ensure the robustness of the network, more nonlinear factors are introduced to fully extract image features, and four kinds of activation functions are used in the present application, which are R-type activation function, L-type function, S-type function and M-type function, and are defined as shown below:

[0099]

[0100] f(x) Mish =x*tanh(ln(1+e x ));

[0101] Step 3, select the appropriate loss function and evaluation index: in the process of training, the loss function selected is the cross-entropy loss function commonly used in classification tasks, the cross-entropy loss function calculates the loss value and adjusts the model parameters by using the gradient back propagation algorithm, the expression of cross-entropy loss function is as follows:

[0102]

[0103] Where, Loss represents the cross-entropy loss value, N represents the total number of samples, C represents the total number of classes, y i,j represents the jth element of the true label of sample i, represents the predicted probability of sample i belonging to class j.

[0104] The evaluation index selects the overall classification accuracy, average classification accuracy, kappa coefficient and confusion matrix to evaluate the classification performance of the model; the overall classification accuracy reflects the overall classification performance of the model, the average accuracy reflects the classification performance of each class of the model, the kappa coefficient reflects the accuracy of the model classification, and the confusion matrix directly reflects the classification results of the model classes; the expressions of classification accuracy, average classification accuracy, kappa coefficient and confusion matrix are as follows:

[0105]

[0106] Where, N represents the total number of samples, L represents the total number of classes, C ii represents the number of correctly classified samples, C ij represents the number of samples classified as class j, C(i,j) represents the number of samples classified as class i, N i represents the total number of class i.

[0107] Step 4, train the network model: in the training of hyperspectral classification network model, the maximum number of training rounds of the training model is set to 200, the back propagation algorithm selects Adam optimizer and uses preheating strategy to gradually increase the learning rate, which ensures the stable convergence of the model, the original learning rate is set to 0.001, and the threshold of loss function is set to 0.0005, when the function value of loss function is less than the threshold, it can be considered that the whole network training is completed, the expression of preheating strategy is as follows:

[0108]

[0109] Where, η represents the maximum learning rate that can be reached, t represents the current step, T represents the total number of training steps, ω represents the number of preheating steps, and the learning rate gradually increases from zero to the learning rate η and then decreases to zero.

[0110] Step 5, save the model: according to the model parameters with the best evaluation index performance, the final hyperspectral image classification model is determined.

[0111] The above description is only preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of hyperspectral image classification, characterized in that, The method comprises the following steps: Step 1, preparing and preprocessing the data set: four hyperspectral data sets are prepared, and each data set is preprocessed to divide the training set and the test set; Step 2, constructing a network model: a network model comprising a spectral calibration block, a spectral feature extraction module, a spatial feature extraction module, a dynamic feature fusion module and an M-type classifier is constructed; wherein the spectral feature extraction module comprises a lightweight transformer encoder module and a multi-scale spectral feature extraction module for extracting global spectral features and multi-scale local spectral features; the spatial feature extraction module comprises a convolution nonlinear module, a similarity self-attention module, a multi-scale spatial feature extraction module and a convolution layer for extracting global spatial features and multi-scale local spatial features; the dynamic feature fusion module adaptively fuses spectral and spatial features based on a cross-attention module; Step 3, selecting a loss function and evaluation index: using a cross-entropy loss function as the loss function, selecting the overall classification accuracy, the average classification accuracy, the kappa coefficient and the confusion matrix as the evaluation index; Step 4, training the network model: setting the maximum number of training rounds, selecting the Adam optimizer and using the preheating strategy to gradually increase the learning rate, training the network model until the value of the loss function is less than the set threshold; The convolution nonlinear module comprises reshaping, a two-dimensional convolution layer, a batch normalization layer and an M-type activation function; The similarity self-attention module comprises a cosine similarity, a Gaussian Euclidean similarity, an S-type function and a residual connection; The multi-scale spatial feature extraction module comprises six convolution blocks, a global average pooling, a fully connected layer, a batch normalization layer and an R-type activation function.

2. The method of claim 1, wherein, In step 1, the four hyperspectral data sets include the Indian Pine Tree data set, the Salinas Valley data set, the University of Pavia data set and the Houston 2013 data set; The step of preprocessing each data set comprises standardizing the hyperspectral image data to eliminate differences between different spectral bands, and using a sliding window method to divide the standardized hyperspectral image into fixed-size cubic blocks.

3. The method of claim 1, wherein, In step 2, the lightweight transformer encoder module comprises layer normalization, a lightweight multi-head self-attention, a feedforward network and a residual connection; The lightweight multi-head self-attention step includes: linearly mapping the input to... Three matrices, processed before the attention operation. Using max pooling with a window size of 2 and a stride of 2, perform matrix... Perform dimensional transformation, and transform the result Then splice them together, and then After passing through the linear layer, it enters the multi-head self-attention; The step of the feedforward network includes: passing through The activation function performs channel expansion, uses a channel splitting technique to divide the features into Two groups, one of which Uses a 1x1x7 convolution and An activation function of the ReLU type to encode the local context, then concatenates the processed And inputs a 1x1x1 convolution to adjust the channels.

4. The method of claim 1, wherein, In step 2, the multi-scale spectral feature extraction module comprises a grouping convolution, a convolution block one, a convolution block two, a convolution block three and a convolution block four.

5. The method of claim 1, wherein, In step 2, the dynamic feature fusion module comprises a two-dimensional adaptive average pooling layer, a three-dimensional adaptive average pooling layer, a flattening layer and a cross-attention module.

6. The method of claim 1, wherein, In step 2, the spectral calibration block uses a three-dimensional convolution layer to reduce the spectral dimension.

7. The method of claim 1, wherein, In step 2, the M-type classifier adopts a multi-layer perceptron to handle the complex nonlinear relationship in the spectral information and connect the features with the classification output.

8. The method of claim 1, wherein, In step 3, four activation functions are selected, respectively , to increase the non-linear factor of the network and fully extract the image features. 9.The hyperspectral image classification method of claim 1, wherein, Further comprising: Step 5, saving the model: solidifying and saving the model parameters with the best evaluation index performance.

Images