End-to-end hyperspectral image classification method, system, device and terminal

Abstract

The present invention discloses an end-to-end hyperspectral image classification method, system, device and terminal. In the spectral feature extraction branch in the network model, the SLIC function is combined with a graph convolutional neural network with a spectral attention mechanism, which makes great use of the spatial context information of the classified pixels. The spectral feature extraction branch adopts a lightweight high-discrimination spectral feature selection module to effectively select and extract features of hyperspectral image data based on the spectral dimension. The spatial feature extraction branch uses a two-dimensional stacking module for dimensional conversion, and most of the subsequent operations are based on two-dimensional convolution blocks. After obtaining the local and global spatial information of the hyperspectral spectrum, a regularized Transformer network is used to extract and fuse features from a global direction, and the feature information extracted by the two multi-head attention mechanism layers is fused and the cross entropy loss function is used to obtain the loss. The gradient is back-transmitted to update the weights to solve the problem of feature propagation capability degradation.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to an end-to-end hyperspectral image classification method, system, device and terminal. Background Art

[0002] As three-dimensional data, the ground object categories between adjacent pixels of hyperspectral images have certain spatial correlation and the correlation information between spectral bands has strong dependence. The hyperspectral image classification technology has higher classification accuracy for these image data, but it requires a lot of storage space, resulting in low training efficiency of the model when processing classification tasks. Therefore, it is of great significance to study a hyperspectral image classification model with high computational efficiency and able to make full use of spatial and spectral information.

[0003] The Chinese patent publication number is CN117218537A, and its name is "Hyperspectral image classification method based on Transformer and non-local neural network dual-branch architecture". This method constructs two branches, one of which is the spectral branch, which uses the Transformer module structure to extract spectral feature information; the other branch is the spatial branch, which uses the structure of the spatial sub-network block combined with the spatial attention mechanism to extract spatial information; finally, the features obtained from the two branches are fused and classified, which has high computational complexity and low efficiency.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] First, the information extraction and information combination of existing hyperspectral image classification technologies in the spatial and spectral domains are not sufficient for high-accuracy modeling and classification, which will cause certain adverse effects on the classification accuracy of hyperspectral images and the generalization ability of classification methods;

[0006] Second, the existing hyperspectral image classification technology model has high computational complexity and many parameters. Since hyperspectral images have many three-dimensional data bands, they require a lot of storage space, which results in a long training time for the model when processing classification tasks, and the classification efficiency is lower than that of two-dimensional image classification.

[0007] Third, the single use of Transformer network is too scattered in spectral feature extraction, making it difficult to effectively utilize the rich spectral information in hyperspectral images, further resulting in the problem of low utilization of spectral information and inefficient network model acquisition of global information.

[0008] In summary, how to effectively solve the above problems is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0009] The main purpose of the embodiments of the present invention is to propose an end-to-end hyperspectral image classification method, system, device and terminal, aiming to design a hyperspectral image classification method that improves the hyperspectral image classification performance.

[0010] The technical solution of the present invention to solve the above technical problem is to provide an end-to-end hyperspectral image classification method, comprising the following steps:

[0011] Acquire a hyperspectral image dataset and preprocess the hyperspectral image dataset;

[0012] Constructing a network model including a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, a feature processing module, and finally classification through a multi-layer perception module;

[0013] Input the preprocessed hyperspectral image data set into the network model for training; obtain the training weight value;

[0014] Select a loss function, set a training loss threshold, and continuously iterate and optimize the network model until the number of training times reaches the set threshold or the value of the loss function reaches the set threshold range; use hyperspectral image classification effect evaluation indicators to measure the classification accuracy and performance of the network model;

[0015] The network model is trained and fine-tuned using a hyperspectral image dataset to optimize the model parameters.

[0016] Furthermore, in the step of obtaining a hyperspectral image dataset and preprocessing the hyperspectral image dataset,

[0017] The hyperspectral image datasets include the Indian Pines dataset, the Salinas dataset, and the KSC dataset.

[0018] Furthermore, the construction includes a network model of a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally in the step of classification through a multi-layer perception module,

[0019] The image division module divides the original hyperspectral image data into multiple super-pixel images through the SLIC function;

[0020] The graph convolutional network module with spectral attention mechanism processes the data into 1×1×C1 through a maximum pooling layer, which is used for local information interaction and global information interaction respectively, and combines the graph convolutional neural network to automatically learn important spectral bands, thereby enhancing the network model's attention to key spectral information.

[0021] The lightweight and highly discriminative spectral feature selection module includes a two-dimensional convolution layer, a batch normalization layer, an activation function layer, a maximum pooling layer, a fully connected layer, a Dropout layer, and a Flatten layer, which selects classification features from the output of the graph convolutional network to reduce redundant information.

[0022] Furthermore, the construction includes a network model of a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally in the step of classification through a multi-layer perception module,

[0023] Firstly, the covariance matrix eigendecomposition of the original hyperspectral image is performed by principal component analysis;

[0024] The two-dimensional stacking module includes multiple batch normalization layers, activation function layers and two-dimensional convolution layers for dimensional conversion;

[0025] The spatial attention module includes a maximum pooling layer, an average pooling layer, a two-dimensional convolution layer, a batch normalization layer and an activation function layer, which is used to transform the spatial information in the hyperspectral image data into another space and retain the key information and output it in a weighted manner;

[0026] The Transformer module includes layer normalization, discriminant regularization, and multi-head attention mechanism layers;

[0027] The feature processing module includes a batch normalization layer, an activation function layer, a one-dimensional convolution layer, a Dropout layer and an average pooling layer, and further processes the spatial features to obtain one-dimensional spatial features.

[0028] Furthermore, the construction includes a network model of a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally in the step of classification through a multi-layer perception module,

[0029] The multi-layer perception module includes an input layer, an output layer, two fully connected layers, two activation function layers and a hidden layer, and fuses and classifies the spectral features extracted by the spectral feature extraction branch and the spatial features extracted by the spatial feature extraction branch.

[0030] Furthermore, the loss function is selected, the training loss threshold is set, and the network model is continuously iterated and optimized until the number of training times reaches the set threshold or the value of the loss function reaches the set threshold range; in the step of using the hyperspectral image classification effect evaluation index to measure the classification accuracy and performance of the network model, the loss function selects the cross entropy loss function, which is used to truly reflect the difference between the predicted value and the true value after the end-to-end model, and to provide feedback on the quality of the classification network model.

[0031] In order to solve the above technical problems, the present invention also proposes an end-to-end hyperspectral image classification system, comprising:

[0032] A hyperspectral image data acquisition module is used to acquire the hyperspectral image data to be processed;

[0033] The classification model construction module is used to construct an end-to-end hyperspectral image classification model based on the joint spatial-spectral attention mechanism and the improved Transformer network;

[0034] Classification model training module, which is used to input hyperspectral image data into the classification model construction module for training, compare the prediction results with the actual training results to obtain the error and fine-tune the model parameters;

[0035] The classification result generation module is used to evaluate the classification results through classification evaluation indicators and verify the feasibility and superiority of the classification method.

[0036] In order to solve the above technical problems, the present invention also proposes a computer device, including a hyperspectral image memory and a hyperspectral image classification processor, and a memory storing a computer program; when the computer program is executed by the processor, the processor executes the steps of the end-to-end hyperspectral image classification method as described above.

[0037] In order to solve the above technical problems, the present invention further proposes an information data processing terminal, which is used to implement the end-to-end hyperspectral image classification system as described above.

[0038] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0039] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0040] (1) The image segmentation module adopted in the present invention can make full use of the long-range and short-range information of the unlabeled sample data in the hyperspectral image dataset, and combines the SLIC function with the graph convolutional neural network with the spectral attention mechanism, which makes great use of the spatial context information of the classified pixels and determines the transformation criteria between pixels and superpixels. Compared with the current data preprocessing method, it is more convenient to realize the subsequent classification process.

[0041] (2) Before the spatial features of the extracted hyperspectral image are input into the spatial attention module, the present invention first uses a two-dimensional stacking module to perform dimensionality conversion. Compared with using a three-dimensional convolution block alone, this reduces the classification cost, is more convenient for model training and calculation, and greatly shortens the time spent on testing.

[0042] (3) The present invention adopts a lightweight and high-discrimination spectral feature selection module that can effectively select and extract features of hyperspectral image data based on the spectral dimension, capture important correlation information between spectral bands, and make the spectral feature extraction branch model reach a lightweight level. When the computing resources of the device are limited, the classification model of the hyperspectral image can still be successfully installed.

[0043] (4) The present invention adopts a regularized Transformer network to fully utilize the non-negative discriminant information items in the traditional machine learning algorithm to mine the spatial structure information in the original hyperspectral image data, so that the network has better discriminability, further optimizes the performance of spatial feature extraction, and can effectively improve the classification accuracy of the overall classification model.

[0044] (5) The present invention adopts an end-to-end hyperspectral image classification model. The network model adopts a dual-branch strategy to fuse the extracted spectral feature information and spatial feature information and jointly uses the cross-entropy loss function to obtain the loss. The gradient is back-propagated to update the weights to solve the problem of feature propagation capability degradation.

[0045] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0046] On the one hand, the overall technical solution proposed in the algorithm research of the present invention can reduce the complexity of the algorithm and the time spent on testing while ensuring good hyperspectral image classification effects. It can be applied to resource-constrained small and medium-sized farms or edge-end distributed computing scenarios for precision agriculture to solve real-time model training problems and real-time end-to-end hyperspectral image classification problems to a certain extent. The present invention is closely combined with market application needs, attaches importance to the application of hyperspectral images in practice, and studies efficient classification technology methods with certain market potential. It is estimated that it can improve the competitiveness of hyperspectral image classification in the application field of remote sensing technology.

[0047] On the other hand, the present invention has shown good generalization ability after comparison and verification with existing methods, and is widely used in many fields. It is estimated that the present invention can show its unique advantages in practical classification applications such as geological exploration and medical imaging, and further promote the iterative upgrade of hyperspectral image data processing methods. While accelerating the training speed, it greatly reduces the classification cost, and can better adapt to the development requirements of intelligent and rapid classification of hyperspectral images.

[0048] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0049] (1) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0050] The present invention adopts an end-to-end hyperspectral image classification model through a lightweight and high-discrimination spectral feature selection module, which can still enable the classification model of hyperspectral images to be successfully installed when the computing resources of the device are limited, filling the gap in this classification technology in the industry at home and abroad. At the same time, the dual-branch strategy is fully utilized to fuse the extracted spectral features with the spatial features extracted by the spatial feature extraction branch and classify them to improve the utilization rate of spatial-spectral information, thereby greatly improving the classification performance of hyperspectral images.

[0051] (2) Whether the technical solution of the present invention overcomes technical prejudice:

[0052] Currently, many researchers believe that preprocessing of hyperspectral images will inevitably cause a large amount of feature information loss, resulting in poor classification results. The technical solution provided by the present invention utilizes an image segmentation module that uses a clustering algorithm to automatically divide the original hyperspectral image into super-pixel blocks, determines the transformation criteria between pixels and super-pixels, and successfully retains the key feature information in the data, so that the classification performance of the hyperspectral image is as unaffected as possible by data preprocessing, and the generalization ability of the hyperspectral image classification model is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0054] Figure 1 This is a flowchart of the end-to-end hyperspectral image classification method of the present invention;

[0055] Figure 2 It is a structural block diagram of the end-to-end hyperspectral image classification system of the present invention;

[0056] Figure 3 The overall network structure diagram of the end-to-end hyperspectral image classification method of the present invention;

[0057] Figure 4 A schematic diagram of the composition of the image segmentation module of the present invention;

[0058] Figure 5 It is a schematic diagram of the composition of the graph convolutional network module adding the spectral attention mechanism described in the present invention;

[0059] Figure 6 Schematic diagram of the composition of the lightweight high-discrimination spectral feature selection module of the present invention;

[0060] Figure 7 This is a schematic diagram of the composition of the two-dimensional stacking module of the present invention;

[0061] Figure 8 Schematic diagram of the composition of the spatial attention module of the present invention;

[0062] Fig. 9 This is a schematic diagram of the composition of the feature processing module of the present invention;

[0063] Fig.10 This is a schematic diagram of the composition of the Transformer module of the present invention;

[0064] Fig.11 This is a schematic diagram of the specific composition of the multi-layer perception module of the present invention;

[0065] Fig.12 A schematic diagram of the specific composition of the spectral attention module of the present invention;

[0066] Fig.13 The figure is a structural block diagram of the computer device described in the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the accompanying drawings of the present invention specification to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0069] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "several" or "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0070] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0072] The present invention proposes an end-to-end hyperspectral image classification method, system, device and terminal, aiming to design a hyperspectral image classification method that improves the hyperspectral image classification performance.

[0073] The end-to-end hyperspectral image classification method proposed by the present invention will be described in a specific embodiment below:

[0074] Example 1

[0075] An end-to-end hyperspectral image classification method, such as Figure 1 As shown, the following steps are included:

[0076] S10: obtaining a hyperspectral image dataset and preprocessing the hyperspectral image dataset;

[0077] Hyperspectral image datasets include: Indian Pines dataset, Salinas dataset and KSC dataset. Indian Pines dataset is a commonly used remote sensing image dataset, which is used for ground object classification and target recognition tasks. Indian Pines dataset includes 16 ground object categories such as forests, corn, soybeans, wheat, etc. The research generally takes the remaining 200 bands after removing 20 water absorption bands as the research object, annotates the data with image space size of 145×145 and uses it for hyperspectral image classification; Salinas dataset is also a commonly used typical hyperspectral image dataset, which is used for ground object classification tasks and has the characteristics of high spatial resolution. Its spatial resolution is 3.7 meters, the size is 512×217, and the original data has 224 bands. Salinas dataset includes 16 ground object categories such as vegetables, bare soil and vineyards, which are used for hyperspectral image classification; KSC dataset was taken by sensors at Kennedy Space Center in Florida. The data contains a total of 224 bands. After removing the water vapor noise, there are 176 bands left. The spatial resolution is 18 meters. There are a total of 13 categories for hyperspectral image classification.

[0078] S20: construct a network model including a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discriminative spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally classification is performed through a multi-layer perception module;

[0079] Specifically, the overall network structure of the network model is as follows Figure 3 As shown in Figure 2. Among them, the image segmentation module is as follows: Figure 4 As described above, the SLIC function in the algorithm uses a clustering algorithm to automatically divide the original hyperspectral image into super-pixel blocks, determines the transformation criteria between pixels and super-pixels, and performs super-pixel segmentation. By performing super-pixel segmentation on the hyperspectral image data, the image is represented as an undirected graph, and the structure of the undirected graph G is represented as:

[0080] G=(V,E)

[0081] Among them, V represents the set of vertices in the undirected graph, and E represents the set of edges in the undirected graph.

[0082] like Figure 5As described above, when the graph convolutional network module with the spectral attention mechanism is added to extract spectral feature information, the graph convolutional network will not pay enough attention to the important spectral band information in the hyperspectral image data. Therefore, the present invention adds spectral attention to the graph convolutional network module to highlight the spectral information that is more valuable for hyperspectral image classification and suppress redundant information, thereby improving the classification accuracy of the overall model. The hierarchical recursive feature propagation rule of the graph convolutional network is:

[0083]

[0084] Y=A+E

[0085] Where Z (l+1) represents the features of the (l+1)th layer, A is the adjacency matrix given by the undirected graph G, Z (l) represents the features of the lth layer, σ represents the activation function, W represents the weight matrix, and E is the identity matrix.

[0086] Spectral attention mechanism such as Fig.12 As shown in the figure, the spatial information of the feature map is aggregated by average pooling and maximum pooling to generate two different feature maps with spatial context information description. The two feature maps are subjected to two-dimensional convolution to generate the spectral domain attention feature map, the attention feature maps are summed, and then the attention information of the two feature maps is aggregated by the activation function to obtain the spectral attention weight. The spectral attention weight T is formed by the attention mechanism as follows:

[0087] T=f(Conv2d(AvgPool(F))+Conv2d(MaxPool(F)))

[0088] Where F is the initial feature map of the input graph convolutional neural network, F(m,n) is the value at (m,n) on the feature map, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, Conv2d is a two-dimensional convolutional neural network, and f is the activation function.

[0089] The initial feature map F of the input graph convolutional neural network is multiplied by the weight T to obtain the output feature F′ of the graph convolutional network module with the spectral attention mechanism:

[0090]

[0091] The feature F′ is input into a lightweight and highly discriminative spectral feature selection module, such as Figure 6As shown in the figure, the lightweight high-discrimination spectral feature selection module can effectively select and extract features of hyperspectral image data based on the spectral dimension. When the computing resources of the device are limited, the hyperspectral image classification model can also be successfully installed. Among them, the first two-dimensional convolution layer uses a convolution kernel of size 1×1 for convolution, with a step size of 1×1 and no padding. A batch normalization layer is added after the convolution layer, and finally the R-type function is used as the activation function. The R-type function is defined as follows:

[0092]

[0093] Among them, x represents the input feature information.

[0094] After the first convolutional layer, the maximum pooling operation is performed on the obtained feature map. In the second convolutional layer, a convolution kernel of size 3×3 is used for convolution, the step size is 1×1, and the padding is 1×1. Similarly, a batch normalization layer is added after the second convolutional layer and the R-type function is used as the activation function. After that, the feature map is pooled. The feature map output by the second pooling layer is expanded into a one-dimensional vector as the input of the fully connected layer, and then input into the Dropout layer with a coefficient set to 0.5 to speed up the training process. After the last fully connected layer, the Flatten layer is used to obtain the one-dimensional spectral features.

[0095] In the spatial feature extraction branch, the principal component analysis method is first used to reduce the dimension of the hyperspectral image data. Taking the Indian Pines dataset as an example, the original hyperspectral image is first reduced in dimension to a size of W×H×C=145×145×80. The hyperspectral image data after dimension reduction is input into the two-dimensional stacking module for dimensional conversion, such as Figure 7 As shown in the figure, it consists of a batch normalization layer, an activation function layer and a two-dimensional convolution. All subsequent operations are mostly based on two-dimensional convolution blocks. Compared with using three-dimensional convolution blocks alone, the classification cost is reduced. In the convolution operation of the two-dimensional convolution layer, all convolution kernel sizes are 1×1, the step size is set to 1×1, and there is no padding. The activation functions involved all use R-type functions.

[0096] After the image data is transformed in dimension, it is input into the spatial attention module, such as Figure 8 As shown in the figure, the spatial attention module includes a maximum pooling layer, an average pooling layer, a two-dimensional convolution layer, a batch normalization layer, and an activation function layer. The spatial attention module is used to transform the spatial information in the hyperspectral image data into another space and retain the key information and weighted output, thereby enhancing the characteristics of important spatial regions while weakening irrelevant background regions, integrating spatial dimension information to solve the problem that ordinary convolution cannot effectively obtain global information. The specific formula of the spatial attention module is as follows:

[0097]

[0098] Where X represents the input data, Indicates the multiplication of corresponding elements, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, Conv2d is a two-dimensional convolutional neural network, FC means fully connected, Sigmoid means the mapping from 0 to 1, and Reshape means resizing to the original size of the hyperspectral image.

[0099] The activation function used in the spatial attention block is the S-type function which is less affected by noise data. The S-type function is defined as follows:

[0100]

[0101] Among them, x represents the input feature information;

[0102] like Fig.10 As shown in the figure, the regularized Transformer module includes layer normalization, multi-head attention mechanism layer and discriminant regularization. The attention mechanism adopted by the multi-head attention mechanism layer is as follows:

[0103] MultiHead(Q,K,V)=Concat(head1,…,head l )W 0

[0104]

[0105] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and W 0 is the concatenation fusion matrix, d k is the input dimension, l is the number of branches, and the value of i ranges from 1 to l.

[0106] Traditional Transformer pays more attention to global correlation, and has many deficiencies in the perception and discrimination of important local spatial feature information. When mining spatial structural information of hyperspectral image data, the differences in local structures between different types of objects in the hyperspectral image data set are often ignored. Therefore, the present invention adds the discriminant regularization used by the traditional machine learning algorithm to the Transformer module, so that the algorithm has better discriminability and further improves the performance of spatial feature extraction. The training sub-block matrix of the hyperspectral image output by the two multi-head attention mechanism layers is:

[0107]

[0108] The hyperspectral image training sub-block matrix is ​​introduced into the local discriminant model, and a class label matrix is ​​defined for the current training sub-block:

[0109] Di =[d1,d2,…,d n ]

[0110] Taking the Indian Pines dataset as an example, the number of training sample categories n is 16, and the class label matrix D i Normalize and redefine a weight class label matrix as:

[0111] F i =D i (D i T D i ) -1 / 2

[0112] In order to mine the discriminant information in the data, the overall scatter matrix A is introduced into the local discriminant model. id , the inter-class scatter matrix A representing the distance between data of different categories ip and the intra-class scatter matrix A representing the spatial distance between data of the same category iw , which are defined as follows:

[0113]

[0114]

[0115]

[0116] In the above formula, the centralized training sub-block matrix is ​​defined as follows:

[0117]

[0118] in, Indicates m i -dimensional vector of all 1s.

[0119] To use the spatial discriminant information of hyperspectral image data to achieve better data segmentation, it is necessary to maximize the inter-class scatter matrix A. ip Make the spatial distance between heterogeneous data farther and minimize the intra-class scatter matrix A as much as possible iw Or the overall scatter matrix A id To shorten the spatial distance between data of the same category, therefore, to further optimize the performance of extracting spatial features with higher discriminability, it is necessary to add regularization terms to achieve regularization and minimize the following objective function:

[0120]

[0121] Wherein, E is the unit matrix, φ is the regularization parameter, in the present invention, the regularization parameter φ is set to 8500, and Rg is the regularization operation.

[0122] The spatial features with higher discriminability extracted by regularization are input into the 7-feature processing module, such as Fig. 9 As shown, the feature processing module includes a batch normalization layer, an activation function layer, a one-dimensional convolution layer, a Dropout layer, and an average pooling layer, and the obtained spatial features are processed to obtain one-dimensional spatial features;

[0123] like Fig.11 As shown in the figure, the multi-layer perception module includes an input layer, a fully connected layer, an activation function layer, a hidden layer, and an output layer. The activation function adopts a G-type function, and the approximate calculation formula of the G-type activation function is: Among them, X represents the feature input.

[0124] The classification method of the present invention finally fuses the one-dimensional spectral features extracted by the spectral feature extraction branch and the one-dimensional spatial features extracted by the spatial feature extraction branch through the constructed multi-layer perception module, and the output categories of the classification results output by the multi-layer perception module are as follows:

[0125] Class = softmax(GELU(FC(X))

[0126] Among them, softmax represents the classification function, GELU is a nonlinear G-type activation function based on Gaussian error linear unit, and X represents the feature input.

[0127] S30: input the preprocessed hyperspectral image data set into the network model for training; obtain the training weight value;

[0128] Specifically, 2% of the sample data in the Indian Pines dataset is used as the training set, 1% of the sample data in the Salinas dataset is used as the training set, and 0.5% of the sample data in the KSC dataset is used as the training set. The number of training times is set to 150, and the number of images input to the network each time is 30. The three training sets are trained separately.

[0129] S40: Select a loss function, set a training loss threshold, and continuously iterate and optimize the network model until the number of training times reaches the set threshold or the value of the loss function reaches the set threshold range; use a hyperspectral image classification effect evaluation index to measure the classification accuracy and performance of the network model;

[0130] Specifically, a suitable loss function is selected and the evaluation index of this method is determined. The learning rate of the training process is set to 0.0008, and the Adam optimizer is selected as the network optimizer. Its advantages mainly lie in optimizing the network and improving the computational efficiency, making the parameters relatively stable, which can adapt to sparse gradients and alleviate the problem of gradient oscillation.

[0131] The choice of loss function affects the quality of the model. It can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model. By calculating the loss function of the classified result image and the label, the loss is obtained using the cross entropy loss function. The loss function value threshold is set to about 0.0006. When the function value threshold is less than 0.0006, it can be considered that the training of the entire network has been basically completed; among them, the cross entropy loss function is defined as follows:

[0132]

[0133] Among them, C represents the cost, x represents the sample, y represents the actual value, a represents the output value, and n represents the total number of samples.

[0134] The evaluation indicators selected are overall accuracy, average accuracy and consistency, which can effectively evaluate the quality of classification and measure the role of the classification network; among them, overall accuracy is an indicator to measure the overall classification accuracy of the classification model, and average accuracy is an indicator to show the classification accuracy of the classification model for a certain category; the calculation formulas of overall accuracy OA and average accuracy AA are as follows:

[0135]

[0136]

[0137] Among them, TP is the positive sample correctly classified by the model, FN is the positive sample incorrectly classified by the model, FP is the negative sample incorrectly classified by the model, and TN is the negative sample correctly classified by the model.

[0138] The consistency coefficient indicates the consistency between the predicted value and the true value, and is used to measure the effect of classification. The calculation formula of the consistency coefficient Kappa is as follows:

[0139]

[0140]

[0141]

[0142] Where C is the total number of categories, T i is the number of samples correctly classified in each category, a i is the number of true samples of each class, b i is the predicted number of samples of each category, and n is the total number of samples.

[0143] S50: Use hyperspectral image datasets to train and fine-tune network models and optimize model parameters ；

[0144] Specifically, the model was trained and fine-tuned using the image dataset, with the learning rate set to 0.0006, for a total of 1000 rounds, and other parameters remained unchanged to further improve the performance of the classification network. The model was trained again to obtain high-quality classification image results, further improving the accuracy of the classification network.

[0145] S60: After the fine-tuning training is completed, the fine-tuned network parameters are solidified, the final image classification network model is determined, and the model is saved. If a hyperspectral image classification task is performed, the pre-processed hyperspectral image can be directly input into the trained classification network model to obtain the final image classification result.

[0146] The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art. The following content is described in conjunction with data, charts, etc. of the test process.

[0147] The comparison of the related indicators of the existing technical method and the method proposed in Example 1 of the present invention on the Indian Pines dataset, the Salinas dataset and the KSC dataset is shown in Table 1, Table 2 and Table 3 respectively. The classification method proposed in the present invention has higher values ​​of the three indicators of overall accuracy, average accuracy and consistency coefficient on the Indian Pines dataset and the KSC dataset, and the test time is greatly shortened. Although the overall accuracy, average accuracy and consistency coefficient are not greatly improved on the Salinas dataset, the test time is greatly shortened; these indicators show that the technical method proposed in the present invention has better classification effect and higher classification efficiency, and it does have great advantages compared with the existing technology and has achieved the expected effect.

[0148] Table 1 Comparison of classification-related indicators of the Indian Pines dataset

[0149]

[0150] Table 2 Comparison of classification-related indicators of the Salinas dataset

[0151]

[0152] Table 3 Comparison of KSC dataset classification related indicators

[0153]

[0154] Example 2

[0155] An end-to-end hyperspectral image classification system is used to apply the end-to-end hyperspectral image classification method of embodiment 1, such as Figure 2 As shown, including:

[0156] A hyperspectral image data acquisition module is used to acquire the hyperspectral image data to be processed;

[0157] The classification model construction module is used to construct an end-to-end hyperspectral image classification model based on the joint spatial-spectral attention mechanism and the improved Transformer network;

[0158] The classification model training module is used to train the hyperspectral image data to be trained, compare the predicted results with the actual training results to obtain the error and fine-tune the model parameters;

[0159] The classification result generation module is used to evaluate the classification results through classification evaluation indicators and verify the feasibility and superiority of the classification method.

[0160] Example 3

[0161] A computer device includes a hyperspectral image memory and a hyperspectral image classification processor, and a memory storing a computer program; when the computer program is executed by the processor, the processor executes the steps of the end-to-end hyperspectral image classification method of embodiment 1.

[0162] Specific as Fig.13 As shown, it includes a user interface, a processor, a display screen, a classification device, a network interface and a memory connected to a system bus;

[0163] The user interface of the device is a medium for interaction and information exchange between the computer system and the user, which can be a command interface, a graphic interface or a program interface; the processor, as the central nervous system of the computer, has the function of calculating and controlling the program; the display screen is a liquid crystal display screen; the classification device can be a touch layer covered on the display screen, or a key or touch pad set on the computer device shell, or an external keyboard, touch pad or mouse, etc.; the network interface is used to communicate with an external terminal through a network connection; the memory stores a computer program, including a non-volatile storage medium and an internal memory, and the non-volatile storage medium stores an operating system, a network communication module, a database and a hyperspectral image classification program; wherein the operating system can reasonably allocate and schedule computer work resources to provide a convenient interface and environment for users and other software; the network communication module can transmit the stored data between network devices to realize the connection and transmission of the network system; the database stores and manages the data of the hyperspectral image; when the computer program is executed by the processor, the steps of implementing an end-to-end hyperspectral image classification method are implemented; the internal memory provides an environment for the operation of the operating system in the non-volatile storage medium and an end-to-end hyperspectral image classification program.

[0164] Example 4

[0165] An information data processing terminal, the information data processing terminal is used to implement the end-to-end hyperspectral image classification system of embodiment 2

[0166] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An end-to-end hyperspectral image classification method, characterized in that: The following steps are involved: Acquire a hyperspectral image dataset and preprocess the hyperspectral image dataset; Constructing a network model including a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, a feature processing module, and finally classification through a multi-layer perception module; Input the preprocessed hyperspectral image data set into the network model for training; obtain the training weight value; Select a loss function, set a training loss threshold, and continuously iterate and optimize the network model until the number of training times reaches the set threshold or the value of the loss function reaches the set threshold range; use hyperspectral image classification effect evaluation indicators to measure the classification accuracy and performance of the network model; Use hyperspectral image datasets to train and fine-tune the network model and optimize model parameters; The construction includes a network model of a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally in the step of classification through a multi-layer perception module, The image division module divides the original hyperspectral image data into multiple super-pixel images through the SLIC function; The graph convolutional network module with spectral attention mechanism processes the data into 1×1×C1 through a maximum pooling layer, which is used for local information interaction and global information interaction respectively, and combines the graph convolutional neural network to automatically learn important spectral bands, thereby enhancing the network model's attention to key spectral information. The lightweight and highly discriminative spectral feature selection module includes a two-dimensional convolution layer, a batch normalization layer, an activation function layer, a maximum pooling layer, a fully connected layer, a Dropout layer, and a Flatten layer, which selects classification features from the output of the graph convolutional network to reduce redundant information.

2. The end-to-end hyperspectral image classification method according to claim 1, characterized in that: In the step of obtaining a hyperspectral image dataset and preprocessing the hyperspectral image dataset, The hyperspectral image datasets include the Indian Pines dataset, the Salinas dataset, and the KSC dataset.

3. The end-to-end hyperspectral image classification method according to claim 1, characterized in that: The construction includes a network model of a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally in the step of classification through a multi-layer perception module, Firstly, the covariance matrix eigendecomposition of the original hyperspectral image is performed by principal component analysis; The two-dimensional stacking module includes multiple batch normalization layers, activation function layers and two-dimensional convolution layers for dimensional conversion; The spatial attention module includes a maximum pooling layer, an average pooling layer, a two-dimensional convolution layer, a batch normalization layer and an activation function layer, which is used to transform the spatial information in the hyperspectral image data into another space and retain the key information and output it in a weighted manner; The Transformer module includes layer normalization, discriminant regularization, and multi-head attention mechanism layers; The feature processing module includes a batch normalization layer, an activation function layer, a one-dimensional convolution layer, a Dropout layer and an average pooling layer, and further processes the spatial features to obtain one-dimensional spatial features.

4. The end-to-end hyperspectral image classification method according to claim 1, characterized in that: The construction includes a network model of a spectral feature extraction branch and a spatial feature extraction branch, wherein the spectral feature extraction branch includes an image segmentation module, a graph convolutional network module with a spectral attention mechanism, and a lightweight high-discrimination spectral feature selection module; the spatial feature extraction branch includes a two-dimensional stacking module, a spatial attention module, a Transformer module, and a feature processing module, and finally in the step of classification through a multi-layer perception module, The multi-layer perception module includes an input layer, an output layer, two fully connected layers, two activation function layers and a hidden layer, and fuses and classifies the spectral features extracted by the spectral feature extraction branch and the spatial features extracted by the spatial feature extraction branch.

5. The end-to-end hyperspectral image classification method according to claim 1, characterized in that: The loss function is selected, the training loss threshold is set, and the network model is continuously iterated and optimized until the number of training times reaches the set threshold or the value of the loss function reaches the set threshold range; in the step of using the hyperspectral image classification effect evaluation index to measure the classification accuracy and performance of the network model, the loss function selects the cross entropy loss function to truly reflect the difference between the predicted value and the true value after the end-to-end model, and to provide feedback on the quality of the classification network model.

6. An end-to-end hyperspectral image classification system, used to implement the end-to-end hyperspectral image classification method according to any one of claims 1 to 5, characterized in that: include; A hyperspectral image data acquisition module is used to acquire the hyperspectral image data to be processed; The classification model construction module is used to construct an end-to-end hyperspectral image classification model based on the joint spatial-spectral attention mechanism and the improved Transformer network; Classification model training module, which is used to input hyperspectral image data into the classification model construction module for training, compare the prediction results with the actual training results to obtain the error and fine-tune the model parameters; The classification result generation module is used to evaluate the classification results through classification evaluation indicators and verify the feasibility and superiority of the classification method.

7. A computer device, characterized in that: The invention comprises a hyperspectral image memory and a hyperspectral image classification processor, and a memory storing a computer program; when the computer program is executed by the processor, the processor executes the steps of the end-to-end hyperspectral image classification method as claimed in any one of claims 1 to 5.

8. An information data processing terminal, the information data processing terminal is used to implement the end-to-end hyperspectral image classification system as claimed in claim 6.

Images