A hyperspectral image classification method, device and computer readable terminal

Abstract

The present invention discloses a hyperspectral image classification method, device and computer-readable terminal, and relates to the technical field of image classification. The present invention constructs a hyperspectral image classification model based on improved YOLOv7, inserts the improved GAM attention mechanism into the ELAN module in the backbone network to form a new ELAN-GAM module, retains important spectral feature information in three dimensions of the hyperspectral image and amplifies the global spatial interaction features, and at the same time, the added pooling operation can reduce parameters to a certain extent, so that the classification efficiency of the model is improved, and the prediction structure in the head network uses the improved R-FCN classification structure to replace the classification prediction structure of the YOLOv7 head network, and uses RPN to replace the selective search to obtain the hyperspectral image object candidate frame, and after an average pooling, three one-dimensional vectors unified into the same dimension are superimposed to obtain a new feature vector for representing the feature information of the hyperspectral image at different scales, thereby improving the classification efficiency and the classification accuracy of the obtained classification results.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a hyperspectral image classification method, device and computer-readable terminal. Background Art

[0002] At present, hyperspectral image classification is one of the most popular research hotspots in remote sensing image processing tasks. Hyperspectral images involve hundreds to thousands of different spectral bands. Compared with traditional full-color images and multispectral remote sensing images, hyperspectral images often contain richer spectral information and are often obtained by carrying hyperspectral cameras on carriers such as satellites and drones. However, the existing hyperspectral image data will inevitably contain noise due to interference from external factors such as equipment errors and atmospheric factors. Noise, as a bad signal, will cause important associated feature information in the image to be destroyed, thereby making the classification results obtained by the adopted classification model unsatisfactory. Therefore, in order to improve the quality of hyperspectral images and facilitate subsequent hyperspectral image classification operations, it is very necessary to eliminate the noise contained in the hyperspectral image data. It is of great significance to study a hyperspectral image classification model with high computational efficiency and the ability to effectively eliminate noise.

[0003] The Chinese patent publication number is CN 114300099A, and its name is "A method for typing atypical lymphocytes based on YOLOv5 and microscopic hyperspectral images". The improved YOLOv5 network is used to fuse shallow position features and deep semantic features to complete the automatic positioning of lymphocytes. The relative position information of the lymphocyte prediction box in the improved YOLOv5 network recognition result is used to obtain its average spectral value as input. A one-dimensional convolutional neural network is used to build a classification model to realize the classification and recognition of atypical lymphocytes, reduce the amount of network calculation, improve the detection efficiency of lymphocytes, and realize the typing and recognition of atypical lymphocytes. However, the problems and defects of the existing YOLOv5 network technology and YOLOv7 network technology are that the application of hyperspectral image classification is not widespread, and it is mostly used for real-time target detection. Since hyperspectral images are three-dimensional data and have many bands, the traditional YOLOv7 network may not be able to extract its rich spectral information, resulting in low classification accuracy. Therefore, how to effectively solve the above problems has become the key to improving the classification performance of hyperspectral images, which deserves further research and improvement. Summary of the invention

[0004] The technical solution of the present invention to solve the above technical problem is to provide a hyperspectral image classification method, comprising the following steps:

[0005] Prepare hyperspectral image datasets: Prepare three classic hyperspectral image public datasets and divide them into training sets and test sets;

[0006] Label and preprocess the image data: Use tools to label and preprocess the hyperspectral image dataset. After preprocessing, generate pseudo-color images by combining representative bands.

[0007] Build a network model: Build an improved YOLOv7 hyperspectral image classification model including a backbone network and a head network. The backbone network includes a CBS module, an ELAN1 module, an ELAN-GAM module, and an MP module. The head network includes an SPPCSPC module, an ELAN2 module, a CBS module, an improved R-FCN module, UpSample, and DownSample. Finally, the classification result is output through a fully connected layer.

[0008] Train the network model: select the loss function and determine the evaluation index, input the preprocessed training set data into the network model for training, and obtain the training weights;

[0009] Fine-tuning the model: The network model uses the obtained classification results and preset labels to calculate the loss, update the network weights and optimize the model parameters to obtain high-quality classification image results;

[0010] Save model: After the network model fine-tuning training is completed, solidify the fine-tuned network parameters and determine the final image classification network model.

[0011] Furthermore, in the step of constructing the network model,

[0012] The CBS module consists of a convolutional layer, a batch normalization layer and a SiLU activation function;

[0013] Among them, the SiLU activation function is defined as follows:

[0014] ；

[0015] in, Represents the input feature information.

[0016] Furthermore, in the step of constructing the network model,

[0017] The ELAN-GAM module is composed of an improved GAM attention mechanism module and a CBS module, wherein the improved GAM attention mechanism module is composed of a maximum pooling layer, an average pooling layer, an MLP, a two-dimensional convolution layer and an activation function layer;

[0018] The channel attention branch in the improved GAM attention mechanism module uses convolution to generate spectral domain attention feature maps, sums the attention feature maps for pooling operations, aggregates the attention information of the two feature maps through activation functions, retains information in three dimensions of the hyperspectral image, and uses MLP to amplify the dependencies between cross-dimensional channels. The formula of the channel attention branch is:

[0019] ; in, represents the initial feature map of the input channel attention branch, where refers to the activation function, represents two-dimensional convolution, MLP represents multi-layer perceptron, AvgPool represents global average pooling, and MaxPool represents global maximum pooling;

[0020] The spatial attention branch in the improved GAM attention mechanism module uses two-dimensional convolution to pay attention to the extraction of global spatial interaction feature information, and performs serial and parallel joint operations on maximum pooling and average pooling. The formula of the spatial attention branch is:

[0021] ;

[0022] in, represents the initial feature map of the input channel attention branch, where refers to the activation function, represents two-dimensional convolution, AvgPool represents global average pooling, and MaxPool represents global maximum pooling; in the above formula The function looks like this:

[0023] ;

[0024] in, Represents the input feature information.

[0025] Furthermore, in the step of constructing the network model,

[0026] The ELAN-GAM module consists of an improved GAM attention mechanism.

[0027] The ELAN1 module is composed of multiple CBSs;

[0028] The MP module consists of a maximum pooling layer and a CBS module, which achieves downsampling while keeping the dimension unchanged and extracting more hyperspectral image features;

[0029] The SPPCSPC module consists of a maximum pooling layer and a CBS module;

[0030] The ELAN2 module is composed of multiple CBSs;

[0031] The R-FCN module consists of a convolutional layer, an RPN network, a ROI pooling layer, and an average pooling layer;

[0032] Furthermore, in the step of preparing a hyperspectral image dataset,

[0033] Three classic hyperspectral image public datasets include dataset 1, dataset 2, and dataset 3.

[0034] Dataset 1 is the Indian Pines dataset. The Indian Pines dataset is a remote sensing image dataset used for ground object classification and target recognition tasks. The Indian Pines dataset includes 16 ground object categories, including forests, corn, soybeans, and wheat.

[0035] The second dataset is the Urban dataset. The Urban dataset is a hyperspectral image dataset used for ground object classification tasks. It has the characteristics of high spatial resolution. It is acquired by sensors with a spatial resolution of 3.7 meters and a size of 307×307. The original data has 210 bands.

[0036] Dataset 3 is the Pavia University dataset. The Pavia University dataset is obtained by an airborne visible infrared imaging spectrometer sensor. The spatial size is 512×614 and the spatial resolution is 18m. The dataset is divided into 9 categories and 115 bands.

[0037] Furthermore, the step of labeling and preprocessing the image data includes:

[0038] The GeoLabel image annotation tool is used to annotate the highlight atlas dataset, and low-rank tensor decomposition is used to denoise the data.

[0039] The specific components of the hyperspectral image with mixed noise are:

[0040] ;

[0041] in, represents a hyperspectral image with mixed noise, represents an image without noise, represents sparse noise, Represents Gaussian noise; introduces parameter , the performance of hyperspectral denoising is optimized and improved by reducing the noise parameter, The expression is:

[0042] ;

[0043] in, Represents a minimize operation. represents a non-negative parameter, represents the sparse norm of the noise, represents the Frobenius-norm of Gaussian noise;

[0044] After reducing the noise through the noise parameter formula, the low-rank tensor decomposition is used to convert any n-order tensor of the image without noise into Decomposed into a linear combination of R rank tensors, the formula for low rank tensor decomposition is:

[0045] ;

[0046] in, represents the vector outer product, represents the weight of a rank tensor, , , Represent different vectors of the rank tensor, that is, the hyperspectral image without noise Different feature vector information in ;

[0047] For a hyperspectral image feature with a channel number of C , it is divided into C feature tensors, and then rank tensor decomposition is used to characterize the low-rank characteristics of the noise-minimized hyperspectral image. Average pooling is used to extract related information in three dimensions respectively. The rank-one vectors of three dimensions are obtained and fused through the nonlinear mapping of convolution and activation functions to refine the image features and obtain a low-rank hyperspectral image feature map.

[0048] Furthermore, in the step of selecting a loss function and determining an evaluation index,

[0049] The predicted box is compared with the true box, and the loss error is calculated using the improved EIOU loss function to reversely update the network weights, which is used to truly reflect the quality of the hyperspectral image classification network model based on the improved YOLOv7 and improve the classification efficiency. The calculation formula of the improved EIOU loss function is as follows:

[0050] ;

[0051] in, Indicates improved loss, Indicates the loss of the object position, represents the prediction box width loss, represents the prediction box height loss, represents the confidence loss, Indicates category loss; object location loss , the calculation formula is as follows:

[0052] ;

[0053] in, Represents the intersection-over-union ratio of the predicted box and the true box, represents the center point of the prediction box, represents the center point of the ground-truth box, is the Euclidean distance between two center points, It represents the diagonal distance of the smallest rectangle that can cover both the prediction box and the target box;

[0054] Prediction box width loss , the calculation formula is as follows:

[0055] ;

[0056] in, Indicates the width of the prediction box, Indicates the width of the real box, represents the Euclidean distance between the widths of two boxes, Indicates the width of the minimum bounding box covering the predicted box and the true box; predicted box height loss , the calculation formula is as follows:

[0057] ;

[0058] in, Represents the height of the prediction box, Indicates the height of the real frame, represents the Euclidean distance between the heights of two boxes, Indicates the height of the minimum bounding box covering the predicted box and the true box;

[0059] Confidence loss , the calculation formula is as follows:

[0060] + ;

[0061] ;

[0062] ;

[0063] ;

[0064] Where N represents the total number of bounding boxes classified by the network, Indicates Whether there is an object in the bounding box of each category, Indicates the target location The confidence of the bounding box, Represents the network prediction Confidence of bounding boxes; Class loss , the calculation formula is as follows:

[0065] ;

[0066] Where N represents the total number of bounding boxes classified by the network, Indicates Whether there is an object in the bounding box of each category, Indicates the target location The probability of each class in the bounding box, Represents the network prediction The probability of each class in the bounding box;

[0067] The evaluation indicators include overall accuracy, average accuracy and recall rate. The overall accuracy is used to measure the overall classification accuracy of the classification model, and the average accuracy is used to indicate the classification accuracy of the classification model for a certain category. The recall rate is used to evaluate whether the network's predicted value is consistent with the true value. The overall accuracy is used to measure the overall classification accuracy of the classification model, and the average accuracy is used to indicate the classification accuracy of the classification model for a certain category. The recall rate is used to evaluate whether the network's predicted value is consistent with the true value. , Average Precision The recall calculation formula is as follows:

[0068] ；

[0069] ;

[0070] ;

[0071] Among them, TP is a positive sample correctly classified by the model, FN is a positive sample incorrectly classified by the model, FP is a negative sample incorrectly classified by the model, and TN is a negative sample correctly classified by the model. In order to solve the above technical problems, the present invention also proposes a hyperspectral image classification device based on an improved YOLOv7, comprising:

[0072] A hyperspectral image data acquisition module is used to acquire the hyperspectral image data to be processed;

[0073] The dataset annotation module is used to annotate the hyperspectral image dataset using the GeoLabel image annotation tool.

[0074] The data preprocessing module is used to denoise the hyperspectral image by using the low-rank tensor decomposition method to generate a pseudo-color image by combining representative bands;

[0075] Classification model construction module, used to construct a hyperspectral image classification model;

[0076] The classification model training module is used to train the denoised hyperspectral image data and select the loss function;

[0077] The classification result generation module inputs the images of the test set or the images to be predicted into the trained model. After obtaining the classification results, the classification results are evaluated through classification evaluation indicators to verify the feasibility and superiority of the classification method.

[0078] In order to solve the above technical problem, the present invention further proposes a computer-readable terminal storing a computer program. When the computer program is executed by a processor, the processor executes the steps of a hyperspectral image classification method as described above.

[0079] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0080] (1) The entire classification model cleverly combines traditional denoising methods and deep learning classification methods. It uses traditional denoising methods to extract image features after noise removal from the original data and uses the improved YOLOv7 hyperspectral image classification network model to learn the features. This greatly reduces the computational time complexity of classifying and learning the features of the original hyperspectral image data and improves the generalization ability of the model.

[0081] (2) The present invention adopts a low-rank tensor method to perform denoising preprocessing on the hyperspectral image and combines it with a band-combining method to generate an image. This can effectively suppress mixed noise while fully retaining the original hyperspectral image structure, so that the quality of the hyperspectral image after denoising is significantly improved, and it is easier to make the generated image data suitable for the YOLOv7 network classification model, with more accurate classification results.

[0082] (3) The present invention inserts the improved GAM attention mechanism module into the ELAN module in the backbone feature network of YOLOv7. The improved GAM attention mechanism can retain important spectral feature information in three dimensions of hyperspectral images and amplify global spatial interaction features. At the same time, the added pooling operation can reduce parameters to a certain extent, thereby improving the classification efficiency of the model.

[0083] (4) The present invention adopts an improved R-FCN classification structure to replace the classification prediction structure of the YOLOv7 head network, and uses RPN to replace the selective search to obtain the candidate frames of the hyperspectral image objects. After an average pooling, the three one-dimensional vectors unified into the same dimension are superimposed to obtain a new feature vector for representing the feature information of the hyperspectral image at different scales, thereby improving the classification efficiency of the model.

[0084] (5) The present invention optimizes the CIOU loss function in YOLOv7 into an improved EIOU loss function, which is more suitable for object detection and classification of hyperspectral image data. It can comprehensively measure the similarity between the predicted box and the true box, help solve the problem of feature propagation capability degradation of the classification model and continuously optimize it, thereby improving the classification accuracy of the obtained classification results.

[0085] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0086] (1) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0087] The present invention adopts a hyperspectral image classification network model based on improved YOLOv7 combined with the traditional denoising method, making full use of the improved GAM attention mechanism module and R-FCN classification structure.

[0088] The quality of hyperspectral images is improved through denoising while retaining important spectral feature information in three dimensions and amplifying global spatial interaction features, which facilitates subsequent classification and effectively improves classification accuracy and efficiency.

[0089] (2) Whether the technical solution of the present invention overcomes technical bias:

[0090] Currently, many researchers believe that the YOLOv7 network is more suitable for target detection scenarios. The hyperspectral image classification network model using YOLOv7 may result in poor classification results. The technical solution provided by the present invention applies the improved YOLOv7 network and loss function to the hyperspectral image classification scenario, comprehensively measures the similarity between the predicted box and the true box, helps to solve the problem of degradation of the feature propagation ability of the classification model and continuously optimizes it, improves the classification accuracy of the obtained classification results, and improves the accuracy and efficiency of hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0092] Figure 1 This is a flowchart of the steps of the hyperspectral image classification method of the present invention;

[0093] Figure 2 It is a structural diagram of a hyperspectral image classification device based on improved YOLOv7 of the present invention;

[0094] Figure 3 This is a schematic diagram of the specific composition of the CBS module of the present invention;

[0095] Figure 4 It is the overall network structure diagram based on the improved YOLOv7 described in the present invention;

[0096] Figure 5 This is a schematic diagram of the specific composition of the MP module of the present invention;

[0097] Figure 6 This is a schematic diagram of the specific composition of the ELAN-GAM module of the present invention;

[0098] Figure 7 It is a schematic diagram of the specific composition of the SPPCSPC module of the present invention;

[0099] Figure 8 Schematic diagram of the specific composition of the R-FCN module of the present invention;

[0100] Fig. 9 This is a schematic diagram of the specific composition of the improved GAM attention mechanism module described in the present invention;

[0101] Fig.10 A schematic diagram of the structure of a computer system of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0102] The present invention proposes a hyperspectral image classification method, a device and a computer-readable terminal, aiming to design a hyperspectral image classification method.

[0103] The hyperspectral image classification method proposed by the present invention will be described in the following specific embodiments:

[0104] Embodiment 1: A hyperspectral image classification method, such as Figure 1 As shown, the following steps are included:

[0105] S10: Prepare hyperspectral image datasets: Prepare three classic hyperspectral image public datasets and divide them into training sets and test sets;

[0106] A certain proportion of sample data is selected from the three data sets as training sets to perform network training and model fine-tuning. In order to verify the advancedness of the present invention, the model is tested using data from the three data sets respectively.

[0107] S20: Annotate and preprocess image data: Use tools to annotate the hyperspectral image data set and preprocess the data. After preprocessing, generate a pseudo-color image by combining representative bands;

[0108] S30: Build network model: Build a hyperspectral image classification model of improved YOLOv7 including backbone network and head network, such as Figure 4 As shown, the backbone network includes a CBS module, an ELAN1 module, an ELAN-GAM module, and an MP module, and the head network includes an SPPCSPC module, an ELAN2 module, a CBS module, an improved R-FCN module, UpSample and DownSample, and finally outputs the classification result through a fully connected layer;

[0109] S40: training the network model: selecting a loss function and determining an evaluation index, inputting the preprocessed training set data into the network model for training, and obtaining training weights;

[0110] S50: Fine-tuning the model: The network model uses the obtained classification results and preset labels to calculate the loss, update the network weights and optimize the model parameters to obtain high-quality classification image results;

[0111] By continuously training the network, the model can acquire the ability to classify images, obtain a trained model, and further improve the accuracy of the classification network;

[0112] S60: Save model: After the network model fine-tuning training is completed, the fine-tuned network parameters are solidified to determine the final image classification network model;

[0113] It can be understood that when performing a hyperspectral image classification task, the processed hyperspectral image can be directly input into the trained classification network model to obtain the final image classification result.

[0114] Furthermore, in the step of constructing the network model,

[0115] like Figure 3 As shown, the CBS module consists of a convolutional layer, a batch normalization layer and a SiLU activation function;

[0116] Among them, the SiLU activation function is defined as follows:

[0117] ；

[0118] in, Represents the input feature information.

[0119] Furthermore, in the step of constructing the network model,

[0120] like Figure 6 As shown, the ELAN-GAM module consists of an improved GAM attention mechanism module and a CBS module, where Fig. 9 As shown in Figure 1, the improved GAM attention mechanism module consists of a maximum pooling layer, an average pooling layer, an MLP, a two-dimensional convolutional layer, and an activation function layer;

[0121] The channel attention mechanism and spatial attention mechanism with added convolution layers are used to extract features in parallel and then perform feature fusion; the channel attention branch in the improved GAM attention mechanism module uses convolution to generate spectral domain attention feature maps, sums the attention feature maps for pooling operations, aggregates the attention information of the two feature maps through activation functions, retains information in three dimensions of the hyperspectral image, and uses MLP to amplify the dependencies between cross-dimensional channels. The formula of the channel attention branch is:

[0122]

[0123] in, represents the initial feature map of the input channel attention branch, where refers to the activation function, represents two-dimensional convolution, MLP represents multi-layer perceptron, AvgPool represents global average pooling, and MaxPool represents global maximum pooling;

[0124] The spatial attention branch in the improved GAM attention mechanism module uses two-dimensional convolution to pay more attention to the extraction of global spatial interaction feature information. The serial and parallel joint operations of maximum pooling and average pooling can effectively reduce the number of parameters while capturing important spatial features. The formula of the spatial attention branch is:

[0125]

[0126] in, represents the initial feature map of the input channel attention branch, where refers to the activation function, Represents two-dimensional convolution, AvgPool represents global average pooling, and MaxPool represents global maximum pooling;

[0127] In the above formula The function looks like this:

[0128] ;

[0129] in, Represents input feature information ；

[0130] The improved GAM attention mechanism module adds the pooling operation deleted by the original GAM attention module back into the module to prevent a significant increase in parameters. After introducing the ELAN module of the backbone network, a new ELAN-GAM structure is obtained, and the ELAN structure in the YOLOv7 model is replaced, so that the network model can better extract multi-scale feature information and effectively improve the efficiency of capturing features. The fusion feature operation adopted in the connection part of the backbone network and the head network is mainly achieved by connecting the ELAN-GAM module in the backbone network with the SPPCSPC module in the head network. The improved GAM attention mechanism module adds the pooling operation deleted by the original GAM attention module back into the module to prevent a significant increase in parameters. At the same time, the channel attention mechanism and the spatial attention mechanism with the added convolution layer extract features in parallel and then perform feature fusion, which can effectively capture important feature information in three dimensions of hyperspectral image data.

[0131] Furthermore, in the step of constructing the network model,

[0132] The ELAN1 module is composed of multiple CBSs;

[0133] like Figure 5 As shown, the MP module consists of a maximum pooling layer and a CBS module, which achieves downsampling while keeping the dimension unchanged and extracting more hyperspectral image features;

[0134] like Figure 7 As shown in the figure, the SPPCSPC module consists of a maximum pooling layer and a CBS module. Its input is divided into two parts. One part enters a triple pooling layer after triple convolution. The three independent maximum pooling layers have three receptive fields. Different pooling layers can be more suitable for the classification of hyperspectral image data. After splicing, they are fused by convolution.

[0135] The ELAN2 module is composed of multiple CBSs;

[0136] like Figure 8As shown in the figure, the R-FCN module (combining the characteristics of the candidate region network and average pooling to form a new structure focused on classification) consists of a convolutional layer, an RPN network, a ROI pooling layer and an average pooling layer. The RPN (candidate region network) is used to replace the selective search, that is, the hyperspectral image object candidate frame is obtained by directly training the RPN, so that the classification speed of the hyperspectral image data is greatly improved;

[0137] Among them, the convolution layer of the R-FCN module selects a convolution layer with a convolution kernel size of 1×1, and after convolution, it is input into the RPN network and the ROI pooling layer respectively. One branch is directly trained through the RPN network to obtain the candidate frame of the hyperspectral image object, which greatly improves the classification speed of the hyperspectral image data. The convolution layer selects a convolution layer with a convolution kernel size of 3×3 / 5×5 / 7×7. After that, the result obtained by the convolution layer is input into the ReLU activation function, and then input into two convolution kernels with a size of 1×1 for fusion, which can improve the generation speed of the image prediction frame. The other branch quickly obtains the fixed The corresponding feature map list of size can use the same input feature map for various ground objects in the framework, which can effectively improve the feature processing speed and save a lot of time. The outputs of the two branches are connected in parallel and fused to obtain the results and input into the average pooling layer, unified into three one-dimensional vectors of the same dimension. By calculating the superposition and sum of the three one-dimensional vectors, a new feature vector is obtained. This vector represents the feature information of the image at multiple scales, and the high classification accuracy of the hyperspectral image classification model based on the improved YOLOv7 is guaranteed as much as possible and the classification efficiency is improved. Finally, a fully connected layer with a neuron as the number of categories is used to obtain the final classification result. The ReLU activation function is defined as follows:

[0138]

[0139] in, Represents the input feature information.

[0140] Furthermore, in the step of preparing a hyperspectral image dataset,

[0141] Three classic hyperspectral image public datasets include dataset 1, dataset 2, and dataset 3.

[0142] Dataset 1 is the Indian Pines dataset. The Indian Pines dataset is a remote sensing image dataset used for ground object classification and target recognition tasks. The Indian Pines dataset includes 16 ground object categories, including forests, corn, soybeans, and wheat. The research generally takes the remaining 200 bands after removing 20 water absorption bands as the research object, annotates the data with an image space size of 145×145 and uses it for hyperspectral image classification.

[0143] The second dataset is the Urban dataset. The Urban dataset is a hyperspectral image dataset used for ground object classification tasks and has the characteristics of high spatial resolution. It is acquired by sensors with a spatial resolution of 3.7 meters and a size of 307×307. The original data has 210 bands. After removing noise and water absorption bands, there are 162 bands that can be processed and analyzed for hyperspectral image classification.

[0144] Dataset 3 is the Pavia University dataset. The Pavia University dataset is obtained by an airborne visible infrared imaging spectrometer sensor. The spatial size is 512×614 and the spatial resolution is 18m. The dataset is divided into 9 categories and 115 bands. After removing 12 noise bands, there are 103 available bands left for hyperspectral image classification.

[0145] Furthermore, the step of labeling and preprocessing the image data includes:

[0146] The highlight atlas dataset is annotated using the GeoLabel image annotation tool, and the data is denoised using low-rank tensor decomposition. Since the spatial information and spectral information of the highlight image data are correlated, and the noise-free hyperspectral images exist in a low-dimensional subspace, the low rank nature of the hyperspectral images in the feature dimension can be described by low-rank tensor decomposition.

[0147] The specific components of the hyperspectral image with mixed noise are:

[0148] ;

[0149] in, represents a hyperspectral image with mixed noise, represents an image without noise, represents sparse noise, represents Gaussian noise;

[0150] Introducing parameters , the performance of hyperspectral denoising is continuously optimized and improved by reducing the noise parameters. The expression is:

[0151] ;

[0152] in, Represents a minimize operation. represents a non-negative parameter, represents the sparse norm of the noise, represents the Frobenius-norm of Gaussian noise;

[0153] After reducing the noise through the noise parameter formula, the low-rank tensor decomposition is used to convert any n-order tensor of the image without noise into Decomposed into a linear combination of R rank tensors, the formula for low rank tensor decomposition is:

[0154] ;

[0155] in, represents the vector outer product, represents the weight of a rank tensor, , , Represent different vectors of the rank tensor, that is, the hyperspectral image without noise Different feature vector information in; for a hyperspectral image feature with C channels , it is divided into C feature tensors, and then the rank tensor decomposition is used to characterize the low-rank characteristics of the hyperspectral image after noise minimization. Average pooling is used to extract related information in three dimensions respectively. The rank-one vectors of three dimensions are obtained and fused through the nonlinear mapping of convolution and activation functions, the image features are refined and the low-rank hyperspectral image feature map is obtained. Taking the Indian Pines dataset as an example, a pseudo-color image is generated by combining the representative bands 8, 106 and 159 in the image data, so that the obtained image data can be correctly read by the YOLOv7 network, which is convenient for the subsequent input of the pseudo-color image into the constructed improved YOLOv7 network for classification.

[0156] Furthermore, in the step of selecting a loss function and determining an evaluation index,

[0157] The predicted box information of the target of interest is obtained through the above-mentioned hyperspectral image classification network model based on the improved YOLOv7, and the predicted box is compared with the real box. The loss error is calculated and the improved EIOU loss function is used to reversely update the network weights to truly reflect the quality of the hyperspectral image classification network model based on the improved YOLOv7 and improve the classification efficiency. The calculation formula of the improved EIOU loss function is as follows:

[0158] ;

[0159] in, Indicates improved loss, Indicates the loss of the object position, represents the prediction box width loss, represents the prediction box height loss, represents the confidence loss, represents the class loss;

[0160] Loss of ground position , the calculation formula is as follows:

[0161] ;

[0162] in, Represents the intersection-over-union ratio of the predicted box and the true box, represents the center point of the prediction box, represents the center point of the ground-truth box, is the Euclidean distance between two center points, It represents the diagonal distance of the smallest rectangle that can cover both the prediction box and the target box;

[0163] Prediction box width loss , the calculation formula is as follows:

[0164] ;

[0165] in,, Indicates the width of the prediction box, Indicates the width of the real box, represents the Euclidean distance between the widths of two boxes, Indicates the width of the minimum bounding box covering the predicted box and the true box;

[0166] Prediction box height loss , the calculation formula is as follows:

[0167] ;in, Represents the height of the prediction box, Indicates the height of the real frame, represents the Euclidean distance between the heights of two boxes, Indicates the height of the minimum bounding box covering the predicted box and the true box;

[0168] Confidence loss , the calculation formula is as follows:

[0169] + ;

[0170] ;

[0171] ;

[0172] ;

[0173] Where N represents the total number of bounding boxes classified by the network, Indicates Whether there is an object in the bounding box of each category, Indicates the target location The confidence of the bounding box, Represents the network prediction The confidence of the bounding box;

[0174] Class loss , the calculation formula is as follows:

[0175] ;

[0176] Where N represents the total number of bounding boxes classified by the network, Indicates Whether there is an object in the bounding box of each category, Indicates the target location The probability of each class in the bounding box, Represents the network prediction The probability of each class in the bounding box;

[0177] The evaluation indicators include overall accuracy, average precision and recall rate, which can effectively evaluate the quality of classification and measure the role of classification network. Among them, overall accuracy is used to measure the overall classification accuracy of the classification model, and average accuracy is used to indicate the classification accuracy of the classification model for a certain category. Recall rate is used to evaluate whether the predicted value of the network is consistent with the true value. Overall accuracy is used to measure the overall classification accuracy of the classification model, and average accuracy is used to measure the classification accuracy of the classification model for a certain category. , Average Precision The recall calculation formula is as follows:

[0178] ；

[0179] ;

[0180] ;

[0181] Among them, TP is the positive sample correctly classified by the model, FN is the positive sample incorrectly classified by the model, FP is the negative sample incorrectly classified by the model, and TN is the negative sample correctly classified by the model. Verification test:

[0182] Comparisons of relevant indicators between the prior art method and the method proposed in the present invention on the Indian Pines dataset, the Salinas dataset, and the Pavia University dataset are shown in Table 1, Table 2, and Table 3, respectively.

[0183] Existing method 1 adopts Chinese patent publication number CN117788889A, entitled "A Transformer-based hyperspectral image classification method and system", which divides the preprocessed hyperspectral image data into multiple hyperspectral image overlapping cubes through a one-dimensional mirror pooling grouping module in a grouping feature tokenization module, extracts embedded features of local spatial-spectral relationships along the spatial dimension, and extracts more hierarchical spatial-spectral interaction features along the spectral dimension through a one-dimensional Swin Transformer module with cross-block normalization connections; finally, the classification results are output through a multi-layer perceptron.

[0184] Existing method 2 adopts Chinese patent publication number CN115471757A, named "Hyperspectral Image Classification Method Based on Convolutional Neural Network and Attention Mechanism", and extracts the spectral and spatial joint features of the preprocessed hyperspectral image data cube through three-dimensional convolution. At the same time, the joint features are input into the attention module to focus on the adjacent spectral correlation and spatial correlation, and the feature map with attention weight is input into the two-dimensional convolution module to further refine the spatial features. Finally, classification is performed through the classification layer to output the classification results.

[0185] By comparison, the classification method adopted in the present invention has higher values ​​of the three indicators of overall accuracy, average precision, and recall rate on the Indian Pines dataset, Urban dataset, and Pavia University dataset, and the test time has been reduced to a certain extent. These indicators show that the technical method proposed in the present invention has better classification effect and higher classification efficiency, and indeed has great advantages compared with the existing technology.

[0186] Table 1 Comparison of classification-related indicators of the Indian Pines dataset

[0187] method Overall accuracy Average precision Recall Test time(s) Existing method 1 94.79% 93.62% 0.9279 504 Existing method 2 92.16% 93.57% 0.9215 295 This method 96.23% 94.48% 0.9532 133.7

[0188] Table 2 Comparison of relevant indicators of Urban dataset classification

[0189] method Overall accuracy Average precision Recall Test time(s) Existing method 1 95.46% 96.88% 0.9584 972 Existing method 2 96.72% 94.17% 0.9563 483 This method 97.34% 97.39% 0.9749 102

[0190] Table 3 Comparison of classification-related indicators of Pavia University dataset

[0191] method Overall accuracy Average precision Recall Test time(s) Existing method 1 96.65% 98.10% 0.9684 168 Existing method 2 97.04% 97.92% 0.9735 521 This method 98.79% 98.35% 0.9864 95.29

[0192] Embodiment 2: A hyperspectral image classification device based on improved YOLOv7, such as Figure 2 As shown, including:

[0193] A hyperspectral image data acquisition module is used to acquire the hyperspectral image data to be processed;

[0194] The dataset annotation module is used to annotate the hyperspectral image dataset using the GeoLabel image annotation tool to improve the efficiency of hyperspectral image data annotation.

[0195] The data preprocessing module is used to denoise the hyperspectral image by using the low-rank tensor decomposition method to generate a pseudo-color image by combining representative bands;

[0196] A classification model construction module, used to construct a hyperspectral image classification model using the hyperspectral image classification method of Example 1;

[0197] A classification model training module, used to train the denoised hyperspectral image data using the improved YOLO v7 network model in Example 1 and select a loss function;

[0198] The classification result generation module inputs the images of the test set or the images to be predicted into the trained model. After obtaining the classification results, the classification results are evaluated through classification evaluation indicators to verify the feasibility and superiority of the classification method.

[0199] The loss function described in this embodiment selects the improved EIOU loss function to improve the quality and classification efficiency of the hyperspectral image classification network model based on the improved YOLOv7.

[0200] The improved YOLO v7 network model described in this embodiment is composed of a backbone network and a head network. The backbone network adds the improved GAM attention mechanism to the YOLO v7 model to obtain a new ELAN-GAM structure, and replaces the ELAN structure in the original YOLO v7 model backbone network; the head network adopts a triple pooling structure for splicing, and then uses convolution to fuse and cross-connect, and uses three scale feature outputs to obtain a feature vector representing the feature information of the image at multiple scales through pooling and unified fusion, and then classifies to effectively improve the accuracy of network classification.

[0201] Embodiment 3: A computer-readable terminal stores a computer program. When the computer program is executed by a processor, the processor executes the steps of a hyperspectral image classification method of Embodiment 1.

[0202] Embodiment 4: A computer device, such as Fig.10As shown, the electronic device specifically includes a CPU, ROM, RAM, I / O interface, storage part, driver and communication part;

[0203] The CPU, as the central nervous system of the computer, has the function of calculating and controlling the program; the display screen is a liquid crystal display screen; the classification device can be a touch layer covering the display screen, or a button or touch pad set on the computer device shell, or an external keyboard, touch pad or mouse, etc.; the ROM has a simple structure and the stored data is stable. Once the data is stored, it can no longer be changed or deleted, and the stored data will not disappear after power failure. This embodiment is used to store hyperspectral image classification solidification programs and data; RAM is an internal memory that directly exchanges data with the CPU, also called main memory, which can read and write programs efficiently and quickly at any time, and is usually used as a temporary data storage medium for operating systems or other running programs; among them, the I / O interface of the device serves as a medium for interaction and information exchange between the computer system and the user, which can be It is a command interface, a graphic interface or a program interface; the storage part stores computer programs, including a non-volatile storage medium and an internal memory, and the non-volatile storage medium stores an operating system, a network communication module, a database and a hyperspectral image classification program; the driver is a power electronic device used to control the operation of the computer system equipment and receive signals from the control system, and convert these signals into instructions that the motor can understand and execute, so as to drive the motor to run the hyperspectral image classification program based on the improved YOLOv7 in a predetermined manner; the communication part can transmit the stored data between network devices to realize the connection of the network system; the database stores and manages the data of the hyperspectral image; when the computer program is executed by the processor, the steps of a hyperspectral image classification method based on the improved YOLOv7 are implemented.

[0204] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A hyperspectral image classification method, It is characterized in that The following steps are involved: Prepare hyperspectral image datasets: Prepare three classic hyperspectral image public datasets and divide them into training sets and test sets; Label and preprocess the image data: Use tools to label and preprocess the hyperspectral image dataset. After preprocessing, generate pseudo-color images by combining representative bands. Build a network model: Build an improved YOLOv7 hyperspectral image classification model including a backbone network and a head network. The backbone network includes a CBS module, an ELAN1 module, an ELAN-GAM module, and an MP module. The head network includes an SPPCSPC module, an ELAN2 module, a CBS module, an improved R-FCN module, UpSample, and DownSample. Finally, the classification result is output through a fully connected layer. Train the network model: select the loss function and determine the evaluation index, input the preprocessed training set data into the network model for training, and obtain the training weights; Fine-tuning the model: The network model uses the obtained classification results and preset labels to calculate the loss, update the network weights and optimize the model parameters to obtain high-quality classification image results; Save model: After the network model fine-tuning training is completed, solidify the fine-tuned network parameters and determine the final image classification network model; In the step of constructing the network model, the ELAN-GAM module is composed of an improved GAM attention mechanism module and a CBS module, wherein the improved GAM attention mechanism module is composed of a maximum pooling layer, an average pooling layer, an MLP, a two-dimensional convolution layer and an activation function layer; the channel attention branch in the improved GAM attention mechanism module uses convolution to generate a spectral domain attention feature map, sums the attention feature map for the pooling operation, aggregates the attention information of the two feature maps through the activation function, retains information in three dimensions of the hyperspectral image, and uses MLP to amplify the dependency between cross-dimensional channels. The formula of the channel attention branch is: ;in, represents the initial feature map of the input channel attention branch, where refers to the activation function, Represents two-dimensional convolution, MLP represents multi-layer perceptron, AvgPool represents global average pooling, and MaxPool represents global maximum pooling. The spatial attention branch in the improved GAM attention mechanism module uses two-dimensional convolution to pay attention to the extraction of global spatial interaction feature information, and performs serial and parallel joint operations on maximum pooling and average pooling. The formula of the spatial attention branch is: ;in, represents the initial feature map of the input channel attention branch, where refers to the activation function, represents two-dimensional convolution, AvgPool represents global average pooling, and MaxPool represents global maximum pooling; in the above formula The function looks like this: ;in, Represents the input feature information.

2. The hyperspectral image classification method according to claim 1, It is characterized in that In the step of constructing the network model, the CBS module is composed of a convolutional layer, a batch normalization layer and a SiLU activation function; wherein the SiLU activation function is defined as follows: ； in, Represents the input feature information.

3. The hyperspectral image classification method according to claim 1, It is characterized in that In the step of constructing the network model, the ELAN1 module is composed of multiple CBSs; the MP module is composed of a maximum pooling layer and a CBS module, which realizes downsampling while maintaining the dimension unchanged, and extracts more hyperspectral image features; The SPPCSPC module consists of a maximum pooling layer and a CBS module; the ELAN2 module consists of multiple CBSs; the R-FCN module consists of a convolutional layer, an RPN network, a ROI pooling layer and an average pooling layer.

4. The hyperspectral image classification method according to claim 1, It is characterized in that In the step of preparing the hyperspectral image dataset, three classic hyperspectral image public datasets include dataset one, dataset two, and dataset three. Dataset one is the Indian Pines dataset. The Indian Pines dataset is a remote sensing image dataset used for ground object classification and target recognition tasks. The Indian Pines dataset includes forests, corn, soybeans, wheat, and a total of 16 ground object categories; Dataset two is the Urban dataset. The Urban dataset is a hyperspectral image dataset used for ground object classification tasks and has the characteristics of high spatial resolution; it is acquired by a sensor with a spatial resolution of 3.7 meters, a size of 307×307, and 210 raw data bands; Dataset three is the Pavia University dataset. The Pavia University dataset is obtained by an airborne visible infrared imaging spectrometer sensor, with a spatial size of 512×614 and a spatial resolution of 18m. The dataset is divided into 9 categories and 115 bands.

5. The hyperspectral image classification method according to claim 1, It is characterized in that The steps of labeling and preprocessing the image data include: labeling the hyperspectral atlas dataset using the GeoLabel image labeling tool, and denoising the data using low-rank tensor decomposition; the specific components of the hyperspectral image with mixed noise are: ;in, represents a hyperspectral image with mixed noise, represents an image without noise, represents sparse noise, Represents Gaussian noise; reducing the noise parameter optimizes and improves the performance of hyperspectral denoising. The expression is: ;in, Represents a minimize operation. represents a non-negative parameter, represents the sparse norm of the noise, Represents the Frobenius-norm of Gaussian noise; after reducing the noise through the noise parameter formula, the low-rank tensor decomposition is used to convert any n-order tensor of the image without noise into Decomposed into a linear combination of R rank tensors, the formula for low rank tensor decomposition is: ;in, represents the vector outer product, represents the weight of a rank tensor, , , Represent different vectors of the rank tensor, that is, the hyperspectral image without noise Different feature vector information in; for a hyperspectral image feature with a channel number C , it is divided into C feature tensors, and then rank tensor decomposition is used to characterize the low-rank characteristics of the noise-minimized hyperspectral image. Average pooling is used to extract related information in three dimensions respectively. The rank-one vectors of three dimensions are obtained and fused through the nonlinear mapping of convolution and activation functions to refine the image features and obtain a low-rank hyperspectral image feature map.

6. The hyperspectral image classification method according to claim 1, It is characterized in that In the step of selecting the loss function and determining the evaluation index, the predicted box is compared with the real box, and the loss error is calculated by using the improved EIOU loss function to reversely update the network weights, which is used to truly reflect the quality of the improved YOLOv7 hyperspectral image classification model and improve the classification efficiency. The calculation formula of the improved EIOU loss function is as follows: ;in, Indicates improved loss, Indicates the loss of the object position, represents the prediction box width loss, represents the prediction box height loss, represents the confidence loss, Indicates category loss; object location loss , the calculation formula is as follows: ;in, Represents the intersection-over-union ratio of the predicted box and the true box, represents the center point of the prediction box, represents the center point of the ground-truth box, is the Euclidean distance between two center points, Represents the diagonal distance of the smallest rectangle that can cover both the prediction box and the target box; prediction box width loss , the calculation formula is as follows: ;in, Indicates the width of the prediction box, Indicates the width of the real box, represents the Euclidean distance between the widths of two boxes, Indicates the width of the minimum bounding box covering the predicted box and the true box; Prediction box height loss , the calculation formula is as follows: ;in, Represents the height of the prediction box, Indicates the height of the real frame, represents the Euclidean distance between the heights of two boxes, Indicates the height of the minimum bounding box covering the predicted value and the true box; confidence loss , the calculation formula is as follows: + ; ; ; ; where N represents the total number of bounding boxes classified by the network, Indicates Whether there is an object in the bounding box of each category, Indicates the target location The confidence of the bounding box, Represents the network prediction Confidence of bounding boxes; Class loss , the calculation formula is as follows: ; where N represents the total number of bounding boxes classified by the network, Indicates Whether there is an object in the bounding box of each category, Indicates the target location The probability of each class in the bounding box, Represents the network prediction The probability of each category in the bounding box is used as the evaluation index; the overall accuracy, average precision and recall rate are selected, among which the overall accuracy is used to measure the overall classification accuracy of the classification model, and the average precision is used to indicate the classification accuracy of the classification model for a certain category; the recall rate is used to evaluate the ability of the network to determine whether the predicted value is consistent with the true value; the overall accuracy , Average Precision and recall The calculation formula is as follows: ； ; ; Among them, TP is the positive sample classified correctly by the model, FN is the positive sample classified incorrectly by the model, FP is the negative sample classified incorrectly by the model, and TN is the negative sample classified correctly by the model.

7. A hyperspectral image classification device based on improved YOLOv7, used to implement the steps of the hyperspectral image classification method as described in any one of claims 1 to 6, It is characterized in that include: A hyperspectral image data acquisition module is used to acquire the hyperspectral image data to be processed; The dataset annotation module is used to annotate the hyperspectral image dataset using the GeoLabel image annotation tool. The data preprocessing module is used to denoise the hyperspectral image using the low-rank tensor decomposition method to generate a pseudo-color image by combining representative bands. The classification model construction module is used to construct a hyperspectral image classification model; the classification model training module is used to train the hyperspectral image data after denoising and select the loss function; the classification result generation module inputs the image of the test set or the image to be predicted into the trained model, and after obtaining the classification result, the classification result is evaluated through the classification evaluation index to verify the feasibility and superiority of the classification method.

8. A computer readable terminal storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the hyperspectral image classification method according to any one of claims 1 to 6.

Images