A neural network hyperspectral image classification method based on multi-level space spectrum fusion

Abstract

The application belongs to the technical field of image processing, and discloses a neural network hyperspectral image classification method based on multi-level space spectrum fusion, which comprises the following steps: acquiring a hyperspectral image, extracting a required image to make a training data set and a test data set, performing normalization processing on all data sets, and using the training data set to perform neural network model training on data; and using the obtained neural network model to classify the test data set. The application introduces a multi-level feature extraction and fusion neural network, enhances the representation ability of the model, and improves the classification effect. The application belongs to the technical field of image processing, and comprises a multi-level space spectrum feature fusion hyperspectral image neural network classification technology.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a neural network hyperspectral image classification method based on multi-level spatial-spectral fusion. Background Technology

[0002] Hyperspectral images contain rich spectral and spatial information, which can be used for various tasks in the field of hyperspectral image analysis, such as image segmentation, object recognition, and anomaly detection. Hyperspectral image classification is one of the important fundamental research areas in hyperspectral image analysis and is an important means of conducting in-depth analysis of hyperspectral images. Hyperspectral image classification methods can generally be divided into two main categories. One category is traditional methods based on manual feature extraction, such as support vector machines, decision trees, minimum distance classification, maximum likelihood classification, spectral angle classification, and hybrid distance classification. The other category is neural network methods based on automatic feature extraction, mainly represented by convolutional neural networks, which have a strong ability to extract features.

[0003] In existing technologies, Bor-Chen Kuo et al. proposed a kernel-based feature selection method and applied it to a support vector machine (SVM) classification model based on radial basis function (RBF) kernels. In this model, the separability of the feature space of the RBF kernel is measured by a criterion that includes both inter-class and intra-class information. Onuwa Okwuashi et al. proposed a deep SVM classification method that combines the characteristics of SVMs and deep neural networks, using a standalone SVM as an internal connection of the deep neural network. All of the above methods train classifiers to obtain spectral information from hyperspectral images based on manually designed feature extraction methods, without effectively utilizing the spatial information of hyperspectral images. They do not effectively extract spectral features.

[0004] In recent years, deep learning has achieved remarkable results in the field of computer vision, and an increasing number of deep learning models are being applied to hyperspectral image processing. Shiqi Yu et al. proposed a convolutional neural network with three 1×1 convolutional layers, adding a global average pooling layer at the end to process the spatial and spectral information of hyperspectral images. Jingxiang Yang et al. proposed a deep convolutional neural network with a dual-branch structure to extract joint spectral-spatial features from hyperspectral images. The two branches of this network are dedicated to extracting features from the spectral and spatial domains. The extracted spectral and spatial features are then concatenated to extract joint spectral-spatial features, which are then input into a fully connected layer for classification. These methods separate the processing of spatial and spectral information, failing to consider the correlation between spatial and spectral information hierarchically. This weakens the representational power of the extracted fused features, thus affecting classification accuracy and the generalization ability of the subsequent models. Summary of the Invention

[0005] This invention provides a neural network-based hyperspectral image classification method based on multi-level spatial-spectral fusion to address the technical problems of low classification accuracy caused by the lack of comprehensive and effective utilization of spatial and spectral information and the need for manual feature extraction in existing technologies. The content of this invention is as follows:

[0006] The purpose of this invention is to provide a neural network-based hyperspectral image classification method based on multi-level spatial-spectral fusion, the technical points of which include the following steps:

[0007] Step 1: Acquire hyperspectral images from the hyperspectral camera and preprocess them. Based on the classification information, extract different categories of hyperspectral images from the preprocessed hyperspectral images and divide each category of hyperspectral images into training datasets and test datasets.

[0008] Step 2: Normalize both the training and test datasets obtained in Step 1 so that the values ​​in all data are normalized to the range [0,1].

[0009] Step 3: Build a neural network model. Use the training dataset after normalization in Step 2 to train the neural network model and obtain the trained neural network model. The neural network model includes an input module, three spectral feature extraction modules, one spatial feature extraction module, two spatial pooling modules, two feature fusion modules, and one classification module.

[0010] Step four: Input the normalized test dataset from step two into the trained neural network model from step three to obtain the classification result.

[0011] In some embodiments of the present invention, the hyperspectral image preprocessing method in step one of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion is as follows: the red, green and blue channel images of the hyperspectral image are fused to obtain an RGB image, while the remaining spectral channel images remain unchanged, and the RGB image is manually labeled.

[0012] In some embodiments of the present invention, the normalization formula in step two of the above-described neural network hyperspectral image classification method based on multi-level spatial-spectral fusion is: , where X （i，j） It is the value of the pixel in the i-th row and j-th column of the current image X, Min(X) is the minimum pixel value in image X, and Max(X) is the maximum pixel value in image X;

[0013] In some embodiments of the present invention, the neural network model training step in step three of the above-described neural network hyperspectral image classification method based on multi-level spatial-spectral fusion is as follows:

[0014] Step 1: The training dataset after normalization in Step 2 is input into the first spectral feature extraction module and the spatial feature extraction module to perform the first spectral feature extraction and spatial feature extraction to obtain feature map A and feature map B;

[0015] Step 2: Input the feature map A from Step 1 into the first spatial pooling module and the second spectral feature extraction module to perform the first pooling operation and the second spectral feature extraction to obtain feature map C and feature map D;

[0016] Step 3: Input feature map B from step 1 and feature map C from step 2 into the first feature fusion module for the first feature fusion and further feature extraction to obtain feature map E;

[0017] Step 4: Input the feature map D from Step 2 into the second spatial pooling module and the third spectral feature extraction module to perform a second pooling operation and a third spectral feature extraction to obtain feature map F and feature map G;

[0018] Step 5: Input the feature map F from step 4 and the feature map E from step 3 into the second feature fusion module for a second feature fusion and further feature extraction to obtain feature map H;

[0019] Step 6: Input the feature map G from step 4 and the feature map H from step 5 into the classification module for processing to obtain the trained neural network model.

[0020] In some embodiments of the present invention, the structure of the first spectral feature extraction module in the neural network model training step one of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion is: convolutional layer -> batch processing -> activation function -> pooling layer -> convolutional layer -> batch processing -> activation function.

[0021] In some embodiments of the present invention, the first spatial pooling module in step two and the second spatial pooling module in step four of the neural network model training step of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion have the same structure: pooling layer -> convolutional layer -> batch processing -> activation function.

[0022] In some embodiments of the present invention, the spatial feature extraction module structure in the first step of the neural network model training of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion is: convolutional layer -> batch processing -> activation function -> pooling layer.

[0023] In some embodiments of the present invention, the first feature fusion module in step three and the second feature fusion module in step five of the neural network model training step of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion have the same structure.

[0024] In some embodiments of the present invention, the second spectral feature extraction module in step two and the third spectral feature extraction module in step four of the neural network model training step of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion have the same structure: pooling layer -> convolutional layer -> batch processing -> activation function.

[0025] In some embodiments of the present invention, the structure of the classification module in step six of the neural network model training step of the above-mentioned neural network hyperspectral image classification method based on multi-level spatial-spectral fusion is as follows: feature fusion layer -> fully connected layer -> batch processing -> activation function -> fully connected layer -> softmax layer.

[0026] Compared with existing technologies, the neural network hyperspectral image classification method based on multi-level spatial-spectral fusion of the present invention can achieve the following beneficial effects:

[0027] 1. The neural network hyperspectral image classification method based on multi-level spatial-spectral fusion of the present invention performs multiple extraction and fusion processes on the spatial and spectral information of the input hyperspectral image, making the resulting network model more efficient in its representation capabilities. It also overcomes the shortcomings of existing methods in extracting and fusing features of spatial and spectral information of hyperspectral images, enabling the present invention to extract feature information from multi-level spectral spatial fusion.

[0028] 2. The neural network model in the hyperspectral image classification method based on multi-level spatial-spectral fusion of this invention is a convolutional neural network model. This invention utilizes convolutional layers, batch processing, and activation functions in the model to implement a self-learning automatic feature extraction method, which can effectively extract features from hyperspectral images. Since feature extraction is a fully automated process and is continuously optimized during training, it overcomes the limitations of traditional methods caused by human factors in feature extraction, giving this invention the advantage of higher universality. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0030] Figure 1 This is a flowchart of the neural network hyperspectral image classification method based on multi-level spatial-spectral fusion of the present invention;

[0031] Figure 2 This is a flowchart of the neural network model training process of the present invention;

[0032] Figure 3The image shows the results of classifying fungal types on wood using the neural network hyperspectral image classification method based on multi-level spatial-spectral fusion according to the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 The neural network-based hyperspectral image classification method based on multi-level spatial-spectral fusion, as shown, includes the following steps:

[0036] Step 1: Acquire hyperspectral images from a hyperspectral camera and preprocess them. Preferably, the preprocessing method of this invention is as follows: the red, green, and blue channel images of the hyperspectral image are fused to obtain an RGB image, while the remaining spectral channel images remain unchanged, and the RGB image is manually labeled. Based on classification information, four categories of hyperspectral images (e.g., ...) are extracted from the preprocessed hyperspectral image. Figure 3 The four types shown are: Clear wood; Softrot; Brown stain; Blue stain. Based on the extracted target image data, each type of hyperspectral image is divided into a training dataset and a test dataset. Preferably, each type of hyperspectral image in this invention is cropped into a 32×32×320 data block.

[0037] Step two: Normalize both the training and test datasets obtained in step one, so that the values ​​in all data are normalized to the range [0,1]. Preferably, the normalization formula of this invention is: , where X （i，j） It is the value of the pixel in the i-th row and j-th column of the current image X. Min(X) is the minimum pixel value in image X, and Max(X) is the maximum pixel value in image X.

[0038] Step 3: Establish a neural network model. Train the neural network model using the normalized training dataset from Step 2 to obtain the trained neural network model. The neural network model includes an input module, three spectral feature extraction modules, one spatial feature extraction module, two spatial pooling modules, two feature fusion modules, and one classification module, as follows: Figure 2 The steps for training the neural network model of the present invention are as follows:

[0039] Step 1: The normalized training dataset from Step 2 is input into the first spectral feature extraction module and the spatial feature extraction module for the first spectral feature extraction and spatial feature extraction to obtain feature map A and feature map B. Preferably, the structure of the first spectral feature extraction module of this invention is: convolutional layer -> batch processing -> activation function -> pooling layer -> convolutional layer -> batch processing -> activation function; the convolutional kernel size of each convolutional layer is 1×1, the activation function is ReLU, and the pooling layer is 1×1×4 max pooling. The structure of the spatial feature extraction module of this invention is: convolutional layer -> batch processing -> activation function -> pooling layer; the convolutional kernel size of each convolutional layer is 3×3×320, the activation function is ReLU, and the pooling layer is 2×2×1 max pooling.

[0040] Step 2: Input feature map A from Step 1 into the first spatial pooling module and the second spectral feature extraction module for the first pooling operation and the second spectral feature extraction to obtain feature map C and feature map D. Preferably, the structure of the first spatial pooling module of the present invention is: pooling layer -> convolutional layer -> batch processing -> activation function; the downsampling sizes of the pooling layers are 2×2×1 and 4×4×1, the kernel sizes of the convolutional layers are 3×3×80 and 3×3×20, and the activation function is ReLU. Preferably, the structure of the second spectral feature extraction module of the present invention is: pooling layer -> convolutional layer -> batch processing -> activation function; the kernel sizes of the convolutional layers are 1×1×3 and 32×32×5, the activation function is ReLU, and the pooling layers are all 1×1×4 max pooling.

[0041] Step 3: Input feature map B from step 1 and feature map C from step 2 into the first feature fusion module for the first feature fusion and further feature extraction to obtain feature map E. Preferably, the structure of the first feature fusion module is: feature fusion layer -> convolutional layer -> batch processing -> activation function; the convolutional kernel sizes of the convolutional layers are 3×3×1 and 8×8×1 respectively, and the activation function is the ReLU function.

[0042] Step 4: Input the feature map D from Step 2 into the second spatial pooling module and the third spectral feature extraction module to perform a second pooling operation and a third spectral feature extraction to obtain feature map F and feature map G. The second spatial pooling module of the present invention has the same structure as the first spatial pooling module in Step 2; the third spectral feature extraction module of the present invention has the same structure as the second spectral feature extraction module in Step 2.

[0043] Step 5: Input the feature map F from step 4 and the feature map E from step 3 into the second feature fusion module for a second feature fusion and further feature extraction to obtain feature map H. The second feature fusion module of the present invention has the same structure as the first feature fusion module in step 3.

[0044] Step 6: Input the feature map G from step 4 and the feature map H from step 5 into the classification module for processing to obtain the trained neural network model. Preferably, the structure of the classification module of the present invention is: feature fusion layer -> fully connected layer -> batch processing -> activation function -> fully connected layer -> softmax layer; the outputs of the fully connected layer are 128 and 4 respectively, and the extracted features of the present invention are fused and connected to two fully connected layers and a softmax layer.

[0045] Step four: Input the normalized test dataset from step two into the trained neural network model from step three to obtain the classification result.

[0046] Simulation Experiment

[0047] The effects of this invention can be further illustrated by the following simulation experiments:

[0048] 1. Simulation conditions:

[0049] Hardware: Intel Core CPU i9-9700k@4.9GHz

[0050] Graphics card: Geforce 2080Ti / 11Gb

[0051] Memory: 32Gb

[0052] The software platform is: MATLAB

[0053] 2. Simulation content and results:

[0054] Experiments were conducted using the method of this invention under the aforementioned simulation conditions. Specifically, 10% of the data blocks from each category of the hyperspectral data were randomly selected as training samples, and the remainder were used as test samples, resulting in the following... Figure 3 The classification results are as follows: Ground truth represents the true value, Clear wood represents unaffected wood areas, Soft rot represents wood areas with soft rot, Brown stain represents wood areas with browning, and Blue stain represents wood areas with blue staining.

[0055] Table 1 Comparison of classification accuracy obtained in simulation using existing technologies

[0056] The method of the present invention 93.41% Support Vector Machine-based methods 90.27% Methods based on convolutional neural networks 91.47%

[0057] As shown in Table 1, for hyperspectral data of wood, the feature extraction accuracy of this invention using this method reaches 93.41%, which is higher than existing traditional methods and neural network methods. The classification accuracy is also improved.

[0058] In summary, this invention introduces feature extraction and fusion of multi-level spatial and spectral information, which effectively improves the image representation capability, enabling the model to learn more discriminative hyperspectral image features and achieve more accurate classification results compared to existing technologies.

[0059] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A neural network-based hyperspectral image classification method based on multi-level spatial-spectral fusion, characterized in that, Includes the following steps: Step 1: Acquire hyperspectral images from the hyperspectral camera and preprocess them. Based on the classification information, extract different categories of hyperspectral images from the preprocessed hyperspectral images and divide each category of hyperspectral images into training datasets and test datasets. The hyperspectral image preprocessing method is as follows: the red, green and blue channel images of the hyperspectral image are fused to obtain an RGB image, while the other spectral channel images remain unchanged, and the RGB image is manually labeled. Step 2: Normalize both the training and test datasets obtained in Step 1 so that the values ​​in all data are normalized to the range [0,1]. The formula for normalization is: , where X （i，j） It is the value of the pixel in the i-th row and j-th column of the current image X, Min(X) is the minimum pixel value in image X, and Max(X) is the maximum pixel value in image X; Step 3: Build a neural network model. Use the training dataset after normalization in Step 2 to train the neural network model and obtain the trained neural network model. The neural network model includes an input module, three spectral feature extraction modules, one spatial feature extraction module, two spatial pooling modules, two feature fusion modules, and one classification module. The steps for training the neural network model in step three are as follows: Step 1: The normalized training dataset from Step 2 is fed into the first spectral feature extraction module and the spatial feature extraction module through the input module to perform the first spectral feature extraction and spatial feature extraction to obtain feature map A and feature map B; The structure of the first spectral feature extraction module is: convolutional layer -> batch processing -> activation function -> pooling layer -> convolutional layer -> batch processing -> activation function; Step 2: Input the feature map A from Step 1 into the first spatial pooling module and the second spectral feature extraction module to perform the first pooling operation and the second spectral feature extraction to obtain feature map C and feature map D; The first spatial pooling module and the second spatial pooling module in Step 4 have the same structure: pooling layer -> convolutional layer -> batch processing -> activation function; Step 3: Input feature map B from step 1 and feature map C from step 2 into the first feature fusion module for the first feature fusion and further feature extraction to obtain feature map E; Step 4: Input the feature map D from Step 2 into the second spatial pooling module and the third spectral feature extraction module to perform a second pooling operation and a third spectral feature extraction to obtain feature map F and feature map G; Step 5: Input the feature map F from step 4 and the feature map E from step 3 into the second feature fusion module for a second feature fusion and further feature extraction to obtain feature map H; Step 6: Input the feature map G from step 4 and the feature map H from step 5 into the classification module for processing to obtain the trained neural network model; Step four: Input the normalized test dataset from step two into the trained neural network model from step three to obtain the classification result.

2. The neural network hyperspectral image classification method based on multi-level spatial-spectral fusion according to claim 1, characterized in that, The spatial feature extraction module in step 1 has the following structure: convolutional layer -> batch processing -> activation function -> pooling layer.

3. The neural network hyperspectral image classification method based on multi-level spatial-spectral fusion according to claim 1, characterized in that, The first feature fusion module in step 3 and the second feature fusion module in step 5 have the same structure: feature fusion layer -> convolutional layer -> batch processing -> activation function.

4. The neural network hyperspectral image classification method based on multi-level spatial-spectral fusion according to claim 1, characterized in that, The second spectral feature extraction module in step 2 and the third spectral feature extraction module in step 4 have the same structure: pooling layer -> convolutional layer -> batch processing -> activation function.

5. The neural network hyperspectral image classification method based on multi-level spatial-spectral fusion according to claim 1, characterized in that, The structure of the classification module in step 6 is as follows: feature fusion layer -> fully connected layer -> batch processing -> activation function -> fully connected layer -> softmax layer.

Images