Hyperspectral image classification method and system based on hybrid convolutional neural network

Abstract

The application provides a hyperspectral image classification method and system based on a hybrid convolutional neural network, and belongs to the technical field of image processing; historical data sets are subjected to band selection, are divided in a patch form and are used as preliminary training samples; training samples are determined based on proportion setting; a hybrid convolutional neural network is trained, that is, a context interaction network is used to extract and fuse shallow layer features of images in the training samples to obtain a first feature map; the first feature map is input into a 3D-2D hybrid neural network for deep feature extraction; model parameters of the hybrid convolutional neural network in feature extraction are adaptively adjusted to determine optimal weights; a hyperspectral image to be classified is input into the hybrid convolutional neural network under the optimal weights for re-feature extraction to obtain a new feature map; and the new feature map is input into a softmax-fc classifier to realize classification of the hyperspectral image. The application can realize efficient and high-precision classification of hyperspectral images.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a hyperspectral image classification method and system based on a hybrid convolutional neural network. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Hyperspectral imaging is a new remote sensing technology based on imaging spectroscopy, which can simultaneously capture spectral information and surface information, and for the first time realizes the true combination of surface image information and spectral information. Hyperspectral image classification has many uses, for example: in agricultural research, it can accurately monitor crop conditions, especially in crop condition assessment, natural disaster monitoring and agricultural management; in addition, the application of hyperspectral imaging technology in geological mineral resources, lake and sea water quality monitoring, drought monitoring, national defense construction, surveying and mapping, military and many other important fields is also expanding. With the rapid rise and full popularization of computer internet technology, services and demands based on hyperspectral image information are more widely used in life, which not only provides information about the surrounding environment for individuals, but also helps enterprises obtain relevant information perception updates.

[0004] Recently, the application mode of "sky-ground deep" integration has been widely applied in agricultural production, natural resource management, urban planning and management, and land resource investigation, etc. More advanced hyperspectral image applications are rapidly emerging and greatly changing people's lives. For example, in the medical field, hyperspectral classification technology can be used for tissue composition analysis, tumor detection, etc., providing higher imaging contrast and resolution; for detecting water body, soil pollution, ecological system change, etc. Hyperspectral images can also help identify different pollutants and their distribution, providing data support for environmental protection and governance. Hyperspectral images have played a huge role in these fields. Existing urban areas use hyperspectral images for land use classification and infrastructure monitoring, assist urban planning and management decision-making, and investigate minerals and geology through hyperspectral images to help identify surface and underground mineral deposits and other geological features, etc. all rely on hyperspectral images.

[0005] However, the inventors have found that there are some technical problems in the existing methods for classifying hyperspectral images, for example:

[0006] (1) In the existing methods, support vector machines, random forests, multinomial logistic regression and other machine learning methods are generally used to classify hyperspectral images; and these methods mainly focus on the extraction of spectral features, so the spatial correlation and spatial locality in the hyperspectral images are ignored, which leads to low classification accuracy.

[0007] (2) As disclosed in the patent with patent application number 202211300601.5, a hyperspectral image classification method, which needs to rely on a large amount of sample data when analyzing the image, although a certain amount of data can be reduced by joint dimension reduction processing; however, the classification accuracy and the data amount after dimension reduction are still positively correlated, so the sample data amount is still high, resulting in low calculation efficiency during classification. SUMMARY

[0008] To overcome the shortcomings of the prior art, the present application provides a hyperspectral image classification method and system based on a hybrid convolutional neural network, which can realize efficient and high-precision classification of hyperspectral images.

[0009] To achieve the above-mentioned purpose, one or more embodiments of the present application provide the following technical solutions:

[0010] The present application provides a hyperspectral image classification method based on a hybrid convolutional neural network.

[0011] The hyperspectral image classification method based on the hybrid convolutional neural network comprises:

[0012] Obtain a historical data set, and use a PCA method to select bands for the historical data set;

[0013] Divide the historical data set after band selection in the form of a patch;

[0014] Use the divided historical data set as a preliminary training sample; based on a proportion setting, determine a training sample from the preliminary training sample;

[0015] Use the training sample to train a hybrid convolutional neural network, which comprises a context interaction network and a 3D-2D hybrid neural network; specifically, use the context interaction network to extract and fuse shallow features of the image in the training sample to obtain a first feature map; input the first feature map into the 3D-2D hybrid neural network to extract deep features to obtain a second feature map; and adaptively adjust the model parameters of the hybrid convolutional neural network during feature extraction to determine the optimal weight;

[0016] Input the hyperspectral image to be classified into the hybrid convolutional neural network under the optimal weight to re-extract features, to obtain a new feature map; and input the newly obtained feature map into a softmax-fc classifier to realize classification of the hyperspectral image.

[0017] Further, the PCA method is used to select bands for the historical data set, comprising:

[0018] Standardize data in the historical data set, and calculate the covariance matrix of the standardized data; select principal component eigenvectors according to the size of the eigenvalues; project the original data to the principal component eigenvectors based on the operation on the covariance matrix.

[0019] Further, the historical data set is divided in the form of a patch, including:

[0020] The historical data set and the label set are imported, the size of the patch is set, then the patch is extracted around each pixel and created in the format processed by keras, the surrounding pixels are filled with data by ZeroPad, and finally the divided data set and the label set are stored.

[0021] Further, the first feature map output by the context interaction network is a feature matrix, and the feature matrix is used to input the first layer of the 3D-2D hybrid neural network.

[0022] Further, the softmax-fc classifier connects each node to the output node and maps the high-dimensional features to specific classification results, then converts the output of the classification results into probabilities to obtain the probability distribution of each category.

[0023] Further, the hyperspectral image classification method based on the hybrid convolutional neural network further includes:

[0024] The classification results of the softmax-fc classifier are evaluated by using a confusion matrix, evaluation parameters and specific classification accuracy of each ground object, and if the classification result evaluation is not up to standard, the model parameters in the feature extraction process of the hybrid convolutional neural network are adjusted again.

[0025] Further, after obtaining the classification results of the softmax-fc classifier, a classification result report corresponding to the name of each ground object is generated and the classification results are visualized, different training sample ratios are set multiple times, and the accuracy fluctuation graph under different ratios is obtained for different training sample ratios.

[0026] The second aspect of the present application provides a hyperspectral image classification system based on a hybrid convolutional neural network.

[0027] The hyperspectral image classification system based on the hybrid convolutional neural network includes:

[0028] The band selection module is configured to: acquire a historical data set, and select bands of the historical data set by using a PCA method;

[0029] The data set division module is configured to: divide the historical data set after band selection in the form of a patch;

[0030] The training sample selection module is configured to: take the divided historical data set as a preliminary training sample; and determine a training sample from the preliminary training sample based on a proportion setting.

[0031] The model training module is configured to: train the mixed convolutional neural network using the training sample, the mixed convolutional neural network comprising a context interaction network and a 3D-2D mixed neural network; specifically, shallow feature extraction and fusion of images in the training sample are performed using the context interaction network to obtain a first feature map; the first feature map is input into the 3D-2D mixed neural network to perform deep feature extraction to obtain a second feature map; and model parameters of the mixed convolutional neural network in the feature extraction process are adaptively adjusted to determine optimal weights.

[0032] The image classification module is configured to: input a hyperspectral image to be classified into the mixed convolutional neural network under the optimal weights to perform feature extraction again to obtain a new feature map; and input the newly obtained feature map into a softmax-fc classifier to implement classification of the hyperspectral image.

[0033] The third aspect of the present application provides a computer readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps in the hyperspectral image classification method based on a mixed convolutional neural network according to the first aspect of the present application.

[0034] The fourth aspect of the present application provides an electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, the processor implementing the steps in the hyperspectral image classification method based on a mixed convolutional neural network according to the first aspect of the present application when executing the program.

[0035] The above one or more technical solutions have the following beneficial effects:

[0036] (1) The present application uses a mixed convolutional neural network to analyze and process image features, that is, shallow feature extraction and fusion of images in a training sample are performed using a context interaction network, and then a feature map composed of shallow features is taken as input of a 3D-2D mixed neural network, and the 3D-2D mixed neural network is used to further extract deep features from the feature map. Therefore, the present application progressively extracts features in a hyperspectral image by using two combined networks, taking into account the spatial correlation and spatial locality in the hyperspectral image, and has higher classification accuracy than the prior art.

[0037] (2) The application firstly retains the main information on the basis of dimension reduction of hyperspectral image data based on principal component analysis (PCA), including dimension information processing of data; at the same time, since the application can evaluate the classification result of the Softmax-fc classifier for multiple times by setting the proportion of training samples for multiple times, even if the sample quantity is extremely small, the optimal weight reaching balanced classification precision can be found. Therefore, the application can ensure the classification precision of the hyperspectral image with less sample data, and further improve the calculation efficiency during classification.

[0038] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the application and are incorporated herein by reference. The illustrations are shown for the purpose of enabling those skilled in the art to implement the application and are not intended to limit the scope of the application.

[0040] Figure 1 The flow chart of the hyperspectral image classification method based on the mixed convolutional neural network in the embodiment one of the application.

[0041] Figure 2 The flow chart of the method for evaluating the classification result of the Softmax-fc classifier in the embodiment one of the application. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0043] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application.

[0044] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0045] The overall idea provided by the present application is: in order to realize efficient and high-precision classification of different ground object remote sensing image data sets, the present application provides a hyperspectral image classification method based on a hybrid convolutional neural network, which realizes high-precision classification of different ground objects in a hyperspectral data set under a small sample condition by building a hybrid convolutional neural network model, that is, on the basis of fully utilizing the hundreds of spectral band information of the hyperspectral image, the image data set is processed; various ground object information in different data sets is classified by an algorithm, the accuracy of each type of information and the accuracy evaluation index of the entire data set are obtained, and finally the classification range visualization result graph of each ground object is displayed.

[0046] Embodiment one

[0047] The present embodiment discloses a hyperspectral image classification method based on a hybrid convolutional neural network.

[0048] As shown in Figure 1 The hyperspectral image classification method based on the hybrid convolutional neural network comprises:

[0049] Step S1, obtaining a historical data set, and selecting bands of the historical data set by using a PCA method;

[0050] Step S2, dividing the historical data set after band selection in a patch form;

[0051] Step S3, taking the divided historical data set as a preliminary training sample; based on a proportion setting, determining a training sample from the preliminary training sample;

[0052] Step S4, training the hybrid convolutional neural network by using the training sample, wherein the hybrid convolutional neural network comprises a context interaction network and a 3D-2D hybrid neural network; specifically, the context interaction network is used to extract and fuse shallow features of an image in the training sample to obtain a first feature map; the first feature map is input into the 3D-2D hybrid neural network to extract deep features to obtain a second feature map; and the model parameters of the hybrid convolutional neural network in the feature extraction process are adaptively adjusted to determine optimal weights;

[0053] Step S5, inputting a hyperspectral image to be classified into the hybrid convolutional neural network under the optimal weights to re-extract features, to obtain a new feature map; and inputting the newly obtained feature map into a softmax-fc classifier to realize classification of the hyperspectral image.

[0054] Based on the above process, efficient and high-precision classification of the hyperspectral image can be realized. In order to facilitate the understanding of the technical scheme of the present application, the above steps in the technical scheme of the present application are further explained and described.

[0055] Step S1, obtain a historical data set, and perform wave band selection on the historical data set by using a PCA method. Specifically, the following steps can be used to achieve the wave band selection:

[0056] Step S1-1, obtain a historical data set.

[0057] The historical data set used in this embodiment is a public data set, and such a public data set can be directly downloaded from a relevant website for use. Specifically, the historical data set downloaded in this embodiment is Indian Pines, which includes a ground truth data set Indian Pines (actual size 6.0 MB) and a label set Indian Pines groundtruth (actual size 1.1 KB) for classification. Since this does not involve the main technical point of the present application, it will not be described in detail.

[0058] Step S1-2, perform wave band selection on the historical data set by using a principal component analysis (PCA) method.

[0059] The data in the historical data set is standardized, and a covariance matrix of the data after the standardization is calculated; principal component eigenvectors are selected according to the sizes of eigenvalues; and the original data is projected to the principal component eigenvectors based on the operation on the covariance matrix.

[0060] Step S1-2-1, the data in the historical data set is standardized to ensure that each feature has the same scale; and a covariance matrix of the processed data is calculated to better evaluate the correlation between features.

[0061] First, the data in the historical data set is standardized so that the mean of each feature is 0 and the variance is 1. Specifically, the formula for standardization is as follows:

[0062]

[0063] where z represents the historical data set after the standardization, x represents the original observation value of a pixel at a wave band (feature), μ represents the mean, and σ represents the standard deviation.

[0064] Subsequently, the covariance matrix of the historical data set after the standardization is calculated, and the covariance matrix can reflect the correlation between features. Specifically, the covariance matrix is represented as follows:

[0065]

[0066] where C represents the covariance matrix, n represents the number of samples (each pixel can be regarded as a sample); X represents the standardized data matrix, X T is the transpose matrix of the data matrix X.

[0067] Step S1-2-2, according to the size of the eigenvalues of the covariance matrix, the eigenvectors corresponding to the first k eigenvalues are selected as principal components.

[0068] Step S1-2-3, by matrix calculation, the original data is projected onto the selected principal components, realizing dimension reduction of the data set.

[0069] First, matrix calculation is performed; specifically, the matrix calculation is realized by the following formula, that is:

[0070] Y = XW;

[0071] Where Y represents the reduced data, and W represents the matrix composed of the selected eigenvectors.

[0072] Subsequently, the original data is projected onto the selected principal components to realize the dimension reduction operation of the data set.

[0073] Step S2, the historical data set after band selection is divided in the form of patch.

[0074] Import the historical data set and label set; set the size of the patch; then extract the patch around each pixel and create the format for keras processing; fill the surrounding pixels with data through ZeroPad; finally, store the divided data set and label set.

[0075] Step S2-1, according to the actual use environment, build a suitable pytorch algorithm environment, and import the applied hyperspectral data set (historical data set) and label set through the loadmat function.

[0076] Step S2-2, in order to better process data information, set a suitable patch = w size, extract the patch around each pixel and create the format for keras processing, that is, take (w-1) / 2 width around the pixel centered at (i,j) for processing; fill the surrounding pixels with data through ZeroPad.

[0077] Due to the large amount of hyperspectral data, by step S2, the data can be more effectively processed and analyzed, and the training and testing are performed on smaller data sets, so that the algorithm calculation is more efficient. After dividing the data set into small blocks, local information can be extracted to better capture details; and the extracted local features help to improve the classification accuracy, speed up the model training time, and reduce the memory usage. Therefore, by step S2, the feature learning strength, model generalization ability and model robustness can be further enhanced.

[0078] Step S2-3, store the divided data set and label set.

[0079] It can be understood that the training set and the test set are respectively stored according to the proportion, so as to facilitate the subsequent training and test of the model.

[0080] Step S3, use the divided historical data set as a preliminary training sample; and determine a training sample from the preliminary training sample based on the proportion.

[0081] The preliminary training sample is divided into a training sample and a test sample, and the proportion of the training sample (training set) and the test sample (test set) in the preliminary training sample is set based on the proportion parameter setting formula test_ratio=p. For example, when the test parameter p=0.9 is set, 99% of the data set in the entire preliminary training sample is the training sample, and the other 1% of the entire preliminary training sample is the test sample. It can be understood that the proportion relationship between the training sample and the test sample can be adjusted according to actual needs, and the embodiment does not make specific limitations thereto.

[0082] Step S4, training the mixed convolutional neural network using the training sample; wherein the mixed convolutional neural network comprises a context interaction network and a 3D-2D mixed neural network; specifically, the context interaction network is used to extract and fuse shallow features of the image in the training sample to obtain a first feature map; the first feature map is input into the 3D-2D mixed neural network to extract deep features to obtain a second feature map; and the model parameters of the mixed convolutional neural network in the feature extraction process are adaptively adjusted to determine the optimal weight.

[0083] Step S4-1, set a 3D convolution layer of one layer (7, 3, 3), that is, the size of the convolution kernel of the convolution layer is 7, 3, and 3; set a suitable channel and perform regularization processing; at the same time, use Relu as the activation function;

[0084] Step S4-2, build the context interaction network through two layers (1, 1, 1) of 3D convolution layer.

[0085] Further, the 3D convolution layer is two layers, and the two layers of 3D convolution layer set in the embodiment are respectively named as the l layer and the l+1 layer; there is a parameter of the convolution kernel size in the convolution layer, and the 3D convolution layer has three parameters, the convolution kernel sizes of the two layers are set to (1, 1, 1), and the two layers are the context interaction network. Among them, the l layer is used to calculate the weighted sum when sliding at each position of the input.

[0086] The data processing flow of the context interaction network is that the input features pass through the lth layer and the l+1th layer, and the input data of the lth layer and the output data of the l+1th layer are added pixel by pixel. The output features obtained in step S4-1 are input into the context interaction network to obtain output features F1, and the output features F1 are input into the context interaction network again to obtain output features F2, so as to realize multiple feature information fusion and extraction.

[0087] The output features obtained in step S4-1 and the output features obtained in the lth layer are added pixel by pixel to fuse the features, and the fused data features are input into the 'l+1'th layer. Feature fusion is performed again in the 'l+1'th layer, that is, the output features obtained in the 'l'th layer and the output features obtained in the 'l+1'th layer are added pixel by pixel to fuse the features again.

[0088] Step S4-3, further depth feature extraction is performed by using a 3D-2D hybrid neural network.

[0089] A 3D convolution layer of one layer (5, 1, 1) and a 2D convolution layer of one layer (3, 3) are set to build a hybrid convolution module, that is, a 3D convolution layer of one layer with a convolution kernel size of 5, 3 and 3 in length, width and height, and a 2D convolution layer of one layer with a convolution kernel size of 3 and 3 in length and width are combined to obtain a 3D-2D hybrid neural network.

[0090] The feature matrix (first feature map) F2 output by the context interaction network is input into the 3D convolution layer of the 3D-2D hybrid neural network to obtain a feature matrix F3. The input and output feature channels are set appropriately; F3 is input into the 2D convolution layer of the hybrid neural network to obtain F4, so as to further extract the depth features of the spatial features of the hyperspectral image fusion superposition; the extracted features are subjected to regularization processing; at the same time, Relu is used as the activation function. Finally, the obtained two-dimensional feature matrix is unfolded to obtain a one-dimensional matrix.

[0091] Further, the two-dimensional features are unfolded to obtain a one-dimensional feature sequence by using a Flatten function, for example, a two-dimensional feature matrix of 3*3 is built:

[0092]

[0093] The above matrix is converted into one dimension, that is, [1, 2, 3, 4, 5, 6, 7, 8, 9].

[0094] Step S4-4, the model parameters of the hybrid convolution neural network in the feature extraction process are adaptively adjusted to determine the optimal weight.

[0095] Put the mixed convolutional neural network on the GPU, and set appropriate learning rate, loss function, optimization algorithm and other hyperparameters. Through the setting of epoch, the model is iterated on the training samples for multiple rounds; in each iteration, forward propagation, loss function calculation, back propagation and weight update operations are performed. Get the curve of the training accuracy and loss of the mixed convolutional neural network model with the number of iterations, further adjust the number of iterations of the mixed convolutional neural network model to ensure the optimization of the model weight.

[0096] Based on step S4, the classification method using the mixed convolutional neural network can automatically learn features, greatly reducing the work of manual labeling; it can be applied to multiple different hyperspectral data sets under the condition of few samples, and has better application universality compared with the prior art.

[0097] Step S5, input the hyperspectral image to be classified into the mixed convolutional neural network with optimal weight for feature extraction again to obtain new feature maps; and input the newly obtained feature maps into the softmax-fc classifier to realize classification of the hyperspectral image.

[0098] In the specific implementation process, the softmax-fc classifier connects each node to the output node, and maps the high-dimensional features to specific classification results; then, the output of the classification result is converted into a probability to obtain the probability distribution of each class, thereby realizing classification of the hyperspectral image.

[0099] Further, in order to further improve the classification accuracy of the mixed convolutional neural network for the hyperspectral image, the application further provides a method for evaluating the classification result of the mixed convolutional neural network. Through this method, it can be determined whether the classification accuracy of the mixed convolutional neural network reaches an ideal state; when it does not reach the ideal state, the model parameters are adjusted again to enhance the classification performance of the convolutional neural network. Therefore, even in the case of extremely small number of samples, the optimal weight reaching balanced classification accuracy can be found; therefore, the application can ensure the classification accuracy of the hyperspectral image with fewer sample data, thereby improving the calculation efficiency during classification.

[0100] As shown in Figure 2 The evaluation of the classification result of the mixed convolutional neural network can be realized through the following process:

[0101] The classification result of the softmax-fc classifier is evaluated by using the confusion matrix (Confusion Matrix), evaluation parameters (OA, AA, Kappa) and specific classification accuracy of each ground object, etc. If the classification result evaluation does not meet the standard, the model parameters in the feature extraction process or the classification process of the mixed convolutional neural network are adjusted again.

[0102] (1) Calculate the confusion matrix, evaluation parameters (OA, AA, Kappa) and the specific classification accuracy of each feature on the obtained results; correspond to the name of each type of feature and generate a specific classification result report. At the same time, the classification results of the classifier are classified pixel by pixel, and the labels of the pixels are predicted pixel by pixel by extracting the classification results to realize the visualization of the classification results; finally, generate a map and analyze its application value.

[0103] Further, the confusion matrix is mainly composed of four values: TP (True Positive), FP (False Positive), False Positive and FN (False Negative). These values are the results calculated by the model during the final test and saved. Specifically, TP means that the actual value and the predicted value are the same, FP means that the actual value is negative, but the model predicts that it is positive, TN means that the actual value and the predicted value have the same meaning, and FN means that the actual value is positive, but the model predicts that it is negative.

[0104] Further, the result report is generated by code to generate a txt report file, including OA, AA, Kappa and the evaluation parameters of each class.

[0105] Further, the classification result map is also generated by pixel-by-pixel prediction classification to generate a classification map; wherein the color represented by each type of classification can be determined according to the label map.

[0106] (2) By setting different classification samples, the fluctuation range of the evaluation indicators is compared to ensure the robustness of the algorithm.

[0107] (3) Multiple different training sample ratios are set; for different training sample ratios, repeat process (1) to obtain the accuracy fluctuation graph under different ratios, and compare to ensure the robustness of the model. It can be understood that the smaller the fluctuation shown by the accuracy fluctuation graph under different ratios, the stronger the robustness of the model.

[0108] The whole classification method and evaluation process of the embodiment meet the characteristics of high efficiency, high precision, less interference, rich information, strong application, etc., and can bring more accurate and reliable classification map or report to the user. The whole algorithm of the embodiment is built on a computer with a GPU graphics processing unit. When processing large-scale hyperspectral image data, it performs better and can use GPU and other hardware acceleration for training and inference, enhancing the ability to process large-scale data. At the same time, the adaptive matrix estimation (Adam) optimizer is used for updating, and the cross entropy loss function (CrossEntropy Loss) is used to calculate the loss function in the training process. The data processing process is carried out on Python.

[0109] Embodiment two

[0110] The embodiment discloses a hyperspectral image classification system based on a hybrid convolutional neural network.

[0111] The hyperspectral image classification system based on the hybrid convolutional neural network comprises:

[0112] The band selection module is configured to: acquire a historical data set, and perform band selection on the historical data set by using a PCA method;

[0113] The data set division module is configured to: divide the historical data set after band selection in a patch form;

[0114] The training sample selection module is configured to: take the divided historical data set as a preliminary training sample; and determine a training sample from the preliminary training sample based on a proportion setting;

[0115] The model training module is configured to: train the hybrid convolutional neural network by using the training sample, wherein the hybrid convolutional neural network comprises a context interaction network and a 3D-2D hybrid neural network; specifically, the context interaction network is used to perform shallow feature extraction and fusion on an image in the training sample, so as to obtain a first feature map; the first feature map is input into the 3D-2D hybrid neural network, so as to perform deep feature extraction, thereby obtaining a second feature map; and the model parameters of the hybrid convolutional neural network in the feature extraction process are adaptively adjusted, so as to determine optimal weights;

[0116] The image classification module is configured to: input a hyperspectral image to be classified into the hybrid convolutional neural network under the optimal weights to perform feature extraction again, so as to obtain a new feature map; and input the newly obtained feature map into a softmax-fc classifier, so as to realize classification of the hyperspectral image.

[0117] Embodiment three

[0118] An object of the embodiment is to provide a computer-readable storage medium.

[0119] A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps in the hyperspectral image classification method based on a hybrid convolutional neural network according to the embodiment one of the present disclosure.

[0120] Embodiment four

[0121] An object of the embodiment is to provide an electronic device.

[0122] An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the hyperspectral image classification method based on a hybrid convolutional neural network according to the embodiment one of the present disclosure when executing the program.

[0123] The steps involved in the devices of the above embodiments two, three and four correspond to the method embodiment one, and the specific embodiments can be seen from the relevant description part of the embodiment one. The term "computer-readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding or carrying a set of instructions for execution by a processor and causing the processor to perform any of the methods in the present disclosure.

[0124] Those skilled in the art should understand that each module or step of the present disclosure described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into each integrated circuit module, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present disclosure is not limited to any specific combination of hardware and software.

[0125] The above describes the specific embodiments of the present disclosure in combination with the accompanying drawings, but is not a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present disclosure without inventive labor are still within the protection scope of the present disclosure.

Claims

1. A hyperspectral image classification method based on a hybrid convolutional neural network, characterized in that, The method comprises the following steps: acquiring a historical data set and performing band selection on the historical data set by using a PCA method; dividing the historical data set after band selection in a patch form; taking the divided historical data set as a preliminary training sample; determining a training sample from the preliminary training sample based on a proportion setting; training a hybrid convolutional neural network by using the training sample, wherein the hybrid convolutional neural network comprises a context interaction network and a 3D-2D hybrid neural network; specifically, shallow feature extraction and fusion are performed on an image in the training sample by using the context interaction network to obtain a first feature map; the first feature map is input into the 3D-2D hybrid neural network to extract deep features to obtain a second feature map; and model parameters of the hybrid convolutional neural network in the feature extraction process are adaptively adjusted to determine optimal weights; re-performing feature extraction on a hyperspectral image to be classified by inputting the hyperspectral image into the hybrid convolutional neural network under the optimal weights to obtain a new feature map; and inputting the newly obtained feature map into a softmax-fc classifier to realize classification of the hyperspectral image. 2.The method of claim 1, wherein, The PCA method is used to perform band selection on the historical data set, which comprises the following steps: standardizing data in the historical data set and calculating a covariance matrix of the data after standardization; selecting principal component feature vectors according to eigenvalue sizes; and projecting original data to the principal component feature vectors based on operation on the covariance matrix. 3.The method of claim 1, wherein, The historical data set is divided in a patch form, which comprises the following steps: importing the historical data set and a label set; setting a patch size; then extracting a patch around each pixel and creating a format for keras processing; filling data around the pixel by using ZeroPad; and finally storing the divided data set and the label set. 4.The method of claim 1, wherein, The first feature map output by the context interaction network is a feature matrix, which is used to input a first layer of the 3D-2D hybrid neural network. 5.The method of claim 1, wherein, The softmax-fc classifier connects each node to an output node and maps high-dimensional features to specific classification results; then, the output of the classification results is converted into a probability to obtain a probability distribution of each class. 6.The method of claim 1, wherein, The method further comprises the following steps: evaluating the classification results of the softmax-fc classifier by using a confusion matrix, evaluation parameters and specific classification accuracies of each ground object; and if the classification result evaluation is not up to standard, adjusting the model parameters of the hybrid convolutional neural network in the feature extraction process again.

7. The method of claim 6, wherein the mixed convolutional neural network-based hyperspectral image classification method is characterized by, After obtaining the classification results of the softmax-fc classifier, a classification result report is generated for the name of each ground object and the classification results are visualized; different training sample proportions are set multiple times, and accuracy fluctuation graphs under different proportions are obtained for different training sample proportions.

8. A hyperspectral image classification system based on a hybrid convolutional neural network, characterized in that, The method comprises the following steps: a band selection module configured to acquire a historical data set and perform band selection on the historical data set by using a PCA method; a data set division module configured to divide the historical data set after band selection in a patch form; The training sample selection module is configured to take the divided historical data set as preliminary training samples; The training samples are determined from the preliminary training samples based on a proportion setting; The model training module is configured to train the hybrid convolutional neural network using the training samples, the hybrid convolutional neural network comprising a context interaction network and a 3D-2D hybrid neural network; specifically, the context interaction network is used to perform shallow feature extraction and fusion on the images in the training samples to obtain first feature maps; the first feature maps are input into the 3D-2D hybrid neural network to perform deep feature extraction to obtain second feature maps; and the model parameters of the hybrid convolutional neural network in the feature extraction process are adaptively adjusted to determine optimal weights; The image classification module is configured to input a hyperspectral image to be classified into the hybrid convolutional neural network under the optimal weights to perform feature extraction again to obtain new feature maps; and input the newly obtained feature maps into a softmax-fc classifier to realize classification of the hyperspectral image.

9. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps in the hyperspectral image classification method based on the hybrid convolutional neural network according to any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and capable of running on the processor, characterized by The processor executes the program to implement the steps in the hyperspectral image classification method based on the hybrid convolutional neural network according to any one of claims 1-7.

Images