A hyperspectral image classification method based on deep neural network

Abstract

The present invention discloses a hyperspectral image classification method based on a deep neural network, comprising: inputting hyperspectral data to be classified into a target deep neural network to obtain a classification result; the target deep neural network training process includes: using labeled hyperspectral training data to train an initial deep neural network to obtain an intermediate deep neural network, the initial deep neural network including a plurality of basic residual network modules and an output module connected in sequence; using a collaborative selection strategy and a competitive selection strategy to select some unlabeled hyperspectral data from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data, and generating new labeled hyperspectral training data from the labeled hyperspectral training data and the pseudo-labeled hyperspectral training data; and using the new labeled hyperspectral training data to train the intermediate deep neural network until a training stop condition is met to obtain a target deep neural network. The present invention improves the classification performance of hyperspectral images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a hyperspectral image classification method based on a deep neural network. Background Art

[0002] With advances in imaging, sensor technology, and aerospace technology, remote sensing technology has rapidly developed. In the 1980s, remote sensing technology entered the hyperspectral remote sensing stage. The significant advantages of hyperspectral remote sensing technology have played a significant role in the research and analysis of remote sensing data, making the task of ground feature classification much simpler.

[0003] In hyperspectral images, individual pixels contain a wealth of spectral information. Therefore, the high-dimensional spectral features of the original image space are crucial for object classification. With the advancement of remote sensing technology, the spatial resolution of hyperspectral images has become increasingly higher, and the spatial information they contain has gradually increased, providing an important foundation for more accurate classification and discrimination. In recent years, deep neural networks, due to their powerful feature extraction and classification capabilities, have been widely used in hyperspectral image object classification research, achieving excellent classification performance. In 2018, Paoletti et al. proposed a novel fast deep convolutional neural network that leverages the spatial spectral information of hyperspectral images. In the same year, Juan et al. proposed a novel three-step training phase, performing active learning on the proposed dropout-based convolutional neural network for the first time. To overcome the gradient explosion and gradient divergence problems associated with training very deep CNNs using hyperspectral image data, Zhong et al. introduced residual neural networks (ResNets) to the hyperspectral image classification task in 2018. By applying skip connections, ResNets demonstrate powerful capabilities for learning spatial spectral features for hyperspectral image classification.

[0004] However, although current hyperspectral remote sensing image classification algorithms widely use deep learning networks to learn raw features, further improving classification accuracy, there are still some unlabeled samples that cannot be classified. Since labeling datasets in the field of hyperspectral remote sensing images usually requires a lot of manpower and material resources, even so, it is difficult to ensure the accuracy of sample labeling, resulting in a small number of high-confidence labeled samples available in hyperspectral images. Therefore, large-scale deep learning networks can easily overfit to the labeled sample set and cannot generalize well to other samples. In addition, the network lacks guided optimization for feature extraction of hyperspectral remote sensing images. Therefore, traditional hyperspectral image classification methods based on deep learning networks find it difficult to obtain a better deep learning network using limited labeled samples, which in turn affects the performance of hyperspectral image classification. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a hyperspectral image classification method based on deep neural network. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a method for hyperspectral image classification based on a deep neural network, comprising:

[0007] Obtain hyperspectral data to be classified;

[0008] Inputting the hyperspectral data to be classified into a target deep neural network to obtain a classification result;

[0009] The target deep neural network is trained based on labeled hyperspectral training data and unlabeled hyperspectral data; the training process includes:

[0010] The labeled hyperspectral training data is used to train an initial deep neural network to obtain an intermediate deep neural network, where the initial deep neural network includes a plurality of basic residual network modules and an output module connected in sequence; a part of the unlabeled hyperspectral data is selected from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data using a collaborative selection strategy and a competitive selection strategy, and new labeled hyperspectral training data is generated from the labeled hyperspectral training data and the pseudo-labeled hyperspectral training data; the intermediate deep neural network is trained using the new labeled hyperspectral training data until a training stop condition is met to obtain the target deep neural network.

[0011] In one embodiment of the present invention, the basic residual network module is designed as a bottleneck residual network module.

[0012] In one embodiment of the present invention, before using the labeled hyperspectral training data to train the initial deep neural network, the method includes:

[0013] The Kaiming initialization method is used to initialize the network parameters corresponding to the initial deep neural network.

[0014] In one embodiment of the present invention, the process of training a deep neural network using labeled hyperspectral training data includes:

[0015] Extracting a three-dimensional cube label sample block corresponding to each sample in the label hyperspectral training data using a first neighborhood window;

[0016] All three-dimensional cube label sample blocks are divided into batches, and the three-dimensional cube label sample blocks in the divided batches are used to train the deep neural network.

[0017] In one embodiment of the present invention, the process of training a deep neural network using the divided batches of three-dimensional cube labeled sample blocks includes:

[0018] Based on the local preservation constraint strategy, a local preservation constraint function is constructed according to the batch of 3D cube label sample blocks after division;

[0019] Construct a cross entropy loss optimization function for a deep neural network based on the divided batches of three-dimensional cube label sample blocks;

[0020] Constructing a target loss optimization function of a deep neural network based on a local preservation constraint strategy according to the local preservation constraint function and the cross entropy loss function;

[0021] The target loss optimization function is solved for an optimal solution to obtain an intermediate deep neural network corresponding to the optimal network parameters.

[0022] In one embodiment of the present invention, the target loss optimization function formula of the deep neural network constructed based on the local preservation constraint strategy is expressed as:

[0023] L(X tr ,θ)=L base (X tr ,θ)+λL em (X tr ,θ);

[0024] Among them, X tr represents the batch of three-dimensional cube label sample blocks after division, θ represents the network parameters of the deep neural network, L(X tr ,θ) represents the target loss optimization function, L base (X tr ,θ) represents the cross entropy loss optimization function, L em (X tr ,θ) represents the locality preserving constraint function, and λ represents the trade-off parameter.

[0025] In one embodiment of the present invention, the selecting of part of the unlabeled hyperspectral data from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data using the collaborative selection strategy and the competitive selection strategy includes:

[0026] Constructing a pseudo-label map of the unlabeled hyperspectral training data according to the classification results of the intermediate deep neural network;

[0027] Extracting a pseudo label mapping matrix corresponding to each sample in the unlabeled hyperspectral data from the pseudo label mapping map using a second neighborhood window;

[0028] Using a neighborhood consistency criterion, samples corresponding to high-confidence three-dimensional cube unlabeled sample blocks are selected from the pseudo-label mapping matrix to form an available candidate pseudo-label dataset;

[0029] Constructing a label probability matrix for the unlabeled hyperspectral training data according to the classification results of the intermediate deep neural network;

[0030] Available candidate pseudo-label data with high probability are selected from the available candidate pseudo-label data set according to the label probability matrix as pseudo-label hyperspectral training data.

[0031] In one embodiment of the present invention, the training process further includes:

[0032] Adding a basic residual network module between the output module of the intermediate deep neural network and the last basic residual network module to form a new intermediate deep neural network;

[0033] The intermediate deep neural network is trained using the new labeled hyperspectral training data until a training stop condition is satisfied to obtain the target deep neural network, including:

[0034] The new intermediate deep neural network is trained using the new labeled hyperspectral training data until the training stop condition is met to obtain the target deep neural network.

[0035] In one embodiment of the present invention, the network structure of the basic residual network module added between the output module of the intermediate deep neural network and the last basic residual network module is the same as the network structure of the last basic residual network module.

[0036] In one embodiment of the present invention, when a new intermediate deep neural network is trained using new label training hyperspectral data, the network parameters corresponding to the intermediate deep neural network at the end of the last training are used as the initialization network parameters of the new intermediate deep neural network; wherein the network parameters of the added basic residual network module are the same as the network parameters of the last basic residual network module.

[0037] Beneficial effects of the present invention:

[0038] The hyperspectral image classification method based on deep neural network proposed in the present invention aims to solve the problem that the existing data set has limited label data, which makes it difficult for the hyperspectral image classification method based on deep learning network to obtain a better deep learning network using limited label data, thereby affecting the hyperspectral image classification performance. During the training process, a strategy for expanding labeled hyperspectral training data is proposed, which uses a collaborative selection strategy and a competitive selection strategy to collaboratively analyze the confidence evaluation method, and considers the assistance between adjacent samples and the competitiveness between similar samples to obtain high-quality unlabeled hyperspectral data to expand the labeled hyperspectral training data, thereby providing more available recognition training data for training, so as to train a better deep neural network and improve the classification performance of the hyperspectral data to be classified.

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 1 is a flow chart of a hyperspectral image classification method based on a deep neural network provided by an embodiment of the present invention;

[0041] Figure 2 1 is a flowchart of the training process of the deep neural network model provided by an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of the structure of a deep neural network model provided by an embodiment of the present invention;

[0043] Figure 4 Schematic diagram of a framework for generating pseudo-labeled hyperspectral training data using collaborative selection strategy and competitive selection strategy provided by an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of a hyperspectral image classification framework based on a deep neural network provided by an embodiment of the present invention;

[0045] Figure 6 Schematic diagram of a framework of a deep neural network based on a local preservation constraint strategy provided by an embodiment of the present invention;

[0046] Figure 7 (a)~ Figure 7 (b) is a schematic diagram of the original hyperspectral image and real ground feature map corresponding to the Indian Pines dataset provided by an embodiment of the present invention;

[0047] Figure 8 (a)~ Figure 8 (b) is a schematic diagram of the original hyperspectral image and real ground feature map taken corresponding to the PaviaU dataset provided by an embodiment of the present invention;

[0048] Figure 9 (a)~ Figure 9 (b) is a schematic diagram of the original hyperspectral image and real ground feature map corresponding to the Salinas dataset provided by an embodiment of the present invention;

[0049] Figure 10 (a)~ Figure 10 (f) is a schematic diagram of mapping the classification results of the method of the present invention, the SVM method, the RF method, the 1D-CNN method, the 2D-CNN method, and the 3D-CNN method on the Indian Pines dataset provided by an embodiment of the present invention;

[0050] Figure 11 (a)~ Figure 11 (f) is a schematic diagram of mapping the classification results of the method of the present invention provided by an embodiment of the present invention and the SVM method, RF method, 1D-CNN method, 2D-CNN method, and 3D-CNN method on the PaviaU dataset;

[0051] Figure 12 (a)~ Figure 12 (f) is a schematic diagram of mapping the classification results of the method of the present invention, the SVM method, the RF method, the 1D-CNN method, the 2D-CNN method, and the 3D-CNN method on the Salinas dataset provided by an embodiment of the present invention;

[0052] Figure 13 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0054] Example 1

[0055] To improve the classification performance of hyperspectral images, see Figure 1 The present invention provides a method for hyperspectral image classification based on a deep neural network, which specifically includes the following steps:

[0056] S10, obtaining hyperspectral data to be classified;

[0057] S20: Input the hyperspectral data to be classified into the target deep neural network to obtain the classification result.

[0058] Due to the limited label information of hyperspectral images, deep networks are difficult to train, which in turn affects the classification performance of hyperspectral images. To address this problem, the present invention proposes a strategy for optimizing network parameters by enriching labeled sample data and mining unlabeled sample data during deep neural network training, thereby better achieving hyperspectral image classification. Specifically:

[0059] The target deep neural network of the embodiment of the present invention is trained based on labeled hyperspectral training data and unlabeled hyperspectral data. There is less labeled hyperspectral training data and more unlabeled hyperspectral data. Figure 2 , the training process includes the following steps:

[0060] S201. Using labeled hyperspectral training data, an initial deep neural network is trained to obtain an intermediate deep neural network. The initial deep neural network includes a plurality of basic residual network modules and an output module connected in sequence.

[0061] Specifically, see Figure 3 The embodiment of the present invention first designs a lightweight deep neural network Net0 comprising a number of sequentially connected basic residual network modules and an output module, and uses the lightweight deep neural network Net0 as the initial deep neural network of the embodiment of the present invention. The lightweight deep neural network Net0 is mainly used to obtain shallow mapping features, which contain more position information and can obtain more local spatial information. Each basic residual network module includes a sequentially connected convolution layer, a batch normalization layer, an activation layer, and a downsampling layer, and the output module includes a sequentially connected pooling layer and a fully connected layer.

[0062] If the number of basic residual network modules in the designed lightweight deep neural network Net0 is large, the network parameters will also increase accordingly. Randomly initializing a large number of parameters will to some extent affect the network's feature extraction ability. Therefore, the embodiment of the present invention preferably uses the Kaiming initialization method to initialize the network parameters corresponding to the initial deep neural network.

[0063] For labeled hyperspectral training data D tr The process of training a deep neural network using labeled hyperspectral training data includes: using the first neighborhood window to extract the three-dimensional cube label sample block corresponding to each sample in the labeled hyperspectral training data; dividing all the three-dimensional cube label sample blocks into batches, and using the divided three-dimensional cube label sample blocks to train the deep neural network.

[0064] Assuming hyperspectral image Where D represents the spectral band, W and H represent the width and height of a single band image respectively. y∈{1,2,…,M} represents the category label of the hyperspectral image D, where M represents the total number of category labels of the hyperspectral image, and y i Indicates the label value belonging to the i-th category. In sample extraction, in order to obtain accurate hyperspectral classification results, for each sample (pixel point) in the labeled hyperspectral training data, a first neighborhood window centered on the sample is constructed, and the first neighborhood window is used to extract the three-dimensional cube label sample block corresponding to each sample in the labeled hyperspectral training data, and the label of each three-dimensional cube label sample block is set to the label corresponding to the sample at the center. In order to better fit the situation where the actual scene labels are limited, a small number of samples are selected as the labeled hyperspectral training data D for different hyperspectral images D. tr , the remaining samples are taken as unlabeled hyperspectral data D te , used to test the classification performance of the proposed method and subsequently generate pseudo-labeled hyperspectral training data. Finally, these 3D cube labeled sample blocks are input into the designed deep neural network for training. The first neighborhood window can be flexibly designed based on actual needs, for example, a 3×3 window or a 4×4 window.

[0065] To further improve the training speed of the deep neural network, the embodiment of the present invention divides the three-dimensional hyperspectral data cube into batches, preferably into equal batches, such as a batch size of 200, and inputs the divided batches of three-dimensional cube label sample blocks into the deep neural network for training.

[0066] Furthermore, during the training process of the embodiment of the present invention, a cross entropy loss optimization function corresponding to the initial deep neural network Net0 is designed, and the formula is expressed as:

[0067]

[0068] Among them, X tr represents the batch of three-dimensional cube label sample blocks after division, θ represents the network parameters of the deep neural network, and p is , Represents X tr The true probability distribution value and predicted probability distribution value of the sample corresponding to the i-th three-dimensional cube unlabeled sample block on the s-th category, and the sample corresponding to the i-th three-dimensional cube labeled sample block is recorded as x i , N tr represents the number of training samples corresponding to the three-dimensional cube label sample block, and M represents the total number of category labels.

[0069] Then, the Adam model optimizer is used to optimize the cross entropy loss optimization function of formula (1) to solve the optimal network parameters of the initial deep neural network. The network structure corresponding to the optimal network parameters is the corresponding intermediate deep neural network. For different training data sets, the Adam learning rate is set to a value in the range of 0.00001 to 0.001. Among them, network training is not limited to the Adam model optimizer, and other model optimizers can also be used.

[0070] S202 , using a collaborative selection strategy and a competitive selection strategy to select part of the unlabeled hyperspectral data from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data, and generating new labeled hyperspectral training data from the labeled hyperspectral training data and the pseudo-labeled hyperspectral training data.

[0071] Specifically, conventional hyperspectral image classification uses the intermediate deep neural network trained in S202 as the target deep neural network to classify the hyperspectral data to be classified. However, due to the limited availability of labeled hyperspectral training data, the trained target deep neural network may not have an optimal network structure, resulting in limited hyperspectral image classification performance. The present invention considers using unlabeled hyperspectral data as labeled hyperspectral training data to improve the training accuracy of the deep neural network. However, the strategy for selecting labeled hyperspectral training data from unlabeled hyperspectral data is crucial; an inappropriate choice may negatively impact the training accuracy of the deep neural network and, in turn, the classification performance of the hyperspectral image.

[0072] In order to obtain high-quality unlabeled hyperspectral data to help train deep neural networks with limited labeled hyperspectral training data, the embodiment of the present invention designs a strategy for selecting part of the unlabeled hyperspectral data from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data using a collaborative selection strategy and a competitive selection strategy. Since the collaborative analysis confidence of the assistivity between adjacent samples and the competitiveness between similar samples is considered at the same time, samples with higher confidence can be selected as pseudo-labeled hyperspectral training data, providing more available recognition information for hyperspectral images for use in high-precision deep neural network training.

[0073] The embodiment of the present invention uses a collaborative selection strategy and a competitive selection strategy to select part of the unlabeled hyperspectral data from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data, including: constructing a pseudo-label mapping diagram of the unlabeled hyperspectral training data according to the classification results of the intermediate deep neural network; extracting a pseudo-label mapping matrix corresponding to each sample in the unlabeled hyperspectral data from the pseudo-label mapping diagram using a second neighborhood window; selecting samples corresponding to high-confidence three-dimensional cube unlabeled sample blocks from the pseudo-label mapping matrix using a neighborhood consistency criterion to form an available candidate pseudo-label data set; constructing a label probability matrix of the unlabeled hyperspectral training data according to the classification results of the intermediate deep neural network; and selecting high-probability available candidate pseudo-label data from the available candidate pseudo-label data set as pseudo-labeled hyperspectral training data according to the label probability matrix. Specifically:

[0074] First, a collaborative selection strategy is used to select high-quality pseudo-labeled hyperspectral training data. Figure 4 First, a pseudo-label map of the unlabeled hyperspectral training data is constructed based on the classification results of the intermediate deep neural network. The pseudo-label map shown on the left side represents the different classification results obtained by the intermediate deep neural network. For each sample in the unlabeled hyperspectral data, the pseudo-label map matrix corresponding to each sample in the unlabeled hyperspectral data is extracted from the pseudo-label map using the second neighborhood window. Spatial information is taken into account when selecting high-quality pseudo-label samples. The neighborhood consistency criterion is used to determine whether the sample corresponding to the unlabeled sample block of the three-dimensional cube can be used as a usable candidate pseudo-label data. The neighborhood consistency criterion is specifically as follows: the maximum threshold T set in advance is used to determine whether the sample can be used as a candidate pseudo-label data. v =N ne The voting (highest consistency) mechanism is used to evaluate the confidence of the sample corresponding to the unlabeled sample block of the three-dimensional cube, and the number of neighborhood samples with the same label is counted from the pseudo-label mapping matrix, N ne Indicates the number of neighborhood samples with the same label corresponding to the sample of the three-dimensional cube unlabeled sample block counted, and the number of neighborhood samples counted around it is not less than N ne The sample corresponding to the 3D cube unlabeled sample block indicates that the sample corresponding to the 3D cube unlabeled sample block has a high confidence level and can be used as available candidate pseudo-label data. The available candidate pseudo-label data is then added to the available candidate pseudo-label set. The second neighborhood window can be flexibly designed based on actual needs. For example, it can be designed as a 3×3 window or a 4×4 window. It can be the same size as the first neighborhood window or a different size.

[0075] For example: the second neighborhood window is 3×3, assuming T v =N ne= 4. If the sample corresponding to the 3D cube unlabeled sample block receives at most 3 votes from the same category, the sample corresponding to the 3D cube unlabeled sample block will not be selected, that is, the confidence is low and it will not be considered as available candidate pseudo-label data. If the sample corresponding to the 3D cube unlabeled sample block receives at most 4 votes from the same category, the sample corresponding to the 3D cube unlabeled sample block will be selected as the data of that category and added to the available candidate sample dataset. The above selection is performed on the samples corresponding to all 3D cube unlabeled sample blocks to obtain the final available candidate pseudo-label dataset D avi , the formula is:

[0076]

[0077] Among them, x i represents the sample corresponding to the i-th 3D cube unlabeled sample block, x it Represents the sample x corresponding to the unlabeled sample block of the three-dimensional cube i The tth neighborhood sample, y i Indicates that N is satisfied ne The label value of the category with the highest vote is obtained when the neighborhood samples vote consistently. It can be seen that the embodiment of the present invention can select the category with T according to the designed criterion based on neighborhood consistency. v Voting unlabeled hyperspectral data, such a criterion can achieve the first selection of high confidence unlabeled hyperspectral data to be added to the available candidate pseudo-label dataset D avi .

[0078] Furthermore, in order to obtain higher quality unlabeled hyperspectral data to help train deep neural networks with limited labeled hyperspectral training data, the embodiment of the present invention also proposes to use the available candidate pseudo-label dataset D avi Based on this, a competitive selection strategy is adopted to achieve the selection of higher quality pseudo-labeled hyperspectral training data. Specifically:

[0079] A softmax classifier is connected after the intermediate deep neural network. The classification results of the softmax classifier are converted into corresponding probability values. The probability values corresponding to the classification results of each sample in the label hyperspectral training data form a label probability matrix. The label probability matrix is used to measure the probability of each sample x in the label hyperspectral training data. i The probability value of belonging to a certain label category. For the available candidate pseudo label dataset D avi The available candidate pseudo-label data of the same category in the dataset D are sorted according to the label probability matrix. avi All available candidate pseudo-label data are sorted, and the formula is expressed as:

[0080] v i =f rank(p(H(x i )=y i )) (3)

[0081] Among them, v i Represents the sample x corresponding to the unlabeled sample block of the three-dimensional cube i The sorting order of the probability value belonging to the label category. The sorting value v of the sample corresponding to each unlabeled sample block of the three-dimensional cube i It can be designed by the label category matrix and the label probability matrix, and its calculation process is: i is the sample x corresponding to the unlabeled sample block of the three-dimensional cube i The label value of the label category, H(·) represents a category matrix, which is used to determine the sample x corresponding to the unlabeled sample block of the three-dimensional cube i Whether it belongs to the label category y i , p(·) represents the sample x corresponding to the unlabeled sample block of the three-dimensional cube i Belongs to the label category y i The probability of f rank (·) represents a ranking function, which can realize the function of sorting the samples corresponding to all three-dimensional cube unlabeled sample blocks in the corresponding label class in descending order according to the category probability value. Specifically, the samples corresponding to the three-dimensional cube unlabeled sample blocks with decreasing probability are assigned a rank value in descending order. For example, the highest probability value is assigned a rank value of 1, the second highest probability value is assigned a rank value of 2, and so on. It can be seen that the higher the rank value, the lower the category probability value. According to such a rank value sorting result, we can select the available candidate pseudo-label dataset D avi The middle classification class selects a certain percentage of available candidate pseudo-label data with low rank values as the final available candidate pseudo-label dataset D for each category avi , to achieve the selection of high-quality pseudo-labeled hyperspectral training data, and the specific percentage is set according to the actual design. Finally, all the selected pseudo-labeled hyperspectral training data are added to the labeled hyperspectral training data D tr Complete label hyperspectral training data D tr Updates.

[0082] It can be seen that the network trained in the embodiment of the present invention is an intermediate deep neural network Net0. The intermediate deep neural network Net0 is directly trained using limited labeled hyperspectral training data, and then pseudo-labeled hyperspectral training data is generated based on the trained intermediate deep neural network Net0. Specifically: See Figure 4, the unlabeled hyperspectral data is sent to the intermediate deep neural network Net0 to complete feature extraction and classification, and the label probability matrix corresponding to the labeled hyperspectral training data is obtained. Then, the collaborative selection strategy and the competitive selection strategy are used to collaboratively analyze the confidence evaluation strategy. The unlabeled hyperspectral data with higher confidence can be selected as pseudo-labeled hyperspectral training data, thereby providing more available recognition information for hyperspectral image classification, which can be used in high-precision deep neural network training.

[0083] The method for updating limited-label hyperspectral training data proposed in the embodiment of the present invention avoids manual labeling operations, improves the accuracy of newly labeled hyperspectral training data, and enables high-confidence labeled hyperspectral training data to provide more prior information for deep neural network training during the training process. The deep learning network based on more prior information can avoid the problem of overfitting that is prone to occur with limited-label hyperspectral training data, thereby training a deep neural network corresponding to the optimal network parameters.

[0084] S203 , using the new labeled hyperspectral training data to perform the above training on the intermediate deep neural network until the training stop condition is met to obtain the target deep neural network.

[0085] Specifically, the embodiment of the present invention uses the newly labeled hyperspectral training data generated in S202 to train the intermediate deep neural network in S201. If the training stop condition is not met, S201 and S202 are repeated until the training stop condition is met, and the intermediate deep neural network at this time is output as the target deep neural network. If the training stop condition is met, the intermediate deep neural network at this time is output as the target deep neural network. The training stop condition can be set to a maximum number of iterations.

[0086] The target deep neural network trained by the above S201 to S203 is used to classify hyperspectral data. Since more labeled hyperspectral training data are used to train the deep neural network during the training process, it has better guidance for feature extraction of hyperspectral images, so that the trained target deep neural network has better generalization ability and higher classification accuracy.

[0087] Furthermore, analyzing and utilizing data structures can provide rich information for obtaining the relationship between the data's own attributes and samples. However, in the feature mapping of traditional neural networks, the spatial structure information that helps to obtain discriminative features cannot be maintained. In recent years, manifold learning has been widely used in various fields due to its advantage in better describing the distribution characteristics of data. Local linear embedding is a basic manifold learning algorithm based on nonlinear unsupervised learning, which transforms the high-dimensional space Ω into a H The data in is projected into the low-dimensional space Ω LIn this way, the global nonlinear structure of the high-dimensional data is converted into a local linear structure, thereby reducing data redundancy. To this end, the embodiment of the present invention designs a deep neural network based on a local preservation constraint strategy, which uses a mapped low-dimensional space Ω L The neighborhood of the sample in the high-dimensional space and the local weight matrix in the high-dimensional space reconstruct the data points, which well preserves the original high-dimensional space Ω H The local geometric structure of the data.

[0088] The deep neural network based on the local preservation constraint strategy of the embodiment of the present invention is also trained using the divided three-dimensional cube label sample blocks. The process of training the deep neural network using the divided three-dimensional cube label sample blocks includes: constructing a local preservation constraint function based on the divided three-dimensional cube label sample blocks based on the local preservation constraint strategy; constructing a cross-entropy loss optimization function of the deep neural network based on the divided three-dimensional cube label sample blocks; constructing a target loss optimization function of the deep neural network based on the local preservation constraint strategy based on the local preservation constraint function and the cross-entropy loss function; and solving the target loss optimization function for the optimal solution to obtain an intermediate deep neural network corresponding to the optimal network parameters. Specifically:

[0089] Assume that the hyperspectral image data D=[x1,x2,…,x N ], N represents the total number of samples including labeled hyperspectral training data and unlabeled hyperspectral data. The local preservation constraint implementation of the embodiment of the present invention includes the following steps:

[0090] S301 , calculating k neighborhoods of each sample in the labeled hyperspectral training data.

[0091] Specifically, the embodiment of the present invention respectively performs the following operations on the sample x in the label hyperspectral training data: i (i=1,2,…,N tr )’s k nearest neighbor neighborhood samples, denoted as {x i1 ,x i2 ,…,x ik}, x ij (j=1,2,…,k) represents the sample x in the labeled hyperspectral training data i The j-th neighborhood samples of are selected by applying the Euclidean distance.

[0092] S302: Calculate the reconstruction weight of each sample in the labeled hyperspectral training data.

[0093] Specifically, the embodiment of the present invention labels the sample x in the hyperspectral training data i It can be reconstructed using its neighborhood samples, and the reconstruction formula is expressed as:

[0094] xi =ω i1 x i1 +ω i2 x i2 +…+ω ik x ik (4)

[0095] Among them, ω i =ω i1 ,ω i2 ,…,ω ik Represents each sample x in the label hyperspectral training data i In the high-dimensional feature space Ω H The weight coefficient in , ω ij Represents the neighborhood sample x ij For the sample x in the labeled hyperspectral training data i Therefore, a high-dimensional feature space Ω can be obtained H The k nearest neighbors in the label hyperspectral training data represent the sample x i The linear representation coefficient is expressed as follows:

[0096]

[0097] S303 , mapping each sample in the labeled hyperspectral training data from the high-dimensional feature space to the low-dimensional feature space.

[0098] Specifically, by putting the labeled hyperspectral training data into the designed deep growing network, we can obtain the low-dimensional feature space Ω L , sample x in the label hyperspectral training data i Mapping to low-dimensional feature space Ω L Labeled hyperspectral training data z i Expressed as: z i =ω i1 z i1 +ω i2 z i2 +…+ω ik z ik , z ij Represents the low-dimensional feature space Ω L Labeled hyperspectral training data z i The jth neighborhood sample of .

[0099] S304: Maintain the local linear structure of the features of the low-dimensional feature space consistent with the features of the high-dimensional data space.

[0100] Specifically, the embodiment of the present invention maps the low-dimensional feature space Ω L The features in the original high-dimensional data space Ω H The consistent local linear structure is expressed as:

[0101]

[0102] Among them, z ij Represents the low-dimensional feature space Ω L Labeled hyperspectral training data z i The jth nearest neighbor sample of , I represents the identity matrix.

[0103] The local preservation constraint function finally designed in the embodiment of the present invention is expressed as follows:

[0104]

[0105] Among them, N tr represents the number of labeled hyperspectral training data, θ represents the network parameters of the deep neural network, represents the square of the F norm. By introducing formula (7), the local reconstruction relationship between each sample in the labeled hyperspectral training data and its neighboring samples can be maintained in the mapping space. It is worth noting that the local preservation constraint function of the constrained mapping feature space changes with the change of the final mapping feature of the designed deep neural network.

[0106] Therefore, the embodiment of the present invention introduces the designed local preservation constraint function into the cross entropy loss optimization function of the designed deep neural network, and targets the labeled hyperspectral training data D tr , a target loss optimization function of a semi-supervised deep neural network based on a local preservation constraint strategy is designed, and the formula is expressed as:

[0107] L(X tr ,θ)=L base (X tr ,θ)+λL em (X tr ,θ) (8)

[0108] Among them, X tr represents a three-dimensional cube label sample block, θ represents the network parameters of the deep neural network, L(X tr ,θ) represents the target loss optimization function, L base (X tr ,θ) represents the cross entropy loss optimization function, which is the main loss function term of deep neural network, L em (X tr,θ) represents the local preservation constraint function, and λ represents an adjustable trade-off parameter. For example, the adjustable trade-off parameter λ is set to 0.001. As can be seen from formula (8), the target loss optimization function designed in the embodiment of the present invention can constrain the relationship between the input feature space and the mapping feature space of the network. It can optimize the network parameters by enriching the labeled data information and mining the unlabeled sample data information, thereby providing guidance for better interpretation of hyperspectral images and better classification of hyperspectral images.

[0109] Furthermore, the above process of the embodiment of the present invention is based on a fixed lightweight deep neural network Net0. This is because it is difficult to obtain a better deep learning network with limited labeled samples. Even if theoretical analysis shows that designing a more complex deep neural network will improve classification accuracy, the limited labeled samples cannot guarantee the acquisition of a better complex deep neural network, so that the accuracy of hyperspectral image classification is more likely to decrease rather than increase. However, the inventors have found that the present invention proposes to use collaborative selection strategies and competitive selection strategies to generate more pseudo-labeled hyperspectral training data, and add these pseudo-labeled hyperspectral training data to the original limited labeled hyperspectral training data to form a larger data set of labeled hyperspectral training data, thereby improving the training accuracy of the deep neural network. Based on the above larger data set of labeled hyperspectral training data, the embodiment of the present invention also proposes to use a self-growth strategy to design a deep neural network. Before self-growth, the initial deep neural network uses the lightweight deep neural network Net0. Similarly, the lightweight deep neural network Net0 preferably uses the Kaiming initialization method to initialize the network parameters corresponding to the initial deep neural network.

[0110] For the implementation of such self-growth in the training process, please refer to Figure 5, including: adding a basic residual network module between the output module of the intermediate deep neural network and the last basic residual network module to form a new intermediate deep neural network; using new labeled hyperspectral training data to perform the above training on the intermediate deep neural network until the training stop condition is met to obtain the target deep neural network, including: using new labeled hyperspectral training data to perform the above training on the new intermediate deep neural network until the training stop condition is met to obtain the target deep neural network. As can be seen, the self-growing network training process is similar to the above process. The difference is that a network adaptive growth method is added during the training process. That is, the network for the next round of training is no longer the original intermediate deep neural network structure, but a new intermediate deep neural network structure. The embodiment of the present invention forms this new intermediate deep neural network structure by adding a basic residual network module between the output module and the last basic residual network module of the lightweight deep neural network Net0 to form a new intermediate deep neural network Net1. The new intermediate deep neural network Net1 is then trained. If the training stop condition is still not met, a basic residual network module is added between the output module and the last basic residual network module of the intermediate deep neural network Net1 to form a new intermediate deep neural network Net2. Training continues until the training stop condition is met and the intermediate deep neural network at this time is used as the target deep neural network. If the training stop condition is met, the intermediate deep neural network at this time is directly used as the target deep neural network. The training stop condition can be the maximum number of iterations or the number of added basic residual network modules reaching a specified module number threshold R, such as R = 2000. If the training stop condition is met, the network training is stopped and the intermediate deep neural network at this time is output as the target deep neural network.

[0111] An embodiment of the present invention provides an optional solution for adding a basic residual network module. The network structure of the basic residual network module added between the output module of an intermediate deep neural network and the last basic residual network module is the same as the network structure of the last basic residual network module. Furthermore, when initializing the newly grown intermediate deep neural network Net r, an efficient and reliable neural network parameter initialization method is designed. When training the new intermediate deep neural network using newly labeled hyperspectral training data, the network parameters corresponding to the previously trained intermediate deep neural network are used as the initialization network parameters of the new intermediate deep neural network. The network parameters of the added basic residual network module are the same as the network parameters of the last basic residual network module. For example, the network parameters of the previously trained intermediate deep neural network Net r-1 are transferred as the initialization network parameters to the new intermediate deep neural network Net r. In particular, the initialization parameters of the newly added basic residual network module in the new intermediate deep neural network Net r remain the same as the parameters of the last basic residual network module in the intermediate deep neural network Net r-1. During the growth process of the deep neural network, as the number of basic residual network modules in the deep neural network increases, its network parameters will also increase accordingly. If the traditional method of directly randomly initializing a large number of parameters in the network is used, the feature extraction ability of the network will be affected to a certain extent. In order to ensure the feature extraction ability, in the training of the embodiment of the present invention, the initial parameters of the trained and grown deep neural network are taken from the parameters of the deep neural network after the last training, and the network parameters of the newly added basic residual network module are taken from the network parameters of the last basic residual network module in the deep neural network after the last training.

[0112] It can be seen that the lightweight deep neural network Net0 in the embodiment of the present invention can be used to extract shallow features, while the gradually growing deep deep neural network can be used to extract high-level features. Such a deep neural network structure can achieve better hyperspectral image classification performance.

[0113] Moreover, in order to control the network training efficiency, during the deep neural network growth training process, the embodiment of the present invention can input the same new label hyperspectral training data generated by the original label hyperspectral training data and the pseudo-label hyperspectral training data into each growing deep neural network. More preferably, different growth processes use the label hyperspectral training data required by the network at this time, that is, if it is a lightweight deep neural network Net r in the growth stage, the pseudo-label hyperspectral training data it uses is generated by the lightweight deep neural network Net r-1 after the previous round of training, r∈{1,2,…,R}, which is used to expand the label hyperspectral training data. The specific corresponding new label hyperspectral training data generation process refers to the above-mentioned S202 process, and then the new label hyperspectral training data is used to train the newly grown deep neural network Net r, thereby ensuring the training accuracy of the growing deep neural network. In addition, this method trains the deep neural network by utilizing prior data information of more hyperspectral image data, which is conducive to learning the real information and discriminative information of the hyperspectral image data.

[0114] At the same time, in the self-growing deep neural network training process of the embodiment of the present invention, the loss function can be designed only as the cross entropy loss optimization function shown in formula (1), or as the target loss optimization function shown in formula (8). The optimal network parameters of the target deep neural network finally formed are different, but the target deep neural networks all achieve better classification performance than traditional deep neural networks. Among them, the network structure trained with the target loss optimization function shown in formula (8) can achieve better classification performance because it constrains the relationship between the input feature space and the mapping feature space of the network. The network parameters can be optimized by enriching the labeled data information and mining the unlabeled sample data information to obtain the target deep neural network corresponding to the better network parameters. Figure 6 Schematic illustration of the deep neural network training process with the introduction of locality-preserving constraints.

[0115] Furthermore, when designing a deep neural network based on residual network modules, since the bottleneck residual network module is more economical than the traditional residual network module, each bottleneck residual network module can reduce the input and output convolutional layers and then restore the depth dimension of the feature map, which allows each bottleneck residual network module to be executed faster. Therefore, the embodiment of the present invention preferably uses the bottleneck residual network module as the basic residual network module of the proposed deep neural network model, that is, a lightweight basic neural network Net0 including a plurality of bottleneck residual network modules connected in sequence and an output module is designed. The bottleneck residual network module includes a convolutional layer, a batch normalization layer, an activation layer, and a downsampling layer connected in sequence. Different from the convolutional layer in the basic residual network module, the convolutional layer in the bottleneck residual network module achieves the same feature extraction result and the same model accuracy as the convolutional layer in the basic residual network module by pre-reducing or expanding the dimension through a convolutional layer using a 1×1 convolutional kernel, but only has a smaller computational amount than the basic residual network module, thus reducing the training time of the deep neural network in the embodiment of the present invention.

[0116] Furthermore, after the above training, the trained target deep neural network is output. Feature extraction is performed through the target deep neural network. Since it contains deep feature maps with more semantic information, it is easier to obtain classification results. For the classification output result Ω of the deep neural network model grown during the r-th iteration out , which is expressed by the formula:

[0117]

[0118] where x label represents a sample in the original limited labeled hyperspectral training data. r = 0 indicates that the deep neural network at this time is the ungrown lightweight deep neural network Net0. θ represents the optimal network parameters of the lightweight deep neural network Net0 obtained through training. 0 < r ≤ R indicates that the deep neural network at this time is the lightweight deep neural network Net r grown through r iterations. represents a sample in the labeled hyperspectral training data during the r-th iteration. θ r-1 represents the optimal network parameters obtained through training of the lightweight deep neural network Net r-1, that is, the initial network parameters of the lightweight deep neural network Net r. r can also represent the number of added basic residual network modules. R represents the maximum module number threshold that can be added. At this time, the updated labeled hyperspectral training data D tr is expressed by the formula:

[0119] D [[ID=第23]] tr = D label ∪ D avi [[ID=2

[0120] Among them, D label represents the labeled hyperspectral training data used for training in r iterations, D avi represents the available candidate pseudo-label dataset generated by the collaborative selection strategy and the competitive selection strategy in r iterations, and ∪ represents the union operation.

[0121] Please see again Figure 5 and Figure 6 In this embodiment of the present invention, a local preservation constraint strategy and limited labeled hyperspectral training data are first used to train the lightweight deep neural network Net0, so that the network has better feature expression capabilities. Then, the trained lightweight deep neural network Net0 is used to complete feature extraction and classification of unlabeled hyperspectral data. For the label probability matrix obtained after classification of unlabeled hyperspectral data, a collaborative selection strategy and a competitive selection strategy are used to generate high-quality pseudo-labeled hyperspectral training data, and then add it to the labeled hyperspectral training data D. tr In the middle, complete the label hyperspectral training data D tr Then, after the lightweight deep neural network Net0 grows, it uses the updated label hyperspectral training data D tr The lightweight deep neural network Net R in the growth phase is trained. The lightweight deep neural network is updated, grown, and trained repeatedly. If the training termination condition in this embodiment of the present invention is that the number of added basic residual network modules reaches a specified module threshold R, network training is complete, and the final target deep neural network is Net R. The hyperspectral data to be classified is then input into the target deep neural network Net R, and the classification result of the hyperspectral data to be classified is output accordingly, thereby completing the classification task.

[0122] In order to verify the effectiveness of the hyperspectral image classification method based on deep neural network provided by the embodiment of the present invention, the following experiments were conducted for verification.

[0123] During the experiment, each experiment was run independently 20 times, and the average of the experimental results was used as the evaluation standard for the classification results. During the training of the deep neural network, the adjustable trade-off parameter λ in the method proposed in the embodiment of the present invention was set to 0.001, and the cross-entropy loss was selected as an indicator to measure the feature learning ability of these deep neural networks during the training process. The optimizer used was Adam, and for different labeled hyperspectral training data, the learning rate of Adam was set to a value in the range of 0.00001 to 0.001. Among them, the batch size and the maximum number of training iterations during training were set to 200 and 2000, respectively.

[0124] During the experiment, various common classification models were used, including: a classification method based on support vector machines (SVM), a classification method based on random forests (RF), a classification method based on 1D-CNN, a classification method based on 2D-CNN, and a classification method based on 3D-CNN. CNN is a convolutional neural network. The method of the present invention and the above five traditional classification methods were tested on three datasets: Indian Pines, PaviaU, and Salinas. The detection result images and detection quantitative indicators were output to verify the practicality of the method proposed in the embodiment of the present invention.

[0125] The first dataset is the Indian Pines image from AVIRIS, see Figure 7 (a)~ Figure 7 (b), the image size is 145×145, the spatial resolution is 20×20m, and the pseudo-color image of this dataset is as follows Figure 7 As shown in (a), the images of 16 real objects covered are as follows Figure 7 (b) shows the original image's spectral bands, which range from 0.2 μm to 2.4 μm. After removing 20 bands due to atmospheric absorption, water absorption, and low noise, the remaining 200 bands are used for classification. This dataset contains 16 different ground features: Alfalfa, Corn-notill, Corn-min, Corn, Grass-mowed, Grass-trees, Grass-pasture, Hay-windrowed, Oats, Soybeans-notill, Soybeans-min, Soybeans-clean, Wheat, Woods, Buildings-grass, and Stones-towers, as shown in Table 1.

[0126] When using the Indian Pines dataset, the size of the three-dimensional cube label sample block input to the deep neural network is set to 11×11×200. About 15% of the samples (1543 samples) in each class are randomly selected to construct the labeled hyperspectral training data D L , used for deep neural network training, and the remaining samples (8706 samples) were used to construct unlabeled hyperspectral data D te, used to test the classification performance of the proposed method and generate pseudo-signed hyperspectral training data. Table 1 shows the quantitative classification results of the proposed method and the traditional classification methods on the Indian Pines dataset. Table 2 shows the overall quantitative classification results of the proposed method and the traditional classification methods on the Indian Pines dataset. The overall quantification is based on three evaluation indicators: Overall Accuracy (OA), Average Accuracy (AA) and Kappa coefficient, as well as Figure 10 (a)~ Figure 10 (f) shows the mapping classification results of the method of the present invention and various traditional classification methods on the Indian Pines dataset.

[0127] Table 1 Comparison of classification results for each category on the Indian Pines dataset

[0128] category Support Vector Machine RF 1D-CNN 2D-CNN 3D-CNN Method of the present invention 1 58.7 60.87 47.83 80 67.39 81.82 2 80.25 70.73 90.27 95.71 85.08 98.52 3 70.96 61.57 87.83 74.78 77.83 97.22 4 53.59 53.59 73.84 82.17 75.11 98.33 5 91.3 88.82 94.2 56.44 95.03 98.77 6 96.44 94.52 99.04 96.62 99.04 99.86 7 82.14 7.14 85.71 91.67 53.57 100 8 96.86 98.33 99.58 100 98.95 99.79 9 25 5 100 52.94 45 100 10 76.85 78.09 90.95 98.42 83.13 99.69 11 79.71 88.02 92.79 71.44 91.07 99.79 12 77.23 65.77 82.12 94.05 74.7 96.86 13 94.15 96.1 100 99.42 99.51 99.03 14 94.78 93.04 97.47 75.464 97.63 99.14 15 66.84 54.66 87.82 99.69 81.35 93.07 16 92.47 90.32 98.92 100 88.17 95.88

[0129] Table 2 Comparison of overall classification results on the Indian Pines dataset

[0130] OA 82.18 80.47 92.02 84.07 88.51 98.65 AA 77.33 69.16 89.27 85.55 82.03 97.98 Kappa 82.1 80.38 90.89 82.06 86.84 98.47

[0131] From Table 1 and Figure 10 (a)~ Figure 10 (f) It can be seen that in each class, the performance of the SVM method and the RF method is inferior to the network-based classification methods (such as 1D-CNN, 2D-CNN, 3D-CNN, and the present invention). It can be clearly seen that these two methods have difficulty in identifying the ninth class of samples composed of oats. This result is partly due to the difficulty of classification caused by the very similar spectral features of hyperspectral images. Another possible reason is the lack of training samples (a total of 20 samples, only 3 of which were used for training). The OA values of the 1D-CNN, 2D-CNN, and 3D-CNN methods are significantly improved compared to the SVM method and RF method, demonstrating that network-based classifiers can extract features with more discriminative information. By comparing the method proposed in this invention with 1D-CNN, 2D-CNN, and 3D-CNN, as shown in Table 2, it can be clearly observed that the classification accuracy obtained by the method proposed in this invention increased to 98.65%, 97.98%, and Kappa to 98.47%. The classification noise in its mapped classification map is very small, and the mapped classification map is also smoother. The above analysis proves that the deep neural network proposed in this paper has advantages in better learning the discriminative spatial and spectral information in hyperspectral images.

[0132] The second dataset is the ROSIS hyperspectral image of the University of Pavia (PaviaU), see Figure 8 (a)~ Figure 8 (b), the pseudo-color image of the dataset is as follows Figure 8 As shown in (a), the images of the 9 real objects covered are as follows Figure 8 (b) This image has a spatial resolution of 1.3 m per sample and contains 610 × 340 samples. After removing 12 noise bands and the water vapor band, 103 bands ranging from 0.43 μm to 0.86 μm remain for classification. This dataset contains nine different ground features: asphalt, meadows, gravel, trees, metal sheets, bare soil, bitumen, brick, and shadows, as shown in Table 3.

[0133] When using the PaviaU dataset, the three-dimensional cube label sample block input to the deep neural network is 11×11×103. About 15% of the samples (1440 samples) are randomly selected from each class to construct the labeled hyperspectral training data D L , used for deep neural network training, and the remaining samples (41336 samples) were used to construct unlabeled hyperspectral data D te , used to test the classification performance of the proposed method and generate pseudo-labeled hyperspectral training data. Table 3 shows the comparison of the quantitative classification results of the proposed method and traditional classification methods on the PaviaU dataset, and Table 4 shows the comparison of the overall quantitative classification results of the proposed method and traditional classification methods on the PaviaU dataset, as well as Figure 11 (a)~ Figure 11 (f) shows the mapping classification results of the method of the present invention and various traditional classification methods on the PaviaU dataset.

[0134] Table 3 Comparison of classification accuracy results for each category on the PaviaU dataset

[0135] category Support Vector Machine RF 1D-CNN 2D-CNN 3D-CNN Method of the present invention 1 95.76 87.41 97.78 96.2 98.55 99.26 2 97.94 92.87 99.43 99.33 99.6 99.97 3 80.37 68.7 89.37 98.48 91.81 97.98 4 93.93 88.15 96.02 99.15 97.22 99.54 5 99.85 97.03 99.85 100 99.47 100 6 88.03 71.58 95.13 96.77 98.42 99.96 7 80.68 74.81 93.08 73.03 95.71 97.07 8 90.6 77.29 96.03 91.08 95.73 99.21 9 100 99.58 99.57 99.62 99.89 99.37

[0136] Table 4. Comparison of overall classification results on the PaviaU dataset

[0137] OA 94.23 91.52 97.45 96.99 98.29 99.56 AA 89.27 88.84 96.25 94.85 97.38 99.56 Kappa 91.91 91.51 96.62 96.01 97.74 99.42

[0138] From Table 3 and Figure 11 (a)~ Figure 11(f) It can be seen that in each category, the performance of the SVM method, RF method, 1D-CNN, 2D-CNN and 3D-CNN is basically inferior to the classification method of the present invention. As can be seen from Table 4, compared with the OA value of 94.23% for the SVM method and 91.52% for RF, the OA value of 1D-CNN can reach 97.45%, the OA value of 2D-CNN can reach 96.99%, and the OA value of 3D-CNN can reach 98.29%. Other AA and Kappa indicators are also that 1D-CNN, 2D-CNN, and 3D-CNN are better than the SVM method and RF method. The results show that the classification method based on convolutional neural network can significantly improve the classification accuracy to a certain extent and can also obtain a better mapping classification result graph. As can be clearly seen in Table 3, the feature extraction capabilities of each category based on the 1D-CNN, 2D-CNN, and 3D-CNN network models are generally lower than those of the classification method proposed in this paper. The classification method proposed in this paper achieved the best results. Table 4 clearly shows that the classification method of this paper achieved an OA value of 99.56%, an AA value of 99.56%, and a kappa coefficient of 99.42%. The experimental analysis results are similar to those on the Indian Pines dataset, indicating that the classification method proposed in this paper can extract more discriminative hyperspectral feature information to assist the network model in classification.

[0139] The third dataset is the Salinas imagery, acquired by the 224-band AVIRIS sensor in the Salinas Valley, California, see Figure 9 (a)~ Figure 9 (b), the pseudo-color image of the dataset is as follows Figure 9 As shown in (a), the image covering 16 real objects is as follows Figure 9 (b) shows the Salinas scene data size of 512 rows and 217 columns, which is characterized by a high spatial resolution (i.e., 3.7 meters). Like the Indian Pines dataset, after discarding 20 noise bands and water vapor bands, the remaining 204 bands are used for classification. This dataset contains 16 different ground objects, namely Brocoli1, Brocoli2, Fallow, Fallow_r, Fallow_s, Stubble, Celery, Grapes, Soil, Corn-s, Lettuce-romaine-4wk, Lettuce-romaine-5wk, Lettuce-romaine-6wk, Lettuce-romaine-7wk, Vinyard-u, and Vinyard-v, as shown in Table 5.

[0140] When using the Salinas dataset, the size of the three-dimensional cube label sample block input to the deep neural network is set to 11×11×204. About 15% of the samples (about 8128 samples) are randomly selected from each class to construct the labeled hyperspectral training data D L , used for deep neural network training, and the remaining samples (about 46058 sample blocks) are used to construct unlabeled hyperspectral data D te , used to test the classification performance of the proposed method and generate pseudo-signed hyperspectral training data. Table 5 shows the comparison of the quantitative classification results of the proposed method and traditional classification methods on the Salinas dataset, and Table 6 shows the comparison of the overall quantitative classification results of the proposed method and traditional classification methods on the Salinas dataset, as well as Figure 12 (a)~ Figure 12 (f) shows the mapping classification results of the method of the present invention and various traditional classification methods on the Salinas dataset.

[0141] Table 5 Comparison of classification accuracy results for each category on the Salinas dataset

[0142] category Support Vector Machine RF 1D-CNN 2D-CNN 3D-CNN Method of the present invention 1 100 99.8 99.95 99.7 99.95 100 2 99.92 99.76 99.86 95.04 99.94 100 3 99.7 99.04 99.69 87.79 99.24 99.39 4 99.5 99.14 98.92 100 99.85 99.78 5 98.77 98.84 99.1 98.28 99.43 99.18 6 99.92 99.82 99.94 99.94 99.97 99.97 7 99.89 99.66 99.8 99.8 99.86 100 8 91.52 87.32 88.4 90.04 89.37 99.83 9 99.89 99.29 99.59 100 99.9 100 10 98.2 94.72 97.83 95.37 96.58 99.57 11 98.97 94.66 98.03 97.91 98.78 99.63 12 100 99.38 100 99.21 99.94 98.97 13 99.89 98.14 99.45 100 99.23 100 14 98.13 96.73 97.01 93.18 96.16 100 15 72.1 76.18 85.04 91.82 79.34 99.77 16 99.39 98.56 98.72 94.53 99.33 100

[0143] Table 6 Comparison of overall classification results on the Salinas dataset

[0144] OA 94.18 93.31 95.13 95.26 94.58 99.79 AA 97.23 96.31 97.58 96.41 97.31 99.82 Kappa 94.17 93.31 94.58 94.72 93.97 99.77

[0145] From Table 5 and Figure 12 (a)~ Figure 12 (f) It can be seen that in each category, the performance of the SVM method, RF method, 1D-CNN, 2D-CNN and 3D-CNN is basically inferior to the classification method of the present invention. Figure 12 (a)~ Figure 12(f) It can be seen that the classification methods based on 1D-CNN, 2D-CNN and 3D-CNN have better classification performance than the SVM method and RF method, and the SVM method and RF method also generate classification maps with more noise. In this comparison, the improvement of the classification performance of the 1D-CNN, 2D-CNN and 3D-CNN methods relative to the SVM method and RF demonstrates the advantages of feature extraction of convolutional neural networks. As can be seen from Table 6, in the experiment of the Salinas dataset, the classification method proposed in the present invention has an OA gain of 4.66%, 4.53% and 5.21% respectively, an AA gain of 2.24%, 3.41% and 2.51%, and a Kappa gain of 5.19%, 3.36% and 5.8% compared with the 1D-CNN, 2D-CNN and 3D-CNN networks. The classification method of the present invention achieves better classification performance. It can be seen that the classification method proposed in this invention outperforms most other methods in terms of classification performance. The OA value of the classification method of this invention is 99.79%, the AA value is 99.82%, and the Kappa coefficient is 99.77%. The classification method of this invention can achieve the best classification performance in most saline-alkali land image categories. Table 5, Table 6 and Figure 12 (a)~ Figure 12 The classification result of (f) shows that the classification method proposed in the present invention can improve the final classification performance by better learning the spectral spatial features of the hyperspectral spectrum.

[0146] Through the above quantitative and qualitative analysis on the three datasets of Indian Pines, PaviaU and Salinas, it can be concluded that the classification method proposed in the present invention shows the advantage of high classification performance on all datasets, that is, the classification method proposed in the present invention can better define category boundaries and better classify the categories of high-dimensional samples based on actual ground truth images, thereby proving the robustness and effectiveness of the classification method of the present invention; at the same time, the classification method proposed in the present invention has better generalization ability and has achieved better classification performance on multiple datasets.

[0147] In summary, the hyperspectral image classification method based on deep neural network proposed in the embodiment of the present invention addresses the problem that the existing data set has limited label data, which makes it difficult for the hyperspectral image classification method based on deep learning network to obtain a better deep learning network using limited label data, thereby affecting the hyperspectral image classification performance. During the training process, a strategy for expanding labeled hyperspectral training data is proposed, which utilizes the collaborative selection strategy and the competitive selection strategy to collaboratively analyze the confidence evaluation method, while considering the assistance between adjacent samples and the competitiveness between similar samples to obtain high-quality unlabeled hyperspectral data to expand the labeled hyperspectral training data, thereby providing more available recognition training data for training, so as to train a better deep neural network and improve the classification performance of the hyperspectral data to be classified.

[0148] At the same time, the embodiment of the present invention also proposes a deep neural network self-growth strategy. The continuously growing deep neural network structure depends on the updated labeled hyperspectral training data, and uses more prior data information of hyperspectral image data to train the deep neural network, which is helpful to learn the real information and discriminative information of the hyperspectral image data. By enriching the labeled data information and mining the unlabeled sample data information to optimize the network parameters, the optimal deep neural network finally obtained can obtain deeper separable features, thereby improving the classification performance of the hyperspectral data to be classified; during the training process, the initial parameters of the grown deep neural network trained by the embodiment of the present invention are taken from the parameters of the deep neural network after the last training, and the network parameters of the newly added basic residual network module are taken from the network parameters of the last basic residual network module in the deep neural network after the last training. This can obtain better feature extraction capabilities of the deep neural network than the traditional method of directly randomly initializing the network parameters.

[0149] Furthermore, during the training process, the embodiment of the present invention introduces a local preservation constraint strategy based on hyperspectral information. The network provides guidance for better interpretation of hyperspectral images by learning the properties of the samples themselves and the relationships between samples, thereby achieving higher hyperspectral data classification performance.

[0150] Based on the same invention idea, please see Figure 13 , an embodiment of the present invention provides an electronic device, including a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other via the communication bus 1304;

[0151] Memory 1303, used for storing computer programs;

[0152] The processor 1301 is configured to implement the steps of the above-mentioned hyperspectral image classification method based on deep neural network when executing the program stored in the memory 1303.

[0153] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored therein. When the computer program is executed by a processor, the steps of the above-mentioned hyperspectral image classification method based on deep neural network are implemented.

[0154] As for the electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0156] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0157] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A hyperspectral image classification method based on deep neural network, characterized in that: include: Obtain hyperspectral data to be classified; Inputting the hyperspectral data to be classified into a target deep neural network to obtain a classification result; The target deep neural network is trained based on labeled hyperspectral training data and unlabeled hyperspectral data; the training process includes: The labeled hyperspectral training data is used to train an initial deep neural network to obtain an intermediate deep neural network, wherein the initial deep neural network includes a plurality of basic residual network modules and an output module connected in sequence; a portion of unlabeled hyperspectral data is selected from the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data using a collaborative selection strategy and a competitive selection strategy, and new labeled hyperspectral training data is generated from the labeled hyperspectral training data and the pseudo-labeled hyperspectral training data; the intermediate deep neural network is trained using the new labeled hyperspectral training data until a training stop condition is satisfied to obtain the target deep neural network; The method of selecting part of the unlabeled hyperspectral data as pseudo-labeled hyperspectral training data from the unlabeled hyperspectral data by using the collaborative selection strategy and the competitive selection strategy includes: A pseudo-label mapping map of the unlabeled hyperspectral data is constructed according to the classification results of the intermediate deep neural network; a pseudo-label mapping matrix corresponding to each sample in the unlabeled hyperspectral data is extracted from the pseudo-label mapping map using a second neighborhood window; samples corresponding to three-dimensional cube unlabeled sample blocks with high confidence are selected from the pseudo-label mapping matrix using a neighborhood consistency criterion to form an available candidate pseudo-label dataset; a label probability matrix of the unlabeled hyperspectral data is constructed according to the classification results of the intermediate deep neural network; and available candidate pseudo-label data with high probability are selected from the available candidate pseudo-label dataset as pseudo-label hyperspectral training data according to the label probability matrix.

2. The hyperspectral image classification method based on deep neural network according to claim 1 is characterized in that: The basic residual network module is designed as a bottleneck residual network module.

3. The hyperspectral image classification method based on deep neural network according to claim 1 is characterized in that: Before using the labeled hyperspectral training data to train the initial deep neural network, the method includes: The Kaiming initialization method is used to initialize the network parameters corresponding to the initial deep neural network.

4. The hyperspectral image classification method based on deep neural network according to claim 1, characterized in that: The process of training the initial deep neural network using labeled hyperspectral training data includes: Extracting a three-dimensional cube label sample block corresponding to each sample in the label hyperspectral training data using a first neighborhood window; All three-dimensional cube label sample blocks are divided into batches, and the initial deep neural network is trained using the divided batches of three-dimensional cube label sample blocks.

5. The hyperspectral image classification method based on deep neural network according to claim 4 is characterized in that: The process of training a deep neural network using the divided batches of three-dimensional cube labeled sample blocks includes: Based on the local preservation constraint strategy, a local preservation constraint function is constructed according to the batch of 3D cube label sample blocks after division; Construct a cross entropy loss optimization function for a deep neural network based on the divided batches of three-dimensional cube label sample blocks; Constructing a target loss optimization function of a deep neural network based on a local preservation constraint strategy according to the local preservation constraint function and the cross entropy loss optimization function; The target loss optimization function is solved for an optimal solution to obtain an intermediate deep neural network corresponding to the optimal network parameters.

6. The hyperspectral image classification method based on deep neural network according to claim 5 is characterized in that: The objective loss optimization function formula of the deep neural network based on the local preservation constraint strategy is expressed as: ； in, Represents the batch of 3D cube label sample blocks after division, represents the network parameters of the deep neural network, represents the target loss optimization function, represents the cross entropy loss optimization function, represents the local preservation constraint function, represents a trade-off parameter.

7. The hyperspectral image classification method based on deep neural network according to claim 1, characterized in that: The training process further includes: Adding a basic residual network module between the output module of the intermediate deep neural network and the last basic residual network module to form a new intermediate deep neural network; The intermediate deep neural network is trained using the new labeled hyperspectral training data until a training stop condition is satisfied to obtain the target deep neural network, including: The new intermediate deep neural network is trained using the new labeled hyperspectral training data until the training stop condition is met to obtain the target deep neural network.

8. The hyperspectral image classification method based on deep neural network according to claim 7 is characterized in that: The network structure of the basic residual network module added between the output module of the intermediate deep neural network and the last basic residual network module is the same as the network structure of the last basic residual network module.

9. The hyperspectral image classification method based on deep neural network according to claim 8, characterized in that: When training a new intermediate deep neural network using new labeled hyperspectral training data, the network parameters corresponding to the intermediate deep neural network at the end of the last training are used as the initialization network parameters of the new intermediate deep neural network; among them, the network parameters of the added basic residual network module are the same as the network parameters of the last basic residual network module.

Images