A Hyperspectral Image Classification Method, System, Device and Storage Medium

Abstract

The present invention discloses a hyperspectral image classification method, system, device and storage medium, comprising the following steps: performing preliminary feature extraction on the acquired hyperspectral image, inputting the extracted preliminary features into a multi-scale spatial spectral feature extraction model to obtain a feature map containing different-scale spatial information and spectral information; performing advanced feature extraction on the feature map of different-scale spatial information and spectral information to obtain a global context representation of the hyperspectral image; inputting the obtained global context representation into a dual decoder for feature recovery, and simultaneously constructing a dual decoder loss function to calculate the supervised loss of the labeled samples, the unsupervised loss of the unlabeled samples and the consistency loss of all samples in the hyperspectral image, and outputting the classification result of the hyperspectral image through a classification module. By using a small amount of labeled data and a large amount of unlabeled data, the overfitting problem caused by insufficient training samples in the supervised mode and the low accuracy in the unsupervised mode are solved; and the problem that it is difficult to fully utilize the spectral and spatial information of hyperspectral data is also solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification processing, and relates to a hyperspectral image classification method, system, device and storage medium. Background Art

[0002] A hyperspectral image is an image with hundreds of adjacent and narrow spectral channels and continuous bands obtained by a hyperspectral remote sensing system by acquiring and recording solar radiation signals with ground object information. Different from ordinary RGB images, a hyperspectral image contains hundreds of continuously banded channels, providing richer spectral information. In recent years, due to the unique properties of hyperspectral data and the huge amount of information it contains, the analysis and processing of hyperspectral images have been widely popular, and the hyperspectral image classification task among them plays a substantial and important role in fields such as geological exploration, crop detection, and national defense and military.

[0003] However, the classification of hyperspectral images still faces many challenges, most of which are related to the characteristics of hyperspectral image data. Hyperspectral images have hundreds of adjacent and narrow spectral channels, with extremely high spectral resolution and low spatial resolution. Although they have rich spectral information, the calculation of this data requires a large amount of time and resources. In addition, due to the increase in the number of parameters, processing this high-dimensional data is a complex task, which will greatly affect the classification performance, especially in the case of supervised learning. At the same time, the lack of labeled hyperspectral data is another major challenge in the hyperspectral image classification task, because labeling hyperspectral images is a time-consuming and expensive task that requires pixel-by-pixel labeling and the determination of pixel categories based on on-site inspections, resulting in fewer labeled samples of hyperspectral images.

[0004] In recent years, due to the strong feature extraction ability and effective information processing ability of deep learning in computer vision and image processing, deep learning has been widely applied to the field of hyperspectral image classification. Initially, Hu et al. proposed using a one-dimensional convolutional neural network to take the original spectral information as input and perform hyperspectral image classification through one convolutional layer, one pooling layer, and two fully connected layers. This method considered spectral information but ignored the spatial information between pixels, resulting in low classification accuracy. Two-dimensional convolutional neural networks can extract the spatial features of hyperspectral images, but require many parameters when processing rich spectral information and are prone to overfitting in the case of fewer training samples. Therefore, in order to solve the problem of overfitting caused by using 2D-CNN for hyperspectral image classification due to more parameters, it is proposed to use principal component analysis to reduce the dimension of hyperspectral images before extracting spatial features, and then use two-dimensional convolution for feature extraction and classification.

[0005] The above-mentioned model has only one branch, either extracting spatial features or spectral features, which cannot fully utilize spatial and spectral information simultaneously. Therefore, it is proposed to combine the spatial and spectral branches, using two branches, one is the spectral branch for extracting spectral features, and the other is the spatial branch for extracting spatial features. The features from different branches are fused to form a combined spectral-spatial feature. Zhang et al. proposed a dual-channel CNN framework, in which 1D CNN is used to hierarchically extract spectral features, 2D-CNN is used to extract spatial features, and then the spatial and spectral features are combined to complete the classification task. He proposed introducing a three-dimensional convolutional neural network (3D-CNN) into end-to-end hyperspectral image classification. 3D-CNN can jointly learn two-dimensional spatial and one-dimensional spectral features. Compared with 1D-CNN and 2D-CNN, the three-dimensional convolutional kernel can jointly slide between spatial and spectral dimensions to meet the requirements of multi-scale and multi-resolution, and can extract more complex spatial and spectral information.

[0006] Although the above methods can already achieve good classification of hyperspectral images, a large amount of labeled data is required during the training process, and it is difficult to obtain labeled samples for hyperspectral images. Currently, the classification task of hyperspectral images still faces some problems. First, there are few labeled samples in the hyperspectral dataset: the acquisition and processing of hyperspectral images are relatively difficult, and at the same time, the manual annotation cost is high. When annotating hyperspectral images, it has the characteristic of high annotation cost. When using deep learning to train a model, a large number of samples are required. If the data volume is insufficient, it will bring certain limitations to the research of the algorithm. Second, it is difficult to fully utilize the spectral information of hyperspectral data. Hyperspectral images have high spectral resolution and a large number of spectra. Each pixel contains rich spectral information, but the spectral information is complex and difficult to be fully utilized. Finally, the high-dimensional characteristic makes the storage and processing of data difficult. The high-dimensional characteristic of hyperspectral images increases the computational difficulty when processing them. Therefore, how to effectively reduce the computational overhead of the model and improve the classification accuracy is also a current challenge. Summary of the Invention

[0007] The purpose of the present invention is to solve the problems in the prior art that there are few labeled samples in the hyperspectral dataset, resulting in insufficient data volume, which will bring certain limitations to the research of the algorithm. Secondly, it is difficult to fully utilize the spectral information of hyperspectral data, and the computational difficulty is large and the classification accuracy is low during processing. The present invention provides a hyperspectral image classification method, system, device and storage medium.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A hyperspectral image classification method includes the following steps:

[0010] S1: Conduct preliminary feature extraction on the acquired hyperspectral image;

[0011] S2: Input the extracted preliminary features into the multi-scale spatial-spectral feature extraction model to obtain feature maps containing different-scale spatial information and spectral information;

[0012] S3: Conduct advanced feature extraction on the feature maps of different-scale spatial information and spectral information to obtain the global context representation of the hyperspectral image;

[0013] S4: Input the obtained global context representation into the dual decoder for feature recovery. Meanwhile, construct the dual decoder loss function, calculate the supervised loss of labeled samples, the unsupervised loss of unlabeled samples, and the consistency loss of all samples in the hyperspectral image, and output the classification result of the hyperspectral image through the classification module.

[0014] A further improvement of the present invention lies in:

[0015] The step S1 includes the following steps:

[0016] Conduct preliminary feature extraction on the acquired hyperspectral image through a 3×3 convolutional layer, a group normalization layer, and a ReLu activation function, and unify different input data into the same number of channels simultaneously.

[0017] The step S2 includes the following steps:

[0018] Input the extracted preliminary features into the multi-scale spatial-spectral feature extraction model DMS based on dilated convolution. In the DMS model, construct different-scale spatial relationships based on dilated convolution with different dilation rates to obtain different-scale spatial features;

[0019] Fuse the different-scale spatial features, input the fused spatial features into the channel-aware attention module to obtain the spectral information of the features, and weight the obtained spatial features and spectral information with the input features to make the output features the same size as the input features.

[0020] The step S3 includes the following steps:

[0021] Conduct advanced feature extraction on the feature maps of different-scale spatial information and spectral information through the Transformer model, including using three Swin Transformer Blocks, and weight the output of each layer through a 1×1 convolutional layer, a group normalization layer, a ReLu activation function, and the decoder of the corresponding layer to obtain the global context representation of the hyperspectral image, i.e., the advanced semantic features.

[0022] The step S4 includes the following steps:

[0023] The double decoder includes the combination of decoder one using bilinear interpolation and decoder two using transposed convolution for processing, to perform feature restoration on the labeled samples and unlabeled samples in the hyperspectral image;

[0024] The features output by the bilinear interpolation decoder are input into the classification module to obtain the classification result of the image.

[0025] Step S5 further includes the following steps:

[0026] The double decoder loss function includes a supervised loss, an unsupervised loss, and a double decoder consistency loss;

[0027] For the labeled samples, the output features of the bilinear interpolation decoder are input into the classification module, and the result output by the classification module and the label are used to calculate the supervised loss of the labeled samples using the cross-entropy loss function;

[0028] For the unlabeled samples, the output of each upsampling layer in the transposed convolution decoder is predicted and processed to have the same size as the input features, and the metric of each prediction result and the average value of the prediction results is used as the unsupervised loss;

[0029] For all samples, the consistency metric of the prediction results output by the bilinear interpolation decoder and the transposed convolution decoder is used as the double decoder consistency loss, and the supervised loss, the unsupervised loss, and the double decoder consistency loss are weighted as the double decoder loss.

[0030] Step S5 further includes the following steps:

[0031] The calculation of the supervised loss includes the following steps:

[0032] The classification module includes a 3×3 convolution and softmax, and the supervised loss function is calculated by formula (1):

[0033] L sup =-∑p(x)log(q(x)) (1)

[0034] In the formula, p(x) is the prediction result of sample x, and q(x) is the label of sample x;

[0035] The unsupervised loss includes the following steps:

[0036] The average value of the prediction results at different levels is calculated by formula (2):

[0037]

[0038] In the formula, p s is the prediction result of the output features of the s-th upsampling layer in decoder two;

[0039] Calculate the variance L between each scale prediction and the average prediction through formula (3). cons and use it as the consistency loss at different levels:

[0040]

[0041] In the formula, p s is the prediction result of the output feature of the s-th upsampling layer in decoder two, and p avg is the average value of the prediction results of the output features of all upsampling layers in decoder two;

[0042] Calculate the KL divergence under the average value and the prediction values at different levels through formula (4), and use it as the spatial frequency metric:

[0043]

[0044] In the formula, is the p of the j-th channel in decoder two s , is the p of the j-th channel avg ;

[0045] Calculate the unsupervised loss of the unlabeled samples through formula (5):

[0046] L unsup = L cons + L kl (5)

[0047] In the formula, L cons is the consistency loss at different scales, and L kl is the loss at different levels;

[0048] The calculation of the dual decoder loss includes the following steps:

[0049] Take the MSE function under the predicted value of the decoder with bilinear interpolation and the average predicted value of the decoder with transposed convolution as the consistency loss of the dual decoder, and calculate it through formula (6):

[0050]

[0051] In the formula, y decoder_1 is the prediction result of the output feature of decoder one, y decoder_2 is the prediction result of the output feature of decoder two, and n is the number of categories;

[0052] Calculate the dual decoder loss through formula (7):

[0053] L total = L sup + λL unsup + μLdcon (7)

[0054] In the formula, λ and μ are average weights used to control the balance between the supervised loss, the unsupervised loss, and the dual-decoder consistency loss.

[0055] A hyperspectral image classification system includes a preliminary feature extraction module, a multi-scale spatial-spectral feature extraction module, an advanced feature extraction module, and a classification result output module;

[0056] The preliminary feature extraction module is used to perform preliminary feature extraction on the acquired hyperspectral image;

[0057] The multi-scale spatial-spectral feature extraction module is used to input the extracted preliminary features into a multi-scale spatial-spectral feature extraction model to obtain feature maps containing spatial information and spectral information at different scales;

[0058] The advanced feature extraction module is used to perform advanced feature extraction on the feature maps of spatial information and spectral information at different scales to obtain the global context representation of the hyperspectral image;

[0059] The classification result output module is used to input the obtained global context representation into a dual decoder for feature recovery, and at the same time construct a dual-decoder loss function to calculate the supervised loss of the labeled samples, the unsupervised loss of the unlabeled samples, and the consistency loss of all samples in the hyperspectral image, and output the classification result of the hyperspectral image through a classification module.

[0060] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any method of the present invention are implemented.

[0061] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of any method of the present invention are implemented.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention discloses a hyperspectral image classification method. First, preliminary feature extraction is performed on the image, and the extracted preliminary features are input into a multi-scale spatial spectral feature extraction model to obtain global spatial context information, solving the problem that it is difficult to fully utilize the spectral and spatial information of hyperspectral data. Further, feature restoration is performed through a dual decoder. The two decoders perform upsampling in different ways, enabling the prediction results of the two decoders to be consistent under slightly different structures, increasing perturbations, enhancing the robustness of the model, and constructing a dual decoder loss function, improving the utilization rate of supervised information for labeled samples, and at the same time, the structural information of unlabeled samples can also be fully utilized to enhance the robustness of the network. By using a small amount of labeled data and a large amount of unlabeled data, the overfitting and low accuracy problems caused by insufficient training samples in the supervised mode are solved. The method disclosed by the present invention reduces the data calculation difficulty and improves the utilization rate of spectral information, can utilize the supervised confidence of both labeled and unlabeled data, avoids the problem of few labeled samples and insufficient data in the hyperspectral dataset, and improves the subsequent image classification accuracy.

[0064] Further, in the present invention, for unlabeled samples, predictions are made through the output of each layer of upsampling in the transposed convolution decoder and processed to be the same size as the input features, reducing the prediction differences at different levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0066] Figure 1 It is the overall structure diagram of the model of the present invention;

[0067] Figure 2 It is the structure diagram of the DMS model of the present invention;

[0068] Figure 3 It is the structure diagram of the second decoder of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0071] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not require further definition and explanation in subsequent drawings.

[0072] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper", "lower", "horizontal", "inner", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is customarily placed during use, it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention. In addition, terms such as "first", "second", etc. are only used for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0073] In addition, if the term "horizontal" appears, it does not mean that the component is required to be absolutely horizontal, but it can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but it can be slightly inclined.

[0074] In the description of the embodiments of the present invention, it should also be noted that unless otherwise clearly specified and limited, if terms such as "set", "installed", "connected", "connected" are understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0075] The following further describes the present invention in detail with reference to the accompanying drawings:

[0076] See Figure 1 , the present invention discloses a hyperspectral image classification method, specifically a semi-supervised hyperspectral image classification method based on Transformer, to solve the problems of overfitting and low classification accuracy caused by fewer labeled samples in current hyperspectral images, and can make full use of the multi-scale spatial features and spectral features of hyperspectral images to improve the classification accuracy of hyperspectral images, including the following steps:

[0077] Step 1: Preprocess the input hyperspectral image and input it into the convolutional module for preliminary feature extraction;

[0078] Preprocess the input hyperspectral image R C×H×W Since the network needs to perform multiple downsamplings later, fill H and W to multiples of 8.

[0079] Input the preprocessed image into the convolutional module for preliminary feature extraction:

[0080] The convolutional module consists of a 3×3 convolutional layer, Group Normalization (GroupNorm), and a non-linear activation function (ReLU). In this network structure, inputting the entire image into the network, using batch normalization at this time will cause a large increase in errors, so Group Normalization is used. It is independent of the batch size and can achieve good results.

[0081] Furthermore, since different hyperspectral images have different numbers of spectral channels, use the convolutional module to convert the number of channels of different hyperspectral images to a fixed size of 96 channels, unifying the number of channels while performing preliminary feature extraction on the input data.

[0082] Step 2: Input the preliminarily extracted features into the multi-scale spatial spectral feature extraction model to obtain a feature map containing different-scale spatial information and spectral information;

[0083] To fully extract the multi-scale spatial information and spectral information of the hyperspectral image, use the multi-scale spatial spectral feature extraction model to fully extract the spatial information of different scales and rich spectral information. The multi-scale spatial spectral feature extraction model is the multi-scale spatial spectral feature extraction model DMS based on dilated convolution;

[0084] See Figure 2 , the DMS uses dilated convolutions with a convolutional kernel of 3×3 and dilation rates of 1, 4, and 6 to obtain different receptive fields, and uses a parallel mode to fuse the features obtained by different convolutional kernels together to obtain spatial features X of different scales m , and the calculation formula is as follows:

[0085] X m = concat((conv2(W1,X)+b1),(conv2(W2,X)+b2),(conv2(W3,X)+b3)) (8)

[0086] The spatial features X of different scales mThe channels are restored to the same size as the input through a 1×1 convolution. Since hyperspectral images contain rich channel information, a channel-aware attention module is used to extract the spectral information of hyperspectral images. After obtaining multi-scale spatial information and spectral information, they are weighted with the input features. This structure uses dilated convolutions with different dilation rates to extract features, fully extracting spatial features at different scales. On the basis of fully extracting spatial features, 1×1 convolutions and a channel attention module are used to fully extract spectral features, effectively combining spatial features and spectral features at different scales.

[0087] The DMS module uses different dilation rates, enabling the acquisition of spatial information of different sizes using different local receptive fields. These features are combined on the channel dimension, enhancing channel information of different sizes. However, in this process, the adaptive adjustment of the receptive field size and the differences in spectral information of features extracted by different receptive fields are not considered.

[0088] The output of the multi-scale spatial-spectral feature extraction module is weighted with two decoders through a 1×1 convolutional layer, a group normalization layer, a ReLu activation function, fusing shallow semantic features and deep semantic features of the decoder.

[0089] Step 3: Input the feature map into the Transformer feature extraction module to obtain the global context representation of the hyperspectral image;

[0090] To adaptively capture the long-range relationships between high-level semantic features of hyperspectral images, the global context representation of the hyperspectral image is obtained by using the Transformer feature extraction module.

[0091] First, the input features are sliced into patches and then input into the Transformer feature extraction module. The Transformer feature extraction module consists of three Swin Transformer blocks, and the output of each layer is weighted with the output of the upsampling layer of the decoder through a 1×1 convolutional layer, a group normalization layer, and a ReLu activation function.

[0092] Furthermore, Swin Transformer is used for semi-supervised hyperspectral image classification. W-MSA in Swin Transformer reduces the computational complexity and effectively captures the local correlations between features. First, the input features are divided into non-overlapping windows, and MSA is calculated in different windows. To learn multiple semantics, the input is first converted into an embedding vector, and each embedding vector is multiplied by a pre-defined learnable weight matrix W q , W k , Wv Three weight matrices Q, K, and V are obtained. Attention scores are calculated by computing Q and K for each vector, and weights are obtained by applying softmax to the attention scores. Generally speaking, the self-attention formula can be expressed as:

[0093] Q = X m W q , K = X m W k , V = X m W v (9)

[0094]

[0095] In the formula: X m is the output of the multi-scale spatial spectral feature extraction module, which is used for normalization.

[0096] Step 4: The obtained global context representation is passed into two decoders (Decoder_1, Decoder_2) to recover the features. A dual-decoder loss function is designed to fully utilize the supervised information of the labeled samples and the unsupervised information of the unlabeled samples. A noise addition layer is added before Decoder_2 to enhance the robustness of the model. At the same time, the output features of Decoder_1 are input into the classification module to obtain the classification result of the image.

[0097] The classifier uses a multi-classification function, specifically implemented using Conv2D + softmax

[0098] For the dual-decoder module, a single decoder is usually used for supervised learning. In order to fully utilize the supervised information of the labeled samples and learn the structural information of the data using the unlabeled samples, a dual-decoder structure is adopted. The two decoders are respectively used to recover the supervised features and the unsupervised features.

[0099] Furthermore, the dual-decoder includes two Decoders. Decoder_1 and Decoder_2 have similar structures, both consisting of 3 layers. Each layer contains a convolutional layer and an upsampling layer, and shallow semantic feature fusion is inserted to fuse the output from the multi-scale spatial spectral feature module and the output from the Transformer with the decoder features to obtain richer semantic features.

[0100] Furthermore, the two Decoders have the same structure except for the different ways of upsampling.

[0101] Furthermore, the two decoders perform upsampling in different ways. Decoder_1 uses bilinear interpolation, and Decoder_2 uses transposed convolution to achieve it. The upsampling layer of Decoder_1 uses bilinear interpolation, and the upsampling layer of Decoder_2 uses transposed convolution to achieve it. The purpose is to make the prediction results of the two decoders consistent when the structures of the two decoders are slightly different. To increase the perturbation and enhance the robustness of the model, a noise addition layer is added before Decoder_2.

[0102] Furthermore, in order to utilize the supervised information of a small number of labeled samples and the unsupervised information of a large number of unlabeled samples, and enhance the robustness of the network, the two decoders are constrained by the dual-decoder loss, which includes supervised loss, unsupervised loss, and dual-decoder consistency loss.

[0103] The supervised loss:

[0104] For the labeled samples, the result output by the classification module and the label are used to calculate the supervised loss of the labeled samples by using the cross-entropy loss function. Specifically:

[0105] The output features of Decoder_1 are input into the classification module, which consists of a 3×3 convolution, a 1×1 convolution, and softmax. The supervised loss L is calculated for the labeled samples by using the cross-entropy loss function. sup The calculation formula is as follows:

[0106] L sup = -∑p(x)log(q(x)) (1)

[0107] In the formula, p(x) is the prediction result of sample x, and q(x) is the label of sample x.

[0108] The unsupervised loss:

[0109] For the unlabeled samples, the output of each upsampling layer in Decoder_2 is predicted and processed to the same size as the input. The measure of each prediction result and the average value of the prediction results is used as the unsupervised loss. Specifically:

[0110] To learn the structural information of data from unlabeled samples simultaneously, this paper adds a multi-level prediction architecture to Decoder_2 and processes the unlabeled data through the consistency of multi-level predictions. To reduce the prediction differences at different levels, a classification prediction module is added after each upsampling module of the decoder. The classification prediction module consists of a 3×3 convolution, a 1×1 convolution, an upsampling layer, and a softmax. The upsampling layer is added to adjust the predictions at different scales to the same size as the input. To effectively utilize the unlabeled data and make the prediction results at different levels of Decoder_2 tend to be consistent, a consistency loss is introduced to minimize the differences in prediction results at different levels. The average prediction value P is calculated for the prediction results at three different levels avg As shown in Equation 2, calculate the variance L between the prediction at each scale and the average prediction cons As the consistency loss at different levels, as shown in Equation 3:

[0111]

[0112] In the formula, p s is the prediction result of the output feature of the sth upsampling layer in Decoder_2

[0113]

[0114] In the formula, p s is the prediction result of the output feature of the sth upsampling layer in Decoder_2, and p avg is the average value of the prediction results of the output features of all upsampling layers in Decoder_2

[0115] Considering that the prediction results at different levels have different spatial resolutions, although the upsampling layer can be used to adjust them to the same size as the input, there are still different spatial frequencies. Directly using the variance to impose consistency cannot capture different spatial frequencies. Therefore, the KL divergence between the average value and the prediction values at different levels is used as a measure of spatial frequency uncertainty, and the calculation formula is as follows:

[0116]

[0117] In the formula, is the jth channel in Decoder_2, p s , is p of the jth channel avg

[0118] The unsupervised loss of unlabeled samples consists of the sum of L cons and L kl :

[0119] L unsup = L cons + Lkl (5)

[0120] In the formula, L cons is the consistency loss at different scales, and L kl is the loss at different levels

[0121] The consistency loss of the dual decoder is as follows:

[0122] See Figure 3 , for all samples, the consistency metric of the prediction results output by Decoder_1 and Decoder_2 is used as the consistency loss of the dual decoder; the sum of the supervised loss, the unsupervised loss, and the consistency loss of the dual decoder is used as the dual decoder loss, fully obtaining the supervised information of a small number of labeled samples and the unsupervised information of a large number of unlabeled samples. Specifically:

[0123] This network uses two decoders. Except for using different upsampling methods, the other structures of the two decoders are the same. The purpose is to make the two decoders have the same prediction results when the structures are slightly different. To increase the perturbation, a noise layer is added before Decoder_2. To enable different decoders to obtain the same predicted values, a consistency loss based on the dual decoder is introduced, and the MSE function under the predicted value of Decoder_1 and the average predicted value of Decoder_2 is used as the consistency loss of the dual decoder. The calculation formula is as follows:

[0124]

[0125] In the formula, y decoder_1 is the prediction result of the output feature of decoder one, and y decoder_2 is the prediction result of the output feature of decoder 2, and n is the number of categories.

[0126] The total loss function is composed of the sum of the supervised loss, the unsupervised loss, and the consistency loss of the dual decoder. The calculation formula is as follows:

[0127] L total = L sup + λL unsup + μL dcon (7).

[0128] In the formula, L sup is the supervised loss calculated by formula 1, L unsup is the unsupervised loss obtained from formula (5), and L dcon is the consistency loss of the dual decoder obtained from formula (6). λ and μ are weights used to control the balance between the supervised loss, the unsupervised loss, and the consistency loss of the dual decoder.

[0129] Method disclosed by the present invention. The present invention solves the problems of overfitting and low accuracy caused by insufficient training samples in the supervised mode by using a small amount of labeled data and a large amount of unlabeled data; a multi-scale spatial-spectral feature extraction module is used to extract spatial and spectral features at different scales, and at the same time, a Transformer feature extraction module is introduced to extract global spatial context information, solving the problem that it is difficult to fully utilize the spectral and spatial information of hyperspectral data.

[0130] An embodiment of the present invention also discloses a hyperspectral image classification system, including a preliminary feature extraction module, a spectral feature extraction module, an advanced feature extraction module, and a classification result output module;

[0131] The preliminary feature extraction module is used to perform preliminary feature extraction on the acquired hyperspectral image;

[0132] The spectral feature extraction module is used to input the extracted preliminary features into a multi-scale spatial-spectral feature extraction model to obtain a feature map containing spatial information and spectral information at different scales;

[0133] The advanced feature extraction module is used to perform advanced feature extraction on the feature map of spatial information and spectral information at different scales to obtain a global context representation of the hyperspectral image;

[0134] The classification result output module is used to input the obtained global context representation into a dual decoder for feature restoration, and at the same time construct a dual decoder loss function, calculate the supervised loss of labeled samples and unlabeled samples in the hyperspectral image, and output the classification result of the hyperspectral image.

[0135] Schematic diagram of a terminal device provided by an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0136] The computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention.

[0137] The terminal device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory.

[0138] The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0139] The memory can be used to store the computer program and / or modules. By running or executing the computer program and / or modules stored in the memory, and by invoking the data stored in the memory, the processor implements various functions of the terminal device.

[0140] If the modules / units integrated in the terminal device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0141] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method, characterized in that, It includes the following steps: S1: Conduct preliminary feature extraction on the acquired hyperspectral image; S2: Input the extracted preliminary features into the multi-scale spatial spectral feature extraction model to obtain feature maps containing different-scale spatial information and spectral information; S3: Conduct advanced feature extraction on the feature maps of different-scale spatial information and spectral information to obtain the global context representation of the hyperspectral image; S4: Input the obtained global context representation into the dual decoder for feature recovery, and at the same time construct the dual decoder loss function to calculate the supervised loss of the labeled samples, the unsupervised loss of the unlabeled samples, and the consistency loss of all samples in the hyperspectral image, and output the classification result of the hyperspectral image through the classification module; The step S2 includes the following steps: Input the extracted preliminary features into the multi-scale spatial spectral feature extraction model DMS based on dilated convolution. In the DMS model, construct different-scale spatial relationships based on dilated convolution with different dilation rates to obtain different-scale spatial features; Fuse the different-scale spatial features, input the fused spatial features into the channel-aware attention module to obtain the spectral information of the features, and weight the obtained spatial features and spectral information with the input features to make the output features the same size as the input features; The step S3 includes the following steps: Conduct advanced feature extraction on the feature maps of different-scale spatial information and spectral information through the Transformer model, including using three Swin Transformer Blocks, and weight the output of each layer through the 1×1 convolutional layer, group normalization layer, ReLu activation function, and the decoder of the corresponding layer to obtain the global context representation of the hyperspectral image, that is, the advanced semantic features; The step S4 includes the following steps: The dual decoder includes the combination of decoder one using bilinear interpolation and decoder two using transposed convolution to process the feature recovery of the labeled samples and unlabeled samples in the hyperspectral image; Input the features output by the bilinear interpolation decoder into the classification module to obtain the classification result of the image; The step S4 further includes the following steps: The dual decoder loss function includes supervised loss, unsupervised loss, and dual decoder consistency loss; For the labeled samples, input the output features of the bilinear interpolation decoder into the classification module, and calculate the supervised loss of the labeled samples by using the cross-entropy loss function for the result output by the classification module and the label; For the unlabeled samples, predict the output of the upsampling of each layer in the transposed convolution decoder and process it to be the same size as the input features, and use the measure of each prediction result and the average value of the prediction results as the unsupervised loss; For all samples, use the consistency measure of the prediction results output by the bilinear interpolation decoder and the transposed convolution decoder as the dual decoder consistency loss, and weight the supervised loss, unsupervised loss, and dual decoder consistency loss as the dual decoder loss.

2. The hyperspectral image classification method according to claim 1, characterized in that, The step S1 includes the following steps: The obtained hyperspectral image is subjected to preliminary feature extraction through a 3×3 convolutional layer, a group normalization layer, and a ReLu activation function, and at the same time, different input data are unified into the same number of channels.

3. A hyperspectral image classification method according to claim 1, characterized in that, The step S4 further includes the following steps: The calculation of the supervised loss includes the following steps: The classification module includes a 3×3 convolution and softmax, and the supervised loss function is calculated by formula (1): In the formula, is the sample prediction result, is the label of the sample ; The unsupervised loss includes the following steps: Calculate the average value of the prediction results at different levels through formula (2): In the formula, is the prediction result of the output feature of the s-th upsampling layer in the second decoder; Calculate the variance between each scale prediction and the average prediction through formula (3) , and use it as the consistency loss at different levels: wherein, is the prediction result of the output feature of the s-th upsampling layer in the second decoder, is the average value of the prediction results of the output features of all upsampling layers in the second decoder; Calculate the KL divergence between the calculated average value and the prediction values at different levels through formula (4), and use it as the spatial frequency metric: In the formula, is the j-th channel in the second decoder , is the of the j-th channel; Calculate the unsupervised loss of the unlabeled samples through formula (5): wherein, is the consistency loss at different scales, is the loss at different levels; The calculation of the dual decoder loss includes the following steps: Take the MSE function between the predicted value of the bilinear interpolation decoder and the average predicted value of the transposed convolution decoder as the consistency loss of the dual decoder, and calculate it through formula (6): wherein, is the prediction result of the output feature of decoder one, is the prediction result of the output feature of decoder two, and n is the number of categories; Calculate the dual decoder loss through formula (7): In the formula, and are uniform weights used to control the balance between the supervised loss, the unsupervised loss, and the dual-decoder consistency loss.

4. A hyperspectral image classification system for the method according to claim 1, characterized in that, It includes a preliminary feature extraction module, a multi-scale spatial-spectral feature extraction module, an advanced feature extraction module, and a classification result output module; The preliminary feature extraction module is used to perform preliminary feature extraction on the obtained hyperspectral image; The multi-scale spatial-spectral feature extraction module is used to input the extracted preliminary features into the multi-scale spatial-spectral feature extraction model to obtain feature maps containing different-scale spatial information and spectral information; The advanced feature extraction module is used to perform advanced feature extraction on the feature maps of different-scale spatial information and spectral information to obtain the global context representation of the hyperspectral image; The classification result output module is used to input the obtained global context representation into the dual decoder for feature recovery, and at the same time construct a dual decoder loss function, calculate the supervised loss of the labeled samples, the unsupervised loss of the unlabeled samples, and the consistency loss of all samples in the hyperspectral image, and output the classification result of the hyperspectral image through the classification module.

5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1-3.

Images