A hyperspectral image classification method and system based on 3DCutMix-Transformer

Abstract

The present invention discloses a hyperspectral image classification method and system based on 3D CutMix-Transformer, which divides hyperspectral data into labeled training data sets and labeled verification data sets; pre-trains 3D CutMix through CNN; uses the training data sets to perform 3D CutMix to obtain enhanced data to train a regional teacher model and a sample-level teacher model; and uses two teacher models and a small amount of labeled data sets to jointly train a student model. The present invention uses CNN to perform 3D CutMix pre-training, then uses 3D CutMix to perform data enhancement on the original labeled data set, and optimizes the self-supervised loss and mutual cross-pseudo-supervised loss of the two teacher models, so that the jointly trained student model has better robustness, enhances and improves the generalization ability and accuracy of the model under small samples in the existing hyperspectral image classification technology, and can be used for hyperspectral image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral remote sensing image classification, and in particular relates to a hyperspectral image classification method and system based on 3D CutMix-Transformer. Background Art

[0002] With the development of artificial intelligence technology, hyperspectral image intelligent classification technology has deeply influenced all aspects of modern life. Its application in precision agriculture, military, ocean, disaster detection and other fields has become more and more extensive. Traditional remote sensing image analysis uses image spatial information, and the core of hyperspectral image analysis is spectral analysis. Hyperspectral remote sensing data is a spectral image cube, such as Figure 1 As shown in the figure, its most important feature is that it combines the image space dimension and spectral dimension information into one. Compared with a single band, it has an additional dimension of spectral information. When acquiring the surface space image, the spectral information of the ground object corresponding to each pixel will be obtained. Therefore, hyperspectral image classification has the advantage of high information richness compared to traditional remote sensing image classification, which means that more features can be learned through hyperspectral data to improve classification accuracy for intelligent image classification.

[0003] At present, hyperspectral image classification has the problem of few labeled samples and poor model robustness. Data-driven deep learning methods will cause the model accuracy and robustness of hyperspectral image classification to be greatly affected by the data set when there are few samples. That is, existing methods such as CNN (Convolutional Neural Network) are inefficient and have poor robustness when there are few training samples, and the effect on cross-datasets (obtained from different sensors) is very poor. The above-mentioned problems have not been effectively solved at present. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a hyperspectral image classification method and system based on 3DCutMix-Transformer in view of the deficiencies in the above-mentioned prior art, use 3D CutMix to enhance the hyperspectral image data to expand the data set, and use a one-dimensional Transformer model to learn the features of the hyperspectral image, thereby improving the classification accuracy and solving the problems of small number of labeled samples and poor model robustness in the existing hyperspectral image classification technology.

[0005] The present invention adopts the following technical solutions:

[0006] A hyperspectral image classification method based on 3D CutMix-Transformer includes the following steps:

[0007] S1, divide the hyperspectral data into labeled training data set Train and labeled verification data set Test;

[0008] S2, build a CNN convolutional neural network, input the labeled training data set Train divided in step S1 into the CNN convolutional neural network, and build a 3D CutMix pre-training model based on CNN;

[0009] S3. Build a regional-level teacher model M for hyperspectral image classification based on the 3D CutMix pre-trained model in step S2 RL and the sample-level teacher model M SL ;

[0010] S4, the hyperspectral image classification regional level teacher model M constructed by integrating step S3 RL and the sample-level teacher model M SL , and the labeled training data set Train divided in step S1 jointly train the student model M S Then, the labeled verification data set Test divided in step S1 is input into the trained student model M S In this paper, hyperspectral image classification based on Transformer teacher-student model is implemented.

[0011] Specifically, in step S1, the labeled training data set Train accounts for 80% of the hyperspectral data, and the labeled verification data set Test accounts for 20% of the hyperspectral data.

[0012] Specifically, step S2 is as follows:

[0013] S201, build CNN convolutional neural network;

[0014] S202, inputting the labeled training data set Train divided in step S1 into the CNN convolutional neural network to obtain the classification result of each pixel element;

[0015] S203, performing contribution analysis on the classification result of each pixel element obtained in step S202 and the band data it contains, and obtaining the band with the highest contribution rate, following the principle of selecting no more than 5% of the number of bands from the candidates;

[0016] S204, designing 3D CutMix according to the several bands with the highest contribution rate to the classification result of each pixel element obtained in step S203, obtaining a 3D Mask of the hyperspectral data, and finally obtaining a pre-trained 3D Mask pre-training model.

[0017] Furthermore, in step S201, the input layer network input of the CNN convolutional neural network uses standard data in mat format, and the original data in the format of (L×W)×H is decentralized and standardized, and the output result is F×H as the input of the first convolutional layer, F=L×W, L is the number of pixels on the long side of the hyperspectral data, W is the number of pixels on the wide side of the hyperspectral data, H is the number of bands of the hyperspectral data, and F is the total number of pixels of the hyperspectral data; the weight of the CNN input layer is constrained to 0~1, if If the input weight is greater than 0.5, it is 1, otherwise it is 0;

[0018] The convolution layer uses a 4x4 convolution kernel / filter, and the convolution kernel slides one pixel at a time. A feature map uses the same convolution kernel, and the activation function uses the Relu function.

[0019] Furthermore, in step S202, each hyperspectral image data is represented as The j-th pixel, j=1, 2, ..., F, i=1, 2, ..., H, the corresponding label is L (j) , the multi-classification cross entropy loss function is as follows:

[0020]

[0021] Where p=[p0,...,p C-1 ] is a probability distribution, each element p i Represents the probability that the sample belongs to the i-th category; y=[y0,...,y C-1 ] is the one-hot representation of the sample label. When the sample belongs to the i-th category, y i =1, otherwise y i =0, C is the number of sample labels.

[0022] Specifically, step S3 is as follows:

[0023] S301, randomly select two pixel elements X in the hyperspectral data (m) and X (n) Perform 3D CutMix operation to get X_Cut (m) and X_Cut (n) ;

[0024] S302: Repeat step S301 N times to obtain original hyperspectral data Enhanced data obtained through 3D CutMix operation j=1, 2,..., F, i=1, 2,..., H;

[0025] S303: The enhanced data obtained in step S302 is Enter the regional teacher model MRL , after a one-dimensional Transformer, the pseudo label L1 is obtained (j) ;

[0026] S304, the original hyperspectral data Input sample-level teacher model M SL , after a one-dimensional Transformer, the pseudo label L2 is obtained (j) ;

[0027] S305: respectively replace the pseudo labels L1 in step S303 (j) The pseudo L2 of step S304 (j) Perform self-supervision loss with the true label and calculate the pseudo label L1 at the same time (j) With pseudo label L2 (j) The cross pseudo-supervision loss between the region-level teacher model M RL The objective function L RL and the sample-level teacher model M SL The objective function L SL .

[0028] Further, in step S305, the regional teacher model M RL The objective function L RL and the sample-level teacher model M SL The objective function L SL They are:

[0029] arg min(L RL )=arg min{L ce (L1,L)+L ce (L1,L2)}

[0030] arg min(L SL )=arg min{L ce (L2,L)+L ce (L1,L2)}

[0031] Among them, L RL , L SL Represents the regional teacher model M RL and the sample-level teacher model M SL The objective function, L ce represents the cross entropy loss function, L ce (L1,L) and L ce (L1,L2) represents the self-supervision loss, L ce (L1,L2) represents M RL The obtained pseudo label is consistent with M SL The cross pseudo-supervision loss between the obtained pseudo-labels.

[0032] Specifically, step S4 is as follows:

[0033] S401, regional teacher model M RL and the sample-level teacher model M SL Perform an integration operation and average the output of the two complementary models to be M mean ;

[0034] S402, Regional Teacher Model M RL and the sample-level teacher model M SL The output of is integrated as guidance to supervise the student model M S Training on unlabeled target data.

[0035] Specifically, in step S401, the regional teacher model M RL and the sample-level teacher model M SL The integration is expressed as:

[0036]

[0037] Among them, E() represents the average regional teacher model M RL and the sample-level teacher model M SL The output, x u represents unlabeled samples, q represents the probability of each category output, and μ represents the weight.

[0038] Another technical solution of the present invention is a hyperspectral image classification system based on 3D CutMix-Transformer, comprising:

[0039] The data module divides the hyperspectral data into labeled training dataset Train and labeled verification dataset Test;

[0040] Build the module, build the CNN convolutional neural network, input the labeled training data set Train divided by the data module into the CNN convolutional neural network, and build a 3D CutMix pre-training model based on CNN;

[0041] Enhancement module, based on the 3D CutMix pre-trained model of the building module to build a regional-level teacher model M for hyperspectral image classification RL and the sample-level teacher model M SL ;

[0042] Classification module, hyperspectral image classification regional level teacher model M constructed by integrating enhancement modules RL and the sample-level teacher model M SL , and the labeled training data set Train divided by the data module jointly train the student model M S, and then input the labeled verification data set Test divided by the data module into the trained student model M S In this paper, hyperspectral image classification based on Transformer teacher-student model is implemented.

[0043] Compared with the prior art, the present invention has at least the following beneficial effects:

[0044] The present invention discloses a hyperspectral image classification method based on 3D CutMix-Transformer, which pre-trains 3D CutMix through CNN; uses a training data set to perform 3D CutMix to obtain enhanced data to train a regional teacher model and a sample-level teacher model; uses two teacher models and a small amount of labeled data sets to jointly train a student model; uses 3D CutMix to enhance the hyperspectral data, solves the problem of insufficient training samples, and improves and enhances the accuracy and robustness of the model; at the same time, the global feature extraction capability and parallel computing capability of Transformer make the feature extraction capability of Transformer more comprehensive than that of CNN which can only extract local information, and can fully utilize the computing power of GPU.

[0045] Furthermore, the data set is divided, 80% of the data is used for training, and the other 20% of the data is used for validation. This 20% does not participate in training, but is used to evaluate the performance of the model. The performance of the parameters is verified in continuous "training-validation" iterations to achieve the purpose of selecting the correct hyperparameters.

[0046] Furthermore, a CNN-based 3D CutMix pre-training model was constructed. The characteristics of CNN, which is easy to build and has high accuracy, were used to pre-train hyperspectral data to obtain the contribution rate of each band in each pixel. Then, the top 5% of the bands with the highest contribution rate were used to obtain the 3D Mask of the hyperspectral data. The 5% number was selected to ensure that significant features were removed while ensuring the distinguishability between pixels to improve the robustness of the subsequent classification model, while also achieving the enhancement of hyperspectral data.

[0047] Furthermore, a classification model module is built: the Transformer network structure. In the Transformer, the attention mechanism is used for parallel computing, which speeds up the calculation speed.

[0048] Furthermore, the cross entropy loss function is used in the classification network. According to the results, when the mean square error loss function is used, the gradient is proportional to the gradient of the activation function. However, for the sigmoid activation function, the input is z and the output is σ(z). When σ(z) is very large, σ′(z) is very small, almost 0, resulting in slow parameter updates. However, for the cross entropy loss function, the gradient at this time is independent of the activation function, and the greater the gap between the predicted result σ(z) and the true result y, the greater the gradient, the faster the parameter update. When the gap between the predicted result and the true result is very small, the gradient becomes smaller, and the parameter is basically in a stable state. Using the cross entropy loss function brings two benefits.

[0049] Furthermore, the hyperspectral data is enhanced by 3D Mask to obtain enhanced data, which is then used to train two one-dimensional Transformer teacher models. The teacher model trained with the enhanced data has strong robustness, and the credibility of the data pseudo-labels obtained by the two teacher models is also high.

[0050] Furthermore, we set the regional teacher model M RL The objective function L RL and the sample-level teacher model M SL The objective function L SL The two objective functions include both the self-supervision loss of the pseudo-labels obtained by the two teacher models, and the cross-pseudo-supervision loss between the pseudo-labels obtained by the two teacher models. This allows the two teacher models to supervise each other during training and continuously approach the true labels, giving them strong guidance capabilities.

[0051] Furthermore, the regional teacher model M RL and the sample-level teacher model M SL The output is averaged to get M mean , and use it as the student model M S The guidance of training on unlabeled target data. The average processing well integrates the output results of the two teacher models, that is, the guidance of the student model has strong reliability, thereby improving the training accuracy of the student model on unlabeled target data.

[0052] Furthermore, in order to transform the regional teacher model M RL and the sample-level teacher model M SL The output of the regional teacher model M is integrated and ensured to have high accuracy. RL and the sample-level teacher model M SL The integrated setting makes the student model M S It can be strongly guided by two teacher models and thus perform well on unlabeled datasets.

[0053] In summary, the present invention integrates 3D CutMix technology into the teacher-student model for hyperspectral image classification through a deep learning method, thereby achieving accurate classification of hyperspectral images.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the hyperspectral data structure;

[0056] Figure 2 This is a training diagram of the CNN model;

[0057] Figure 3 Model M for regional teachers RL and the sample-level teacher model M SL Training diagram of

[0058] Figure 4 Schematic diagram of 3D CutMix operation.

[0059] Figure 5 This is the structural diagram of the Transformer model.

[0060] Figure 6 Model M for students S Training diagram of . DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0063] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0064] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship.

[0065] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0066] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0067] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0068] The present invention provides a hyperspectral image classification method based on 3D CutMix-Transformer, which divides the hyperspectral data into a labeled training data set and a labeled verification data set; pre-trains 3D CutMix through CNN (Convolutional Neural Network); uses the training data set to perform 3D CutMix to obtain enhanced data to train a regional teacher model and a sample-level teacher model; and uses two teacher models and a small amount of labeled data sets to jointly train a student model. The present invention uses CNN to perform 3D CutMix pre-training, then uses 3D CutMix to perform data enhancement on the original labeled data set, and optimizes the self-supervision loss and mutual cross pseudo-supervision loss of the two teacher models, so that the jointly trained student model has better robustness, enhances and improves the generalization ability and accuracy of the model under small samples in the existing hyperspectral image classification technology, and can be used for hyperspectral image classification.

[0069] The present invention provides a hyperspectral image classification method based on 3D CutMix-Transformer, comprising the following steps:

[0070] S1, divide the hyperspectral data into labeled training data set Train and labeled verification data set Test;

[0071] The hyperspectral data in the format of (L×W)×H is converted to F×H, where F=L×W, L is the number of pixels on the long side of the hyperspectral data, W is the number of pixels on the wide side of the hyperspectral data, H is the number of bands of the hyperspectral data, and F is the total number of pixels of the hyperspectral data. Considering the classification of multidimensional data into one-dimensional data, the training efficiency of the model will be greatly improved. 80% of the data is divided into a labeled training data set Train, and the remaining 20% ​​of the data set is divided into a labeled verification data set Test.

[0072] For example, when the input hyperspectral data set is the Indian Pines data set, L = 145, W = 145, H = 200, then F = L × W = 21025. At this time, the hyperspectral data in the format of (145 × 145) × 200 is converted into one-dimensional data 21025 × 200, 80% of which is divided into a labeled training data set Train (16820 × 200), and the remaining 20% ​​of the data set is divided into a labeled verification data set Test (4205 × 200).

[0073] S2, build a 3D CutMix pre-trained model based on CNN convolutional neural network;

[0074] S201, such as Figure 2 Build CNN as shown;

[0075] The input layer (INPUT) network input uses standard data in mat format, and the original data (L×W)×H is decentralized and standardized, and the output result F×H is used as the input of the first convolutional layer.

[0076] At the same time, the CNN input layer weights are constrained to 0 to 1, that is, if The input weight is 1 if it is greater than 0.5, otherwise it is 0.

[0077] The convolution layers all use 4x4 convolution kernels / filters, and the convolution kernel slides one pixel each time (stride = 1), and one feature map uses the same convolution kernel. In addition, the activation function uses the Relu function.

[0078] S202, inputting the labeled training data set Train divided in step S1 into CNN to obtain the classification result of each pixel element;

[0079] Each hyperspectral image data is represented as The jth pixel, j = 1, 2, ..., F, i = 1, 2, ..., H, its corresponding label is L (j) , and the multi-classification cross entropy loss function is used as follows:

[0080]

[0081] Where p=[p0,...,p C-1 ] is a probability distribution, each element p i Represents the probability that the sample belongs to the i-th category; y=[y0,...,y C-1 ] is the one-hot representation of the sample label. When the sample belongs to the i-th category, y i =1, otherwise y i =0, C is the number of sample labels.

[0082] S203, performing contribution rate analysis on the classification result of each pixel element obtained in step S202 and the band data contained therein, and obtaining several bands with the highest contribution rates;

[0083] The method for selecting the band with the highest contribution rate of each pixel element to the classification result is: constrain the weight of the CNN input layer from 0 to 1. After the training, the band with a weight of 1 is the candidate for the band with the highest contribution rate to the classification result. The number of bands with the highest contribution rate of each pixel element to the classification result is selected according to the principle of selecting no more than 5% of the number of bands from the candidates.

[0084] S204, designing 3D CutMix according to the several bands with the highest contribution rate to the classification result of each pixel element obtained in step S203, and obtaining a 3D Mask of the hyperspectral data, such as Figure 4 shown.

[0085] Taking the Indian Pines dataset as an example, first initialize the 0-1 Mask matrix j=1,2,…,21025,i=1,2,…,200,then according to step S203, the jth pixel element in the hyperspectral data is obtained. Select the k bands with the highest contribution to the classification results from the 200 bands, k≤21025×5%, and then The corresponding k positions are set to 0, and finally the pre-trained 3D Mask pre-trained model is obtained.

[0086] S3. Construct a regional-level teacher model M for hyperspectral image classification based on 3D CutMix-Transformer RL and the sample-level teacher model M SL ;

[0087] See also Figure 3 , a regional-level teacher model for hyperspectral image classification based on 3D CutMix-Transformer RL and the sample-level teacher model M SL Specifically:

[0088] S301, such as Figure 4 As shown, two pixels X are randomly selected from the hyperspectral data. (m) and X (n) Perform 3D CutMix operation to get X_Cut (m) and X_Cut (n) ;

[0089] 3D CutMix is ​​described as follows:

[0090] X_Cut (m) =M (m) ⊙X m +(1-M (m) )⊙X n

[0091] X_Cut (n) =M (m) ⊙X n +(1-M (m) )⊙X m

[0092] Where M is the 3D Mask matrix, ⊙ is the matrix element multiplication, X_Cut (m)and X_Cut (n) They are the original hyperspectral data X (m) and X (n) Enhanced data obtained after 3D CutMix operation.

[0093] S302: Repeat step S301 N times to obtain original hyperspectral data Enhanced data obtained through 3D CutMix operation j=1, 2,..., F, i=1, 2,..., H;

[0094] S303: The enhanced data obtained in step S302 is Enter the regional teacher model M RL , after a one-dimensional Transformer, the pseudo label L1 is obtained (j) ;

[0095] S304, the original hyperspectral data Input sample-level teacher model M SL , after a one-dimensional Transformer, the pseudo label L2 is obtained (j) ;

[0096] S305, respectively set the pseudo labels L1 (j) With pseudo label L2 (j) Perform self-supervision loss with the true label and calculate the pseudo label L1 at the same time (j) With pseudo label L2 (j) The cross pseudo-supervision loss between .

[0097] The Transformer structure used is as follows Figure 5 As shown, the regional teacher model M RL and the sample-level teacher model M SL The objective function L RL and L SL They are:

[0098] arg min(L RL )=arg min{L ce (L1,L)+L ce (L1,L2)}

[0099] arg min(L SL )=arg min{L ce (L2,L)+L ce (L1,L2)}

[0100] Among them, L RL , L SL Represents the regional teacher model M RL and the sample-level teacher model MSL The objective function, L ce represents the cross entropy loss function, L ce (L1,L) and L ce (L1,L2) represents the self-supervision loss, L ce (L1,L2) represents M RL The pseudo label L1 obtained (j) With M SL The pseudo label L2 obtained (j) Cross pseudo supervision between .

[0101] S4, the hyperspectral image classification regional level teacher model M constructed by integrating step S3 RL and the sample-level teacher model M SL , and the labeled training data set Train divided in step S1 jointly train the student model M S Then, the labeled verification data set Test divided in step S1 is input into the trained student model M S In this paper, hyperspectral image classification based on Transformer teacher-student model is implemented.

[0102] S401, regional teacher model M RL and the sample-level teacher model M SL Perform an integration operation and average the output of the two complementary models to be M mean ;

[0103] Regional Teacher Model M RL and the sample-level teacher model M SL The integration (knowledge distillation) is expressed as:

[0104]

[0105] Among them, E() means taking the average of the two teachers’ outputs, x u Represents a large number of unlabeled samples, q represents the probability of each category output, and μ represents the weight.

[0106] S402, Regional Teacher Model M RL and the sample-level teacher model M SL The output of is integrated as a stronger guide to supervise the student model M S is trained on unlabeled target data. In addition, the student model M S It is also supervised by a small amount of ground-truth labeled labels.

[0107] Student Model M S By minimizing M RL 、M SL The KL-divergence between the outputs of the two models is used to obtain knowledge.S Also received a small amount of label Y l supervision.

[0108] Student Model M S The objective function L S for:

[0109] L S =λ kl L KL (E(M RL (X u ),M SL (X u )))+λ ce L ce (M S (X l ),Y l )

[0110] Among them, L KL represents the KL divergence loss, λ kl and λ ce are the KL divergence weight and the cross entropy loss weight respectively. u is a large amount of unlabeled data; see Figure 6 .

[0111] In another embodiment of the present invention, a hyperspectral image classification system based on 3D CutMix-Transformer is provided, which can be used to implement the above-mentioned hyperspectral image classification method based on 3D CutMix-Transformer. Specifically, the hyperspectral image classification system based on 3D CutMix-Transformer includes a data module, a building module, an enhancement module and a classification module.

[0112] Among them, the data module divides the hyperspectral data into labeled training dataset Train and labeled verification dataset Test;

[0113] Build the module, build the CNN convolutional neural network, input the labeled training data set Train divided by the data module into the CNN convolutional neural network, and build a 3D CutMix pre-training model based on CNN;

[0114] Enhancement module, based on the 3D CutMix pre-trained model of the building module to build a regional-level teacher model M for hyperspectral image classification RL and the sample-level teacher model M SL ;

[0115] Classification module, hyperspectral image classification regional level teacher model M constructed by integrating enhancement modules RL and the sample-level teacher model MSL , and the labeled training data set Train divided by the data module jointly train the student model M S , and then input the labeled verification data set Test divided by the data module into the trained student model M S In this paper, hyperspectral image classification based on Transformer teacher-student model is implemented.

[0116] In summary, the present invention provides a hyperspectral image classification method and system based on 3D CutMix-Transformer, which pre-trains 3D CutMix through CNN; uses the training data set to perform 3D CutMix to obtain enhanced data to train the regional-level teacher model and the sample-level teacher model; uses the two teacher models and a small amount of labeled data sets to jointly train the student model; uses 3D CutMix to enhance the hyperspectral data, which solves the problem of insufficient training samples and improves and enhances the accuracy and robustness of the model; at the same time, the global feature extraction capability and parallel computing capability of Transformer make the feature extraction capability of Transformer more comprehensive than that of CNN that can only extract local information, and can fully utilize the computing power of GPU.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A hyperspectral image classification method based on 3D CutMix-Transformer, characterized in that: The following steps are involved: S1, divide the hyperspectral data into a labeled training dataset Train and a labeled verification dataset Test; S2, build a CNN convolutional neural network, input the labeled training data set Train divided in step S1 into the CNN convolutional neural network, and build a 3D CutMix pre-training model based on CNN; S3. Build a regional-level teacher model for hyperspectral image classification based on the 3D CutMix pre-trained model in step S2 and sample-level teacher model , specifically: S301, randomly select two pixel elements in the hyperspectral data and Perform 3D CutMix operation to obtain and ; S302: Repeat step S301 N times to obtain original hyperspectral data Enhanced data obtained through 3D CutMix operation , =1,2,…,F, =1, 2, ..., H, H is the number of bands of hyperspectral data, and F is the total number of pixels of hyperspectral data; S303: The enhanced data obtained in step S302 is Enter the District-Level Teacher Model , and get the pseudo label through a one-dimensional Transformer ; S304, the original hyperspectral data Input sample-level teacher model , after a one-dimensional Transformer to get a pseudo label ; S305: respectively assign the pseudo labels of step S303 to The pseudo Perform self-supervisory loss with the true label and calculate the pseudo label at the same time With pseudo labels Cross pseudo-supervision loss between the region-level teacher model The objective function and sample-level teacher model The objective function , regional teacher model The objective function and sample-level teacher model The objective function They are: in, , Represents the regional teacher model and sample-level teacher model The objective function is represents the cross entropy loss function, and represents the self-supervised loss, express The pseudo labels obtained are The cross pseudo-supervision loss between the obtained pseudo-labels; S4, regional-level teacher model for hyperspectral image classification constructed by integrating step S3 and sample-level teacher model , and the labeled training data set Train divided in step S1 to jointly train the student model , and then input the labeled verification data set Test divided in step S1 into the trained student model In the implementation, hyperspectral image classification based on the Transformer teacher-student model is implemented as follows: S401, regional teacher model and sample-level teacher model Perform an integration operation and average the output of the two complementary models to be , regional teacher model and sample-level teacher model The integration is expressed as: Among them, E() represents the average regional teacher model and sample-level teacher model The output, represents unlabeled samples, represents the probability of each category output, represents weight; S402, Regional Teacher Model and sample-level teacher model The output of is integrated as guidance to supervise the student model Training on unlabeled target data.

2. The hyperspectral image classification method based on 3D CutMix-Transformer according to claim 1, characterized in that: In step S1, the labeled training data set Train accounts for 80% of the hyperspectral data, and the labeled verification data set Test accounts for 20% of the hyperspectral data.

3. The hyperspectral image classification method based on 3D CutMix-Transformer according to claim 1, characterized in that: Step S2 is specifically as follows: S201, build CNN convolutional neural network; S202, inputting the labeled training data set Train divided in step S1 into the CNN convolutional neural network to obtain the classification result of each pixel element; S203, performing contribution analysis on the classification result of each pixel element obtained in step S202 and the band data it contains, and obtaining the band with the highest contribution rate, following the principle of selecting no more than 5% of the number of bands from the candidates; S204, designing 3DCutMix according to the several bands with the highest contribution rate to the classification result of each pixel element obtained in step S203, obtaining a 3D Mask of the hyperspectral data, and finally obtaining a pre-trained 3D Mask pre-training model.

4. The hyperspectral image classification method based on 3D CutMix-Transformer according to claim 3, characterized in that: In step S201, the input layer network input of the CNN convolutional neural network uses standard data in mat format, and the original data in the format of (L×W)×H is decentralized and standardized, and the output result is F×H as the input of the first convolutional layer, F=L×W, L is the number of pixels on the long side of the hyperspectral data, W is the number of pixels on the wide side of the hyperspectral data, H is the number of bands of the hyperspectral data, and F is the total number of pixels of the hyperspectral data; the weight of the CNN input layer is constrained to 0~1. If If the input weight is greater than 0.5, it is 1, otherwise it is 0; The convolution layer uses a 4x4 convolution kernel / filter, and the convolution kernel slides one pixel at a time. A feature map uses the same convolution kernel, and the activation function uses function.

5. The hyperspectral image classification method based on 3D CutMix-Transformer according to claim 4, characterized in that: In step S202, each hyperspectral image data is represented as , No. Pixels, =1,2,…,F, =1, 2, ..., H, the corresponding labels are , the multi-classification cross entropy loss function is as follows: in, is a probability distribution, each element Represents the probability that the sample belongs to the i-th category; is the one-hot representation of the sample label. When the sample belongs to the i-th category ,otherwise , is the number of sample labels.

6. A hyperspectral image classification system based on 3D CutMix-Transformer, characterized in that: include: The data module divides the hyperspectral data into labeled training dataset Train and labeled verification dataset Test; Build the module, build the CNN convolutional neural network, input the labeled training data set Train divided by the data module into the CNN convolutional neural network, and build a 3D CutMix pre-training model based on CNN; Enhancement module, builds a regional teacher model for hyperspectral image classification based on the 3D CutMix pre-trained model of the building module and sample-level teacher model , specifically: S301, randomly select two pixel elements in the hyperspectral data and Perform 3D CutMix operation to obtain and ; S302: Repeat step S301 N times to obtain original hyperspectral data Enhanced data obtained through 3D CutMix operation , =1,2,…,F, =1, 2, ..., H, H is the number of bands of hyperspectral data, and F is the total number of pixels of hyperspectral data; S303: The enhanced data obtained in step S302 is Enter the District-Level Teacher Model , and get the pseudo label through a one-dimensional Transformer ; S304, the original hyperspectral data Input sample-level teacher model , and get the pseudo label through a one-dimensional Transformer ; S305: respectively assign the pseudo labels of step S303 to The pseudo Perform self-supervisory loss with the true label and calculate the pseudo label at the same time With pseudo labels Cross pseudo-supervision loss between the region-level teacher model The objective function and sample-level teacher model The objective function , regional teacher model The objective function and sample-level teacher model The objective function They are: in, , Represents the regional teacher model and sample-level teacher model The objective function is represents the cross entropy loss function, and represents the self-supervised loss, express The pseudo labels obtained are The cross pseudo-supervision loss between the obtained pseudo-labels; Classification module, a hyperspectral image classification regional teacher model built by integrating enhancement modules and sample-level teacher model , and the labeled training data set Train divided by the data module to jointly train the student model , and then input the labeled verification data set Test divided by the data module into the trained student model In the implementation, hyperspectral image classification based on the Transformer teacher-student model is implemented as follows: S401, regional teacher model and sample-level teacher model Perform an integration operation and average the output of the two complementary models to be , regional teacher model and sample-level teacher model The integration is expressed as: Among them, E() represents the average regional teacher model and sample-level teacher model The output, represents unlabeled samples, represents the probability of each category output, represents weight; S402, Regional Teacher Model and sample-level teacher model The output of is integrated as guidance to supervise the student model Training on unlabeled target data.

Images