A spatial correlation guided prototype distillation few-shot classification method

Abstract

The application discloses a kind of space correlation guided prototype distillation small sample classification methods, including using unlabelled optical remote sensing image to CNN-ViT double-branch hybrid heterogeneous network model is pre-trained to obtain the CNN-ViT double-branch hybrid heterogeneous network model after self-supervised pre-training;The mode feature, context relationship feature and spatial feature of the labeled optical remote sensing image are used to meta-training to the above network model, to obtain the CNN-ViT double-branch hybrid heterogeneous network model for small sample classification;The optical remote sensing image to be classified is input into the CNN-ViT double-branch hybrid heterogeneous network model for small sample classification, to obtain the classification result of the optical remote sensing data to be classified.The application can effectively capture potential spatial discriminative semantic information, improve the optical remote sensing image classification performance under small sample condition, solve the problem of poor robust feature representation ability and easy overfitting under limited sample condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning classification, and particularly relates to a space correlation guided prototype distillation small sample classification method. BACKGROUND

[0002] In recent years, with the development of remote sensing earth observation technology, remote sensing data presents the characteristics of multi-platform, multi-load and multi-spatial resolution, and the overall observation data volume increases exponentially. Therefore, for the PB-level remote sensing data accumulated at the present stage, it is difficult to meet the interpretation application requirements of remote sensing data by manual interpretation. At present, the automatic interpretation technology of remote sensing data based on machine learning is more efficient and accurate than manual interpretation, but the existing convolutional neural network relies heavily on large amounts of labeled data for model training. For massive remote sensing data, high-quality labeling not only requires domain expertise, but also is difficult for humans to implement long-time uninterrupted high-quality data labeling on the growing massive remote sensing observation data. In addition, using small-scale high-quality data labeling for model training is prone to overfitting, resulting in poor model generalization ability and inability to apply to the intelligent automatic interpretation field of space-based remote sensing data. In view of the above problems, high-precision classification technology based on small sample learning emerges as the times require, which can further improve the intelligent processing and interpretation level of remote sensing data.

[0003] The basic idea of small sample learning is to transfer knowledge or experience from seen tasks (tasks sampled from base class data) to unseen tasks (tasks sampled from new class data). Following this basic theory, the earliest transfer learning method is used to solve the small sample classification problem, that is, an initial model is obtained by training an image classification task model through cross-entropy loss in all samples of the base class, and then the model is fine-tuned with a small number of samples of the new class. However, since the samples of the new class are only a few, the model often has difficulty in learning discriminative feature representations on the new class, resulting in poor classification performance. In order to overcome the above shortcomings, a meta-learning small sample classification framework is proposed, that is, when training on the base class, the same task is completed by setting up a small sample classification task through data sample sampling during testing, so that knowledge is more easily transferred from the base class to the new class. Although various meta-learning based methods have designed complex algorithms and network architectures, these methods still have difficulty in learning robust features from a small number of samples, resulting in low performance of these models in practical applications and failing to meet user needs. Therefore, it is urgent to explore the learning mechanism of remote sensing scene classification algorithm under the condition of limited sample labeling, and to construct a robust feature representation method, so as to further improve the data classification performance under the condition of small sample. SUMMARY

[0004] To solve the above problems, the application provides a space correlation guided prototype distillation small sample classification method, which comprises the following steps: performing self-supervised pre-training on a CNN-ViT double-branch hybrid heterogeneous network model by using unlabeled optical remote sensing images to obtain a CNN-ViT double-branch hybrid heterogeneous network model of the CNN-ViT double-branch hybrid heterogeneous network model after self-supervised pre-training; wherein the CNN-ViT double-branch hybrid heterogeneous network model comprises a CNN branch and a ViT branch; performing meta-training on the CNN-ViT double-branch hybrid heterogeneous network model after self-supervised pre-training by using mode features, context relationship features and spatial features of labeled optical remote sensing images to obtain a CNN-ViT double-branch hybrid heterogeneous network model of the CNN-ViT double-branch hybrid heterogeneous network model for small sample classification; wherein the label indicates the scene category of the optical remote sensing image; and inputting an optical remote sensing image to be classified into the CNN-ViT double-branch hybrid heterogeneous network model for small sample classification to obtain a classification result of the optical remote sensing data to be classified, wherein the classification result indicates the scene category of the optical remote sensing image to be classified.

[0005] Further, the self-supervised pre-training of the CNN-ViT double-branch hybrid heterogeneous network model by using unlabeled optical remote sensing images comprises: performing self-supervised pre-training on the CNN branch and the ViT branch respectively; the CNN branch is pre-trained in a self-supervised manner by using MOCO contrast learning, and the ViT branch is pre-trained in a self-supervised manner by using MAE mask prediction.

[0006] Further, the meta-training of the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model using the pattern features, contextual features, and spatial features of the labeled optical remote sensing images includes: constructing a first number of tasks on the labeled optical remote sensing images, each task including a support set and a query set; inputting the labeled optical remote sensing images into the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model, and extracting features to obtain pattern features, contextual features, and spatial features; wherein, the pattern features indicate high-dimensional pattern information extracted by the CNN branch, the contextual features indicate contextual information extracted by the ViT branch, and the spatial features indicate spatial distribution information extracted by the CNN branch; The spatial feature correlation between the support set and the query set is calculated using the CNN spatial features; the context relationship features are reweighted using the spatial feature correlation to obtain reweighted context relationship features; the mean is calculated based on the pattern features and the reweighted context relationship features to obtain ViT branch prototypes and CNN branch prototypes; wherein, one category corresponds to one ViT branch prototype and one CNN branch prototype; prototype distillation is performed on the ViT branch prototypes and CNN branch prototypes; and the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model is optimized based on the label information to obtain the CNN-ViT dual-branch hybrid heterogeneous network model for few-shot classification.

[0007] Furthermore, the labeled optical remote sensing image is input into the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model, and feature extraction is performed to obtain pattern features, contextual relationship features, and spatial features. The expressions for the pattern features and contextual relationship features are as follows:

[0008] Z vit =f vit (x i )

[0009] Z cnn =f cnn (x i )

[0010] Among them, Z vit f represents the contextual relationship feature. vit () represents the feature extraction function of the ViT branch, x i Z represents the input labeled optical remote sensing image data, where i represents the index of the input image (a non-zero integer), and Z... cnn Representative pattern features, f cnn () represents the feature extraction function of a CNN branch;

[0011] The spatial feature expression is as follows:

[0012]

[0013] wherein Z S-loc represents the spatial feature of the support set of the CNN branch, Z Q-loc represents the spatial feature of the query set of the CNN branch, f block3 () represents a partial feature extraction function of the CNN branch, x s represents the support set image data input, x q represents the query set image data input, R represents a real number field, c' represents the number of channels of the spatial feature, and h' and w' represent the length and width of the spatial feature, respectively.

[0014] Further, the calculation of the spatial feature correlation between the support set and the query set using the CNN spatial feature comprises: obtaining the spatial feature correlation M according to the CNN spatial feature, and the expression is as follows:

[0015] M = similarity<Z Q-loc , Z S-loc > ∈ R h'w'×h'w'

[0016] wherein Similarity< > represents a similarity measurement function, Z S-loc represents the spatial feature of the support set of the CNN branch, Z Q-loc represents the spatial feature of the query set of the CNN branch, h' represents the length of the spatial feature, and w' represents the width of the spatial feature.

[0017] Further, the re-weighting processing of the context relationship feature using the spatial feature correlation comprises: re-weighting calculation of the context relationship feature to obtain the re-weighted context relationship feature Z re , and the calculation formula is as follows:

[0018]

[0019] wherein Z vit represents the context relationship feature, and M represents the spatial feature correlation; the re-weighted context relationship feature Z re indicates the part with high correlation in the context relationship feature.

[0020] Further, the obtaining the CNN branch prototype and the ViT branch prototype based on the pattern feature and the reweighted context relationship feature respectively comprises: obtaining the CNN branch prototype based on the pattern feature and the reweighted context relationship feature comprises: performing pooling on the pattern feature to obtain a pooled pattern feature Z' cnn :

[0021] Z' cnn = pool(Z cnn ) ∈ R B×C

[0022] wherein, pool() represents a pooling function, Z cnn represents the pattern feature, R represents a real number field, B represents a number of samples per training, and C represents a channel number of Z' cnn ; and

[0023] calculating the CNN branch prototype corresponding to each class based on the pooled pattern feature, wherein the CNN branch prototype C cnn,k of the class k is expressed as follows:

[0024]

[0025] wherein, S k represents a number of support set images of the class k, and Z' cnn,k represents the pooled pattern feature of the class k;

[0026] obtaining the ViT branch prototype based on the pattern feature and the reweighted context relationship feature comprises:

[0027] performing pooling on the context relationship feature to obtain a pooled context relationship feature Z' vit :

[0028] Z' vit = avgpool(Z re ) ∈ R B×d

[0029] wherein, avgpool() represents an average pooling function, Z re represents the reweighted context relationship feature, B represents a number of samples per training, and d represents a feature dimension of Z' vit ;

[0030] calculating the ViT branch prototype corresponding to each class k based on the pooled context relationship feature, wherein the ViT branch prototype C vit,k of the class k is calculated according to the following formula:

[0031]

[0032] Z' vit,k represents the pooled ViT branch feature of the category k; wherein the CNN branch prototype is used to assist the ViT branch to obtain an accurate ViT branch prototype, and the ViT branch prototype is used to indicate the category information and calculate the probability distribution of the output.

[0033] Further, the prototype distillation of the ViT branch prototype and the CNN branch prototype comprises: projecting the pooled mode feature Z' cnn into the same feature space as the pooled context relationship feature, to obtain the projected mode feature Z project : Z project = project(Z' cnn ) ∈ R B×d , wherein project() represents a projection function, R represents a real number field, B represents the number of samples for each training, and d represents the feature dimension of the pooled context relationship feature Z' vit ; and

[0034] calculating the Euclidean distance between the projected mode feature and the context relationship feature, performing prototype distillation at the feature level, and the prototype distillation is used to distill the feature information captured by the CNN branch into the features of the ViT branch, and the expression is as follows:

[0035] argmin(||Z project -Z' vit ||2)

[0036] Z' vit represents the pooled ViT branch feature, and || ||2 represents the Euclidean distance norm, and argmin represents the minimum value of the expression.

[0037] Further, the optimization of the CNN-ViT dual-branch hybrid heterogeneous network model based on the self-supervised pre-training comprises: calculating the distance between the CNN branch and the CNN branch prototype of each category to obtain the CNN branch probability distribution

[0038]

[0039] calculating the distance between the ViT branch and the ViT branch prototype of each category to obtain the ViT branch probability distribution

[0040]

[0041] wherein, represents the predicted category, and x qrepresenting a query sample, d() representing a distance metric function, Z q-cnn and Z q-vit representing a query sample x q representing features encoded by the CNN branch and the ViT branch, C vit,k representing a ViT branch prototype of class k, C cnn,k representing a CNN branch prototype of class k; the CNN branch probability distribution indicates the likelihood of the prediction result output by the CNN branch, and the ViT branch probability distribution indicates the likelihood of the prediction result output by the ViT branch;

[0042] A joint loss function L is constructed by using the ViT branch probability distribution, the CNN branch probability distribution, the projected pattern feature, and the pooled context relationship feature, wherein the joint loss function L includes a CNN branch classification loss L cnn , a ViT branch classification loss L vit , and a distillation loss L KD :

[0043]

[0044]

[0045] wherein λ represents the weight of the CNN branch in the joint loss function, N represents the number of classes of each task, M represents the number of query set samples in each task, i and n are natural numbers not equal to 0, I() represents a conditional function, representing a predicted class, y representing a true class, x q representing query set image data, p cnn representing a CNN branch probability distribution, p vit representing a ViT branch probability distribution, representing a projected pattern feature, and representing a pooled context relationship feature; and

[0046] The CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification is obtained by optimizing the joint loss function L and supervising the CNN-ViT dual-branch hybrid heterogeneous network model after self-supervised pre-training.

[0047] Further, the inputting the optical remote sensing image to be classified into the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification to obtain a classification result of the optical remote sensing image to be classified comprises: inputting the optical remote sensing image to be classified into the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification to obtain a pooled context relationship feature; obtaining the ViT branch prototype according to the pooled context relationship feature; and obtaining a class probability according to the ViT branch prototype, and the class probability with the maximum value is the classification result, which indicates a scene class of the optical remote sensing image to be classified.

[0048] The application is based on a dual-branch hybrid heterogeneous network model of CNN (convolutional neural network) and ViT (vision transformer), can complete a remote sensing scene classification task under limited labeled sample conditions, can realize efficient and high-precision small sample optical remote sensing image scene classification, and can achieve excellent performance on more challenging small sample tasks, greatly improving the classification precision and effect of existing small sample optical remote sensing image scene classification.

[0049] It should be understood that the content described in the summary part is not intended to limit the key features or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail the following embodiments with reference to the attached drawings. In the drawings, the same or similar reference numerals refer to the same or similar elements, and:

[0051] Figure 1 is a method flowchart of the application;

[0052] Figure 2 is a method flowchart of the application for obtaining a CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification;

[0053] Figure 3 is a whole schematic diagram of the application;

[0054] Figure 4 is a classification effect schematic diagram of the application. DETAILED DESCRIPTION

[0055] Embodiments of the present application will be described below in greater detail with reference to the accompanying drawings. While several embodiments of the application are described in the drawings, it is understood that the application can be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0056] In the description of embodiments of the application, the term "includes" and its variants are to be read as open-ended terms that mean "comprises but is not limited to." The term "based on" is to be read as "based, at least in part, on." The term "one embodiment" or "an embodiment" are to be read as "at least one embodiment." The term "some embodiments" is to be read as "at least some embodiments." Other explicit or implicit definitions can also be included below.

[0057] In the following, only some exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the application. Therefore, the drawings and the description are considered to be exemplary in nature and not limiting. There is no order requirement for the description of the method steps, as long as they can be implemented within the scope of the application.

[0058] Small sample learning is given a set of data D, which contains a test set D novel with a small number of labels, and a training set D base with all labels, and satisfies D base and D novel The label space categories are disjoint, that is where c(·) is the set of data set categories. The goal of small sample learning is to construct a mapping function f that uses the prior knowledge learned from D base to map the samples in D novel to the correct category under the condition of only a small number of labeled data, so as to realize the classification of test set samples.

[0059] In the meta-learning paradigm, a task T is constructed on data D, which includes at least two subsets: a support set S and a query set Q, both of which are sampled from the data D and have the same label space, wherein the support set S is a sample with labeled information, and the query set Q is a sample whose label information needs to be predicted. As there are N categories in the support set S, each category includes K different labeled support samples, K is small, usually less than 10, such as 1 or 5, etc., which are only exemplary and should not be construed as limiting the present application. The query set includes the same N categories as the support set, and each category includes M different query samples, referred to as an N-Way K-Shot M query task.

[0060] The application adopts a CNN-ViT dual-branch hybrid heterogeneous network model to complete a small sample classification task following a pre-training-meta-training-meta-testing paradigm. First, the CNN-ViT dual-branch hybrid heterogeneous network model is self-supervised pre-trained to obtain a self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model of the CNN-ViT dual-branch hybrid heterogeneous network model, which has strong task-agnostic transfer feature extraction capability. Then, meta-training is performed on the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model, the context relationship features are re-weighted through the spatial feature correlation of the support set and the query set, and the ViT branch is guided to learn the region with strong transfer ability. Then, a prototype distillation is introduced in the re-weighted context relationship features to introduce an inductive bias similar to CNN, enhance the robustness of the ViT branch prototype representation, improve the small sample classification ability, and finally obtain a classification result by simple metric classification of the output of the ViT branch.

[0061] The application will be described in detail below with reference to the accompanying drawings Figures 1-4 The application will be described in detail below with reference to the accompanying drawings

[0062] A space correlation guided prototype distillation small sample classification method, specifically comprising:

[0063] S101: A CNN-ViT dual-branch hybrid heterogeneous network model is self-supervised pre-trained using unlabeled optical remote sensing images to obtain a self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model of the CNN-ViT dual-branch hybrid heterogeneous network model, wherein the CNN-ViT dual-branch hybrid heterogeneous network model includes a CNN branch and a ViT branch.

[0064] In some embodiments, both the CNN branch and the ViT branch can adopt methods in the prior art such as Resnet, ResNeXt, DenseNet, ViT-Base, SWIN, etc. as the backbone network, as long as the feature spaces are the same in the subsequent steps, which are not limited here.

[0065] The CNN branch and the ViT branch are respectively pre-trained in a self-supervised manner by inputting a large number of unlabeled optical remote sensing images, thereby obtaining a CNN-ViT dual-branch hybrid heterogeneous network model after self-supervised pre-training. Compared with conventional supervised training, self-supervised pre-training can extract features with stronger transfer performance and better generalization ability. Different self-supervised pre-training strategies for different branches can better improve the classification effect. The following will introduce the self-supervised pre-training method adopted by each branch in detail. This is only exemplary and does not limit the present application:

[0066] For the ViT branch, in some embodiments, the MAE self-supervised pre-training method can be used to transfer the knowledge in natural scenes to the remote sensing scene by the idea of not stopping pre-training. The MAE self-supervised pre-training method includes an encoding part and a decoding part. The encoding part is used for feature extraction of the unlabeled optical remote sensing image, mainly realized by an encoder. The decoding part is used for restoring the extracted features to the original unlabeled optical remote sensing image, mainly realized by a decoder. The specific steps are as follows:

[0067] Randomly mask part of the representation of the input unlabeled optical remote sensing image X to obtain the unlabeled optical remote sensing image after partial masking

[0068] Reconstruct the unlabeled optical remote sensing image X to optimize the loss function Loss, thereby obtaining a ViT branch model with strong feature representation ability; and

[0069] Select the encoder part θ of the ViT branch model with strong feature representation ability encoder The expression of the CNN-ViT dual-branch hybrid heterogeneous network model after self-supervised pre-training of the ViT branch is as follows:

[0070]

[0071]

[0072] Y = decoder(H)

[0073]

[0074] Wherein, encoder() and decoder() represent the encoder and decoder respectively, H represents the image features extracted by the encoder, Y represents the image reconstructed by the decoder, || ||1 represents the L1 norm, represents the number of masked pixel points.

[0075] For the CNN branch, in some embodiments, the training method of MOCO can be used for self-supervised pre-training, and the auxiliary task can be constructed by using the idea of contrastive learning to realize the self-supervised pre-training of the CNN branch. The specific steps are as follows:

[0076] Each input unlabeled optical remote sensing image X is processed by two different data augmentation methods, i.e. type 1 augmentation DataAug1() and type 2 augmentation DataAug2(), to obtain data augmentation result 1 X i and data augmentation result 2 X j .

[0077] The different data augmentation results of the same unlabeled remote sensing image are taken as positive pairs, and the different data augmentation results of different unlabeled remote sensing images are taken as negative pairs; and

[0078] By optimizing the loss function L q , the CNN branch distinguishes between positive and negative pairs to obtain strong feature representation capability, wherein τ represents the temperature hyperparameter, K represents the queue length, the self-supervised pre-trained model is taken as the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model of the CNN branch, and the expression is as follows:

[0079] X i = DataAug1(X)

[0080] X j = DataAug2(X)

[0081]

[0082] S102: The meta-training is performed on the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model by using the mode feature, context relationship feature and spatial feature of the labeled optical remote sensing image, to obtain a CNN-ViT dual-branch hybrid heterogeneous network model of the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification; wherein the label indicates the scene category of the optical remote sensing image.

[0083] In order to make the CNN-ViT dual-branch hybrid heterogeneous network model more suitable for small sample task setting, the meta-learning paradigm is used to train the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model, which can also be called meta-training.

[0084] The following will be described in detail with reference to the accompanying drawings Figure 2 Detailed description of meta-training. Meta-training can be implemented in the following ways, including:

[0085] S1021: A first number of tasks are constructed on the labeled optical remote sensing image, each of the tasks including a support set and a query set; wherein the first number is an integer not less than 0, and the specific value can be flexibly selected according to actual conditions.

[0086] S1022: The labeled optical remote sensing image is input into the CNN-ViT dual-branch hybrid heterogeneous network model pre-trained in a self-supervised manner, and feature extraction is performed to obtain pattern features, context relationship features and spatial features; wherein the pattern features indicate high-dimensional pattern information extracted by the CNN branch, the context relationship features indicate context relationship information extracted by the ViT branch, and the spatial features indicate spatial distribution information extracted in the CNN branch

[0087] S1023: The spatial feature correlation of the support set and the query set is calculated using the CNN spatial features;

[0088] S1024: The context relationship features are re-weighted using the spatial feature correlation to obtain re-weighted context relationship features;

[0089] S1025: The pattern features and the re-weighted context relationship features are respectively averaged to obtain ViT branch prototypes and CNN branch prototypes; wherein one class corresponds to one ViT branch prototype and one CNN branch prototype;

[0090] S1026: Prototype distillation is performed on the ViT branch prototypes and the CNN branch prototypes;

[0091] and

[0092] S1027: The CNN-ViT dual-branch hybrid heterogeneous network model pre-trained in a self-supervised manner is optimized based on the label information to obtain the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification.

[0093] The above steps will be described in detail below.

[0094] S1022: The labeled optical remote sensing image (hereinafter referred to as “sample”) is input into the CNN-ViT dual-branch hybrid heterogeneous network model pre-trained in a self-supervised manner, and feature extraction is performed to obtain pattern features, context relationship features and spatial features, including:

[0095] The input sample x iAfter the feature extraction processing of the ViT branch and the feature extraction processing of the CNN branch respectively, the encoded context relationship feature Z vit and the encoded mode feature Z cnn , the expression is as follows:

[0096] Z vit =f vit (x i )

[0097] Z cnn =f cnn (x i )

[0098] Wherein, x i represents the input labeled optical remote sensing image, f vit () represents the feature extraction function of the ViT branch, and f cnn () represents the feature extraction function of the CNN branch.

[0099] The mode feature Z cnn ∈R B×C×H×W , that is, the extracted input sample has high-dimensional modal information, B represents the number of samples batchsize for each training, C represents the number of channels, and H and W represent the size of the feature map. The context relationship feature Z vit ∈R B ×p×d , that is, the extracted input sample indicates the context relationship, d represents the feature dimension, and p represents the number of local blocks tokens.

[0100] Considering that it is difficult to establish an effective context relationship feature description under the condition of insufficient samples, therefore, under the condition of limited samples, the present application considers fully mining spatial features to establish a prototype representation with stronger migration representation ability. Compared with the context relationship feature, the spatial feature contains more local information and detailed information.

[0101] In the present application, the CNN branch preferably adopts the Resnet18 network architecture, and the ViT branch preferably adopts ViT-Base as the backbone network. They have a wide range of applications and high universality, and can prove the effectiveness of the present application, which is only exemplary and does not limit the present application. The CNN branch includes four residual blocks, which are the first residual block, the second residual block, the third residual block and the fourth residual block. The first two residual blocks only extract low-level features such as texture and contour that are not discriminative, while with the increase of the number of residual blocks, the spatial information contained in the extracted features becomes less and less, and the output of the last residual block is considered as a mode feature without spatial information. Therefore, the output feature of the third residual block is used as the spatial feature, and here the number of support set image data and query set image data is taken as an example, and the spatial feature is as follows:

[0102]

[0103] Z S-loc spatial feature of the support set represented by the CNN branch, Z Q-loc spatial feature of the query set represented by the CNN branch, f block3 () represents a partial feature extraction function in the CNN branch, preferably a feature extraction function of the third residual block, x s represents the support set image data input, x q represents the query set image data input, R represents the real number field, c' represents the number of channels of the spatial feature, h' represents the length of the spatial feature, and w' represents the width of the spatial feature.

[0104] S1023: calculating the spatial feature correlation of the support set and the query set using the CNN spatial feature, specifically comprising:

[0105] According to the spatial feature, the spatial feature correlation M is obtained, and the expression is as follows:

[0106] M = similarity<Z Q-loc , Z S-loc > ∈ R h'w'×h'w' ,

[0107] wherein Similarity< > represents a similarity measurement function, Z S-loc spatial feature of the support set represented by the CNN branch, Z Q-loc spatial feature of the query set represented by the CNN branch, h' represents the length of the spatial feature, and w' represents the width of the spatial feature.

[0108] In some embodiments, a cosine similarity function can be preferred as the similarity measurement function.

[0109] Each value in the spatial feature correlation M represents the correlation of the spatial feature of the support set and the spatial feature of the query set at each position. The higher the value, the more similar the spatial features, and the lower the value, the more different the spatial features. In a remote sensing scene image, the part with high correlation between the spatial feature of the support set and the spatial feature of the query set is usually the key invariant class determination information, which has strong transferability.

[0110] S1024: re-weighting the context relationship feature using the spatial feature correlation to obtain a re-weighted context relationship feature, specifically comprising:

[0111] correlation calculation is performed on the context relationship feature to obtain a re-weighted context relationship feature Z re, the calculation formula is as follows:

[0112]

[0113] Wherein, Z vit represents the context relationship feature, the reweighted context relationship feature Z re indicates the context relationship feature of the region with high spatial feature correlation.

[0114] This step utilizes the spatial feature correlation between the support set and the query set in the CNN branch to process the context relationship feature. The redundant information in the reweighted context relationship feature is suppressed, and the key information is highlighted, which is beneficial to the class determination of the query sample.

[0115] S1025: Based on the mode feature and the reweighted context relationship feature, respectively, the mean value processing is performed to obtain the CNN branch prototype and the ViT branch prototype, specifically including:

[0116] The method for obtaining the CNN branch prototype includes:

[0117] The mode feature is pooled, and the expression is as follows:

[0118] Z' cnn =pool(Z cnn )∈R B×C ,

[0119] Wherein, Z' cnn represents the pooled mode feature, pool() represents the pooling function, Z cnn represents the mode feature, R represents the real number field, B represents the number of samples for each training, and C represents the channel number of Z' cnn ; and

[0120] Based on the pooled mode feature, the CNN branch prototype corresponding to each class is calculated, wherein the CNN branch prototype C cnn,k of class k is expressed as follows:

[0121]

[0122] Wherein, S k represents the number of support set images of class k, and Z' cnn,k represents the pooled mode feature of class k.

[0123] The method for obtaining the ViT branch prototype includes:

[0124] The context relationship feature is pooled, and the expression is as follows:

[0125] Z' vit= avgpool(Z re )∈R B×d ,

[0126] wherein Z' vit represents the pooled context relationship feature, avgpool() represents an average pooling function, Z re represents the re-weighted context relationship feature, B represents the number of samples per training, and d represents the feature dimension of Z' vit ; and

[0127] The ViT branch prototype corresponding to each class is calculated based on the pooled context relationship feature, wherein the ViT branch prototype C vit,k of the class k is calculated according to the following formula:

[0128]

[0129] wherein Z' vit,k represents the pooled ViT branch feature of the class k.

[0130] The CNN branch prototype is used to assist the ViT branch to obtain an accurate ViT branch prototype, and the ViT branch prototype is used to indicate the class information and calculate the class prediction result.

[0131] S1026: Prototype distillation is performed on the ViT branch prototype and the CNN branch prototype, specifically including:

[0132] The pooled mode feature Z' cnn is projected into the same feature space as the pooled context relationship feature Z' vit , to obtain the projected mode feature Z project .

[0133] Z project = project(Z' cnn )∈R B×d ,

[0134] wherein project() represents a projection function, R represents a real number field, B represents the number of samples per training, and d represents the feature dimension of the pooled context relationship feature Z' vit ;

[0135] The Euclidean distance between the projected mode feature and the context relationship feature is calculated, and the prototype distillation is performed at the feature level, which is used to distill the feature information captured by the CNN branch into the feature of the ViT branch, and the prototype distillation expression is as follows:

[0136] argmin(||Z project -Z' vit||2)

[0137] Z' vit represents the pooled ViT branch feature, represents the Euclidean distance norm, and argmin represents the minimum value of the expression.

[0138] The feature information captured by the CNN branch and the ViT branch is different, and the information contained therein is also different. The purpose of the prototype distillation is to distill the feature information captured by the CNN branch into the feature information of the ViT branch, so that the feature information of the distilled ViT branch contains both the feature information captured by the ViT branch and the feature information captured by the CNN branch, thereby enhancing the discriminability of the obtained ViT branch feature.

[0139] The prototype distillation is performed on the context relationship feature and the pattern feature after the reweighting, which can introduce an inductive bias similar to the CNN into the ViT branch. The introduction of the inductive bias similar to the CNN branch into the ViT branch model can help the ViT branch to capture features with strong separability. The prototype knowledge distillation method proposed in the present application injects the prototype representation inductive bias of the CNN branch into the prototype representation of the ViT branch, thereby enhancing the robustness of the prototype representation of the ViT branch.

[0140] With the increase of the amount of small sample classification tasks in the N way K shot meta-training, both the CNN branch and the ViT branch become more and more suitable for the remote sensing small sample classification task. Initially, the prototype representation of the CNN branch is not sufficient to help the ViT branch to have better performance, but with the cooperative training of the CNN branch and the ViT branch, the CNN branch captures more accurate pattern features, and through the soft fusion method of prototype distillation, the inductive bias of the CNN branch is gradually injected into the feature information of the ViT branch, as shown in Figure 4 , which expands the activation range of the feature information of the ViT branch, and alleviates the problem of insufficient robustness of the ViT branch feature information activation area.

[0141] S1027: optimizing the CNN-ViT dual-branch hybrid heterogeneous network model after self-supervised pre-training based on the label information to obtain the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification, specifically comprising:

[0142] calculating the distance between the CNN branch and each class CNN branch prototype to obtain a CNN branch probability distribution

[0143]

[0144] calculating the distance between the ViT branch and each class ViT branch prototype to obtain a ViT branch probability distribution

[0145]

[0146] wherein, represents the predicted class, x q represents the query sample, p cnn represents the CNN branch prediction probability, p vit represents the ViT branch probability distribution, d() represents a distance metric function, Z q-cnn and Z q-vit respectively represent the query sample x q the features encoded by the CNN branch and the ViT branch, C vit,k represents the ViT branch prototype of the class k, C cnn,k represents the CNN branch prototype of the class k; the CNN branch probability distribution indicates the possibility of the prediction result output by the CNN branch, and the ViT branch probability distribution indicates the possibility of the prediction result output by the ViT branch;

[0147] The probability distribution can be understood as the possibility of the prediction result of each class: for example, the probability of a river is 0.8, the probability of a street is 0.05, the probability of a palace is 0.1, and the probability of a farmland is 0.05. The possibility of each class of prediction is collectively referred to as a probability distribution.

[0148] The joint loss function L is constructed by using the ViT branch probability distribution and the CNN branch probability distribution, the projected mode feature and the pooled context relationship feature, and the joint loss function L includes the CNN branch classification loss L cnn , the ViT branch classification loss L vit and the distillation loss L KD :

[0149]

[0150]

[0151] wherein λ represents the weight of the CNN branch in the joint loss function, N represents the number of classification categories of each task, M represents the number of query set samples in each task, i and n are natural numbers not equal to 0, I() represents a conditional function, represents the predicted class, y represents the true class, x q represents the query set sample, p cnn represents the CNN branch probability distribution, p vit represents the ViT branch probability distribution, Z project represents the projected mode feature, Z' vit represents the pooled context relationship feature.

[0152] and the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification is obtained by optimizing the joint loss function L.

[0153] S103: inputting the optical remote sensing image to be classified into the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification to obtain a classification result of the optical remote sensing data to be classified, the classification result indicating a scene category of the optical remote sensing image to be classified.

[0154] In some embodiments, the step specifically comprises:

[0155] S1031: inputting the optical remote sensing image to be classified into the CNN-ViT dual-branch hybrid heterogeneous network model for small sample classification to obtain the pooled context relationship feature;

[0156] S1032: obtaining the ViT branch prototype according to the pooled context relationship feature; and

[0157] S1033: obtaining a category probability according to the ViT branch prototype, the category probability with the maximum value being the classification result, the classification result indicating a collection scene of the optical remote sensing image to be classified.

[0158] Experimental verification:

[0159]

[0160]

[0161] Note: the NWPU-RESISC45 dataset is an optical remote sensing image scene classification dataset created by the Northwestern Polytechnical University. The dataset contains 31,500 images and covers 45 scene categories, with 700 images for each category.

[0162] According to the experimental results, it can be concluded that the method proposed in the application achieves the best effect on the small sample remote sensing scene classification task, and compared with the previous method, the method improves by 30.92% on the 5way 1shot task and by 20.05% on the 5way 5shot task, proving the effectiveness of the method proposed in the application.

[0163] The beneficial effects of the application are as follows:

[0164] 1. The application proposes a CNN and ViT dual-branch hybrid model to complete the small sample optical remote sensing image scene classification task, the ViT model has strong spatial context global feature extraction capability, can effectively capture potential spatial discriminative semantic information, and promote the classification ability of the model.

[0165] 2. The application uses CNN to establish the spatial feature correlation between the support set and the query set, and uses the correlation weight to guide the ViT model to mine effective spatial features with migration.

[0166] 3. Under the guidance of the CNN establishing the spatial feature correlation between the support set and the query set, the prototype description bias of the CNN is introduced into the prototype representation of the ViT through prototype distillation, which improves the small sample learning representation ability and robustness of the prototype of the ViT model, and can assist the rapid convergence of the ViT model.

[0167] 4. The application can significantly improve the small sample optical remote sensing image scene classification ability, and even maintain excellent classification performance under more challenging tasks, and has good practical application value.

[0168] Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the application, which are used to illustrate the technical solutions of the application, but not to limit it, the protection scope of the application is not limited to this, although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: any person skilled in the art in the technical range disclosed by the application can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features, and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and all should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A spatial correlation-guided prototype distillation small-sample classification method, characterized in that, A self-supervised pre-training of a CNN-ViT dual-branch hybrid heterogeneous network model is performed using unlabeled optical remote sensing images to obtain the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model; wherein, the CNN-ViT dual-branch hybrid heterogeneous network model includes a CNN branch and a ViT branch; The self-supervised pre-training of the CNN-ViT dual-branch hybrid heterogeneous network model using unlabeled optical remote sensing images includes: Self-supervised pre-training was performed on the CNN branch and the ViT branch respectively; The CNN branch uses MOCO contrastive learning for self-supervised pre-training, and the ViT branch uses MAE mask prediction for self-supervised pre-training. Meta-training is performed on the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model using pattern features, contextual features, and spatial features of labeled optical remote sensing images to obtain the CNN-ViT dual-branch hybrid heterogeneous network model for few-shot classification; wherein, the label indicates the scene category of the optical remote sensing image; The meta-training of the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model using pattern features, contextual features, and spatial features of labeled optical remote sensing images includes: A first number of tasks are constructed on the labeled optical remote sensing image, each task including a support set and a query set; The labeled optical remote sensing image is input into the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model. After feature extraction, pattern features, contextual relationship features, and spatial features are obtained. The pattern features indicate the high-dimensional pattern information extracted by the CNN branch, the contextual relationship features indicate the contextual relationship information extracted by the ViT branch, and the spatial features indicate the spatial distribution information extracted by the CNN branch. The spatial feature correlation between the support set and the query set is calculated using the spatial features. The spatial feature correlation indicates the degree of correlation between the spatial features, and the positions with a high degree of correlation are the parts with strong discriminative power. The spatial feature correlation is used to reweight the context relationship features to obtain reweighted context relationship features. The mean is calculated based on the pattern features and the reweighted contextual relationships to obtain the ViT branch prototype and the CNN branch prototype; wherein, one category corresponds to one ViT branch prototype and one CNN branch prototype. Prototype distillation is performed on the ViT branch prototype and the CNN branch prototype; And optimize the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model based on the label information to obtain the CNN-ViT dual-branch hybrid heterogeneous network model for few-shot classification; The optical remote sensing image to be classified is input into the CNN-ViT dual-branch hybrid heterogeneous network model for few-sample classification to obtain the classification result of the optical remote sensing data to be classified, and the classification result indicates the scene category of the optical remote sensing image to be classified.

2. The method as described in claim 1, characterized in that, The labeled optical remote sensing images are input into the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model. After feature extraction, pattern features, contextual relationship features, and spatial features are obtained. Feature extraction is performed on the labeled optical remote sensing images to obtain pattern features, spatial features, and contextual relationship features. The expressions for the pattern features and contextual relationship features are as follows: Z vit =f vit (x i ) Z cnn =f cnn (x i ) Among them, Z vit f represents the contextual relationship feature. vit () represents the feature extraction function of the ViT branch, x i Z represents the input labeled optical remote sensing image data, where i represents the index of the input image (a non-zero integer), and Z... cnn Representative pattern features, f cnn () represents the feature extraction function of a CNN branch; The spatial feature expression is as follows: Among them, Z S-loc Z represents the spatial features of the support set of the CNN branch. Q-loc The spatial features of the query set representing the CNN branch, f block3 () represents a partial feature extraction function of a CNN branch, x s The image data representing the support set input, x q The input query set image data represents R, which represents the real number field, c' represents the number of channels of the spatial feature, h' represents the length of the spatial feature, and w' represents the width of the spatial feature.

3. The method as described in claim 1, characterized in that, The step of calculating the spatial feature correlation between the support set and the query set using the CNN spatial features includes: The spatial feature correlation M is obtained based on the CNN spatial features, and the expression is as follows: M=similarity<Z Q-loc ,Z S-loc >∈R h'w'×h'w' Where Similarity<> represents the similarity measurement function, Z S-loc Z represents the spatial features of the support set of the CNN branch. Q-loc The spatial features of the query set representing the CNN branch are defined as follows: h' represents the length of the spatial feature, and w' represents the width of the spatial feature.

4. The method as described in claim 1, characterized in that: The step of reweighting the contextual relationship features using the spatial feature correlation includes: The context relationship features are reweighted to obtain the reweighted context relationship features Z. re The calculation formula is as follows: Among them, Z vit The context relationship feature Z represents the spatial feature correlation; M represents the reweighted context relationship feature Z. re Indicates the highly relevant parts of the contextual relationship features.

5. The method as described in claim 1, characterized in that: The step of averaging the pattern features and the reweighted contextual relationship features to obtain the ViT branch prototype and the CNN branch prototype includes: The CNN branch prototype obtained based on the pattern features and the reweighted contextual relationship features includes: The pattern features are pooled to obtain the pooled pattern features Z'. cnn : WITH' cnn =pool(Z cnn )∈R B×C Where pool() represents the pooling function, Z cnn Representing pattern features, R represents the real number field, B represents the number of samples in each training iteration, and C represents Z'. cnn The number of channels; and Based on the pooled pattern features, calculate the CNN branch prototype corresponding to each category. Among them, for the CNN branch prototype C of category k cnn,k The expression is as follows: Among them, S k Z' represents the number of support set images for category k. cnn,k The pooled pattern features representing category k; The ViT branch prototype is obtained based on the pattern features and the reweighted context relationship features, including: The reweighted contextual relationship features are pooled to obtain the pooled contextual relationship features Z'. vit : Z' vit =avgpool(Z re )∈R B×d Where avgpool() represents the average pooling function, Z re Z' represents the reweighted contextual relationship features, B represents the number of samples trained each time, and d represents Z'. vit Feature dimensions; and Based on the pooled context relationship features, calculate the ViT branch prototype corresponding to each category k. Among them, for the ViT branch prototype C of category k vit,k The calculation formula is as follows: Z' vit,k The pooled ViT branch features represent category k; The CNN branch prototype is used to assist the ViT branch in obtaining an accurate ViT branch prototype, which is used to indicate category information and calculate the probability distribution of the output.

6. The method according to claim 1, characterized in that, The prototype distillation of the ViT branch prototype and the CNN branch prototype includes: The pooled pattern feature Z' cnn Project the pattern features Z onto the same feature space as the pooled context features to obtain the projected pattern features. project : Z project =project(Z' cnn )∈R B×d Where project() represents the projection function, R represents the real number field, B represents the number of samples trained each time, and d represents the pooled contextual features Z'. vit Feature dimensions; and The Euclidean distance between the projected pattern features and the pooled context features is calculated, and prototype distillation at the feature level is performed. Prototype distillation is used to distill the feature information captured by the CNN branch into the features of the ViT branch. The prototype distillation expression is as follows: argmin(||Z project -WITH' vit ||2) Z' vit The pooled context features are represented by ||2, which represents the Euclidean distance norm, and argmin represents taking the minimum value of the expression.

7. The method according to claim 1, characterized in that, The optimization of the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model based on the label information includes: Calculate the distance between the CNN branch and the CNN branch prototype for each category to obtain the CNN branch probability distribution. : Calculate the distance between the ViT branch and the ViT branch prototype for each category to obtain the ViT branch probability distribution. : in, Represents the prediction category, x q Represents the query sample, d() represents the distance metric function, and Z represents the distance metric function. q-cnn and Z q-vit These represent the image data x in the query set, respectively. q Based on the pattern features and contextual relationship features extracted from the CNN branch and the ViT branch, C vit,k The ViT branch prototype representing category k, C cnn,k The CNN branch prototype represents category k; the CNN branch probability distribution indicates the probability of the prediction result output by the CNN branch, and the ViT branch probability distribution indicates the probability of the prediction result output by the ViT branch. A joint loss function L is constructed using the ViT branch probability distribution, the CNN branch probability distribution, the projected pattern features, and the pooled context relationship features. The joint loss function L includes the CNN branch classification loss L. cnn ViT branch classification loss L vit and distillation loss L KD : Where λ represents the weight of the CNN branch in the joint loss function, N represents the number of classes for each task, M represents the number of query set samples for each task, i and n are both non-zero natural numbers, and I() represents the condition function. y represents the predicted category, and x represents the true category. q p represents the image data in the query set. cnn p represents the probability distribution of CNN branches. vit Z represents the probability distribution of the ViT branch. project Z' represents the projected mode features. vit Represents the contextual relationship features after pooling; and By optimizing the joint loss function L, the self-supervised pre-trained CNN-ViT dual-branch hybrid heterogeneous network model is supervised to obtain the CNN-ViT dual-branch hybrid heterogeneous network model for few-shot classification.

8. The method according to claim 1, characterized in that, The step of inputting the optical remote sensing image to be classified into the CNN-ViT dual-branch hybrid heterogeneous network model for few-sample classification to obtain the classification result of the optical remote sensing image to be classified includes: The optical remote sensing image to be classified is input into the CNN-ViT dual-branch hybrid heterogeneous network model for few-sample classification to obtain the pooled contextual relationship features; The ViT branch prototype is obtained based on the pooled context relationship features; and The category probability is obtained based on the ViT branch prototype, and the category with the highest probability is the classification result, which indicates the scene category of the optical remote sensing image to be classified.

Images