Cross-domain small sample classification method and system for hyperspectral images based on heterogeneous data

Abstract

The embodiment of the specification provides a hyperspectral image cross-domain small sample classification method and system based on heterogeneous data, wherein the method comprises the following steps: first stage training of a spatial feature extraction backbone network based on a metric-based meta-learning method by using labeled natural images in a heterogeneous source domain, extracting spatial features of the natural images to obtain a spatial feature extractor; second stage training of a spatial-spectral feature extraction network composed of the spatial feature extractor and a spectral feature extractor by using labeled hyperspectral images in the heterogeneous source domain and a target domain by adopting a small sample classification method FSL, obtaining spatial-spectral features of the hyperspectral image HSI which is domain adaptive, and obtaining a spatial-spectral feature extractor; and classifying unlabelled hyperspectral images in the target domain by using the spatial-spectral feature extractor.

Description

Technical Field

[0001] This document relates to the field of image classification technology, and in particular to a cross-domain small sample classification method and system for hyperspectral images based on heterogeneous data. Background Art

[0002] Hyperspectral imagery (HSI) captures rich spectral information across hundreds of spectral bands, enabling better discrimination of land cover than traditional multispectral imagery. Classification, as a fundamental task in HSI interpretation, has garnered widespread attention. Recent HSI classification methods primarily employ deep learning techniques and have achieved remarkable performance. Representative approaches include those based on convolutional neural networks (CNNs), graph convolutional neural networks, and, more recently, Transformer-based methods. However, these methods typically require large amounts of labeled data for model training.

[0003] Few-shot learning (FSL) is an effective technique for alleviating the problem of scarce training data. It is inspired by the human cognitive process of learning new categories. FSL typically follows a meta-learning framework, consisting of meta-training and meta-testing phases. The former acquires useful meta-knowledge from different meta-training tasks, which can be generalized to the classification of new, unseen categories in the meta-testing phase.

[0004] Applying few-shot learning (FSL) to hyperspectral image (HSI) classification has achieved excellent performance with limited training data. However, existing HSI few-shot classification methods often suffer from insufficient domain transfer knowledge, which is either derived only from natural images or only from HSI, resulting in insufficient learning of useful meta-knowledge for the target domain. Summary of the Invention

[0005] One or more embodiments of this specification provide a cross-domain small sample classification method for hyperspectral images based on heterogeneous data, including:

[0006] Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to train the spatial feature extraction backbone network in the first stage to extract the spatial features of natural images and obtain a spatial feature extractor.

[0007] Using labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of spatial feature extractor and spectral feature extractor to obtain the spatial-spectral features of domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor.

[0008] Unlabeled hyperspectral images in the target domain are classified by the proposed spatial-spectral feature extractor.

[0009] Furthermore, the metric-based meta-learning method performs the first-stage training of the spatial feature extraction backbone network as follows:

[0010] In each meta-training task, a set of labeled natural images D in the source domain is sn Randomly select N classes from the dataset, and then randomly select K labeled samples from the N classes as the support set Randomly select T labeled data from the remaining labeled samples of the selected category to construct the query set

[0011] Calculate the sample x by formula 1 j Probability of belonging to class k:

[0012]

[0013] Among them, F s a (·) is the backbone network for spatial feature extraction, n k is the support set S s The centroid of the kth class in , d(·) represents the Euclidean distance between two vectors;

[0014] The first stage of training uses a small sample classification loss function based on cross entropy. The cross entropy-based FSL loss function is calculated using Formula 2:

[0015]

[0016] S sn and Q sn Refers to the support set and query set constructed from natural image data labeled in heterogeneous source domains.

[0017] Furthermore, the method further comprises:

[0018] The rotation prediction auxiliary task is introduced to enhance the robustness of the backbone network to different orientations of objects in HSI. Specifically:

[0019] The labeled natural image set D in the source domain sn The image in the image is rotated in four directions R = {0°, 90°, 180°, 270°}, using a prediction head consisting of an activation function and a fully connected layer To predict the score of the rotation category, the resulting cross entropy loss is calculated by formula 3:

[0020]

[0021] in, The expanded data set obtained after rotating the image in four directions; x r (r∈R) represents The rotated image of the original image x in .

[0022] Furthermore, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI. The specific method is as follows:

[0023] Combined with the spatial feature extractor described in the first stage, a 2D convolution layer and a 2D batch normalization layer are added to the front of it to reduce the number of channels of the input data to 3 channels, thus obtaining the spatial feature extractor of the second stage;

[0024] A spectral feature extraction branch is designed using 1D convolution, and combined with the spatial feature extractor in the second stage to obtain a dual-branch spatial-spectral feature extractor;

[0025] Taking a square block centered on each pixel of the HSI as input data, processing the input data, and uniformly mapping the dimensions of the input data to 100 through a mapping layer constructed by a two-dimensional convolutional neural network;

[0026] Based on the processed input data, training the dual-branch spatial-spectral feature extractor using a metric-based meta-learning method;

[0027] The spatial and spectral features of the hyperspectral image are extracted through the trained dual-branch spatial-spectral feature extractor;

[0028] By concatenating the extracted spatial features and spectral features, the embedded spatial-spectral features are obtained.

[0029] Furthermore, the specific method of using the metric-based meta-learning method to train the dual-branch spatial-spectral feature extractor is:

[0030] N-way K-shot is used to construct the corresponding support set and query set from the labeled hyperspectral images in the source domain and the hyperspectral images in the target domain, respectively. Formula 4 is used as the small sample classification loss function for the second stage training:

[0031]

[0032] in and are the FSL losses in SD and TD of HSI, respectively.

[0033] Furthermore, the method further includes adopting a contrastive learning method based on category matching to promote similar representations of similar categories across domains and alleviate domain shift:

[0034] and Denotes the mean of the spatial spectral features of the kth class in the HSI source domain and the target domain, respectively. For each class in the hyperspectral image in the target domain, its best matching class is found in the hyperspectral image in the source domain according to the cosine similarity. The matching class with the highest similarity in the two domains is regarded as a positive sample, and the remaining unmatched classes are regarded as negative samples. The contrast loss based on class matching is calculated using Formula 5:

[0035]

[0036] in, || is the indicator function and τ is the temperature parameter.

[0037] Furthermore, the method further includes using a self-supervised contrastive loss to promote consistent representation of similar samples within a class:

[0038] In the hyperspectral image of the target domain, the labeled data is expanded by data augmentation, and the self-supervised contrast loss is calculated using Formula 6:

[0039]

[0040] Among them, D KL (·||·) represents the Kullback-Leibler divergence, H(·) represents the entropy of a specific probability distribution, B represents the batch size, and represents the original view and enhanced view, P i 1 and P i 2 represents their class distributions, which are obtained by inputting spatial spectral features into the softmax classifier;

[0041] The total training loss for the second stage is calculated using Formula 7:

[0042]

[0043] Among them, λ and η are two parameters used to balance different terms.

[0044] One or more embodiments of this specification provide a cross-domain small sample classification system for hyperspectral images based on heterogeneous data, including:

[0045] The first-stage training module is used to use labeled natural images in heterogeneous source domains and perform the first-stage training of the spatial feature extraction backbone network based on the metric meta-learning method to extract the spatial features of natural images and obtain the spatial feature extractor;

[0046] The second stage training module is used to use the hyperspectral images labeled in the heterogeneous source domain and the target domain, and adopt the small sample classification method FSL to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor, so as to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the hyperspectral image classification model;

[0047] Image classification module: used to classify unlabeled hyperspectral images in the target domain through the hyperspectral image classification model.

[0048] One or more embodiments of this specification provide an electronic device, including:

[0049] processor; and,

[0050] A memory is arranged to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the above-mentioned method for cross-domain small sample classification of hyperspectral images based on heterogeneous data.

[0051] One or more embodiments of this specification provide a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the above-mentioned method for cross-domain small sample classification of hyperspectral images based on heterogeneous data.

[0052] By adopting the embodiment of the present invention, full use is made of different types of labeled images in heterogeneous source domains, and complementary advantages are achieved through two-stage training, thereby improving the comprehensiveness and accuracy of feature extraction. The obtained spatial-spectral feature extractor is used to efficiently and specifically classify unlabeled hyperspectral images in the target domain, which helps to improve classification accuracy under limited labeled data and expand the application scenarios of hyperspectral images.

[0053] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1A flowchart of a cross-domain small sample classification method for hyperspectral images based on heterogeneous data provided in one or more embodiments of this specification;

[0056] Figure 2 A schematic diagram of the overall framework of a cross-domain small sample classification network for hyperspectral images based on heterogeneous data provided in one or more embodiments of this specification;

[0057] Figure 3 The classification results of different classification methods compared in the specific embodiments of this specification on the IP dataset;

[0058] Figure 4 The classification results of different classification methods compared in the specific embodiments of this specification on the HT dataset;

[0059] Figure 5 This is a visualization diagram of the matching degree between the classification diagram and the true value of different methods on IP in the specific embodiments of this specification;

[0060] Figure 6 The effects of different losses on the performance of the method in the specific embodiments of this specification;

[0061] Figure 7 A schematic diagram of the composition of a cross-domain small sample classification system for hyperspectral images based on heterogeneous data provided in one or more embodiments of this specification;

[0062] Figure 8 A schematic diagram of the structure of an electronic device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0064] Method Example

[0065] According to an embodiment of the present invention, a cross-domain small sample classification method for hyperspectral images based on heterogeneous data is provided. Figure 1 A flowchart of a cross-domain small sample classification method for hyperspectral images based on heterogeneous data is provided for one or more embodiments of this specification, such as Figure 1As shown, the cross-domain small sample classification method for hyperspectral images based on heterogeneous data according to an embodiment of the present invention specifically includes:

[0066] S1. Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to perform the first stage training of the spatial feature extraction backbone network to extract the spatial features of natural images and obtain a spatial feature extractor.

[0067] The overall framework of the cross-domain small sample classification network for hyperspectral images based on heterogeneous data is as follows: Figure 2 As shown in the figure, it includes two stages, namely the bone intervention training stage of spatial feature extraction and the cross-domain small sample learning stage. Let D sn and D sh denote the labeled natural images and HSI in the source domain, D t Represents the hyperspectral data in the target domain. The classes in the source domain are taken as base classes, and the classes in the target domain are taken as new classes. The number of base classes is usually greater than the number of new classes. sn and D sh Rich information to support D t The classification of the new category, among which D t Contains only a small amount of labeled data.

[0068] The metric-based meta-learning method performs the first-stage training of the spatial feature extraction backbone network as follows:

[0069] In each meta-training task, a set of labeled natural images D in the source domain is sn Randomly select N classes from the dataset, and then randomly select K labeled samples from the N classes as the support set Randomly select T labeled data from the remaining labeled samples of the selected category to construct the query set

[0070] In the first stage, a metric-based meta-learning method is used to train the spatial extraction backbone network, which makes samples belonging to the same class closer in the feature space and samples belonging to different classes farther away.

[0071] Formula 1 calculates sample x j Probability of belonging to class k:

[0072]

[0073] Among them, F s a (·) is the backbone network for spatial feature extraction, n k is the support set S s is the centroid of the kth class in , and d(·) represents the Euclidean distance between two vectors.

[0074] The first stage of training uses a small sample classification loss function based on cross entropy. The cross entropy-based FSL loss function is calculated using Formula 2:

[0075]

[0076] S sn and Q sn Refers to the support set and query set constructed from natural image data labeled in heterogeneous source domains.

[0077] Objects in remote sensing usually have different orientations, while the orientations of objects in natural images are relatively consistent. In order to enhance the spatial feature extractor F s a (·) To improve the robustness of HSI objects in different orientations, we introduce a rotation prediction auxiliary task. Specifically:

[0078] The labeled natural image set D in the source domain sn The image in the image is rotated in four directions R = {0°, 90°, 180°, 270°}, using a prediction head consisting of an activation function and a fully connected layer To predict the score of the rotation category, the resulting cross entropy loss is calculated by formula 3:

[0079]

[0080] in, The expanded data set obtained after rotating the image in four directions; x r (r∈R) represents The rotated image of the original image x in .

[0081] S2. Using the labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor.

[0082] Combined with the spatial feature extractor described in the first stage, a 2D convolution layer and a 2D batch normalization layer are added at the front to reduce the number of channels of the input data to 3 channels, thereby obtaining the spatial feature extractor of the second stage; a spectral feature extraction branch is designed using 1D convolution, and combined with the spatial feature extractor of the second stage to obtain a dual-branch spatial-spectral feature extractor; a square block centered on each HSI pixel is used as input data, the input data is processed, and the dimension of the input data is uniformly mapped to 100 through a mapping layer constructed by a two-dimensional convolutional neural network; based on the processed input data, the dual-branch spatial-spectral feature extractor is trained using a metric-based meta-learning method.

[0083] The specific method of training the dual-branch spatial-spectral feature extractor based on the metric-based meta-learning method is as follows:

[0084] N-way K-shot is used to construct the corresponding support set and query set from the labeled hyperspectral images in the source domain and the hyperspectral images in the target domain, respectively. Formula 4 is used as the small sample classification loss function for the second stage training:

[0085]

[0086] in and are the FSL losses in SD and TD of HSI, respectively.

[0087] The spatial features and spectral features of the hyperspectral image are extracted by a trained dual-branch spatial-spectral feature extractor; the extracted spatial features and spectral features are connected to obtain embedded spatial-spectral features.

[0088] Due to the influence of different variations in categories, land cover, sensors, noise, and degradation, the HSI of the source and target domains usually show a large inter-domain gap. Therefore, in order to alleviate the domain shift, a contrastive learning method based on category matching is adopted to promote similar representations of similar classes across domains. Specifically:

[0089] A contrastive learning method based on category matching is used to promote similar representations of similar classes across domains and alleviate domain shift:

[0090] and Denotes the mean of the spatial spectral features of the kth class in the HSI source domain and the target domain, respectively. For each class in the hyperspectral image in the target domain, its best matching class is found in the hyperspectral image in the source domain according to the cosine similarity. The matching class with the highest similarity in the two domains is regarded as a positive sample, and the remaining unmatched classes are regarded as negative samples. The contrast loss based on class matching is calculated using Formula 5:

[0091]

[0092] in, ‖ is the indicator function and τ is the temperature parameter.

[0093] In the target HSI domain, hyperspectral data usually exhibit large intra-class variations due to noise and spectral mixing. The limited training data in the target domain makes it difficult for the model to provide a consistent representation for intra-class data, thereby reducing the subsequent classification performance. To alleviate this problem and obtain a smooth classification map, self-supervised learning techniques are used to promote consistent representation of similar samples. In HSI TD, data augmentation is used to expand the labeled data. and represents the original view and enhanced view, P i 1 and P i 2 Represent their class distributions, which are obtained by inputting spatial spectral features into the softmax classifier, and introducing self-supervised contrast loss to promote consistent representation of similar samples within the class:

[0094] In the hyperspectral image of the target domain, the labeled data is extended by data augmentation, and the self-supervised contrast loss is calculated using Formula 6:

[0095]

[0096] Among them, D KL (·‖·) represents the Kullback-Leibler divergence, H(·) represents the entropy of a specific probability distribution, B represents the batch size, and represents the original view and enhanced view, P i 1 and P i 2 represents their class distributions, which are obtained by inputting spatial spectral features into the softmax classifier;

[0097] The total training loss for the second stage is calculated using Formula 7:

[0098]

[0099] Among them, λ and η are two parameters used to balance different terms.

[0100] S3. Classify the unlabeled hyperspectral image in the target domain through the spatial-spectral feature extractor.

[0101] The trained spatial spectral feature extractor is used to extract the spatial spectral embedding features of the samples, and the features and labels of a small number of labeled samples are used to fit a neural network classifier, which is used to classify the unlabeled hyperspectral images.

[0102] The following is a specific example of cross-domain small sample classification of hyperspectral images based on heterogeneous data:

[0103] We use the natural image dataset MiniImageNet and the HSI dataset Chikusei as heterogeneous source domain data. MiniImageNet contains 100 classes, and each image is sized 84×84×3; Chikusei is sized 2517×2335×128 and contains 19 classes. We choose Indian Pines (IP) and Houston13 (HT) as target domains HSI, respectively, which contain new classes to be classified. The IP data size is 145×145×200 and contains 16 classes, and the HT data size is 349×1905×155 and has 15 classes.

[0104] In this embodiment, VGG16 is used as the backbone network for the first stage of spatial feature extraction. The input block size of natural images is 33×33×3. In each meta-training of FSL, N is set to the number of categories of the target HSI, K is set to 1, and the number of samples of each class in the support set and query set is 1 and 19 respectively. t In , 5 labeled samples are selected for each class of FSL.

[0105] From D sh 80 labeled samples are randomly selected for each class of FSL. For the target domain contrast loss, D is obtained by adding random Gaussian noise. t The augmented data is used, the optimizer is Adam, and the learning rate is 0.001; in the second stage, we set the number of iterations to 1000. For IP and HT, λ is set to 1, η is set to 0.2 and 0.5 respectively, and the batch size is 128. The evaluation indicators include overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ). The compared methods include the traditional deep learning-based supervised classification method SSRN and four recent HSI few-shot classification methods DCFSL, Gia-CFSL

[28] , and FSCF-SSL. Figure 3 The average results of ten runs for all methods are shown.

[0106] Depend on Figure 3 As can be seen, this example performs best in terms of OA, AA, and κ. All few-shot learning-based methods achieve better performance than the traditional supervised method SSRN, demonstrating the effectiveness of FSL for HSI classification with limited training data. Compared to DCFSL, which uses only HSI as source data, it achieves a 14.02% improvement in OA; compared to FSCF-SSL, which uses only natural images as source data, it achieves a 1.9% improvement in OA. The performance improvements compared to DCFSL and FSCF-SSL demonstrate the benefits of using both natural images and HSI as source data. This method achieves top-2 class-specific accuracy across 13 categories. Figure 5The visualization results in show that the classification map has the highest match with the true value and is less contaminated by impulse noise than the other classification maps.

[0107] Figure 4 The results of different methods on HT are shown in Figure 2. All FSL-based methods achieve better results than the supervised method SSRN. In addition, the results show that the method of this embodiment outperforms DCFSL and FSCF-SSL, thereby promoting discriminative spatial-spectral feature extraction.

[0108] Figure 3 and Figure 4 The effectiveness of FSL and fusing heterogeneous data in the source domain has been verified. Here, we analyze the impact of different losses on the performance of this method. Figure 6 In the above example, “Baseline” represents a simplified version of the above method, which uses D sn and D sh The model was trained using the FSL loss. "Cross-domain" and "Target-domain" are the corresponding cross-domain and target-domain contrast losses. Results show that, overall, these two losses improve performance on different datasets. Compared to Scheme 2, OA improves by 3.74% on the HT dataset and by 2.32% on the HT dataset. Compared to the baseline, Scheme 4 improves OA by 3.84% on the IP dataset and by 2.47% on the HT dataset.

[0109] The beneficial effects of the present invention are as follows:

[0110] The present invention makes full use of different types of labeled images in heterogeneous source domains, achieves complementary advantages through two-stage training, improves the comprehensiveness and accuracy of feature extraction, and uses the obtained spatial spectral feature extractor to efficiently and specifically classify unlabeled hyperspectral images in the target domain, which helps to improve classification accuracy under limited labeled data and expand the application scenarios of hyperspectral images.

[0111] System Example

[0112] According to an embodiment of the present invention, a cross-domain small sample classification system for hyperspectral images based on heterogeneous data is provided. Figure 3 A schematic diagram of a cross-domain small sample classification system for hyperspectral images based on heterogeneous data is provided for one or more embodiments of this specification, such as Figure 3 As shown, the cross-domain small sample classification system for hyperspectral images based on heterogeneous data according to an embodiment of the present invention specifically includes:

[0113] The first-stage training module is used to use labeled natural images in heterogeneous source domains and perform the first-stage training of the spatial feature extraction backbone network based on the metric meta-learning method to extract the spatial features of natural images and obtain the spatial feature extractor;

[0114] The second stage training module is used to use the hyperspectral images labeled in the heterogeneous source domain and the target domain, and adopt the small sample classification method FSL to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor, so as to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the hyperspectral image classification model;

[0115] Image classification module: used to classify unlabeled hyperspectral images in the target domain through the hyperspectral image classification model.

[0116] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0117] Device Example 1

[0118] An embodiment of the present invention provides an electronic device, such as Figure 5 As shown, it includes: a memory 50, a processor 52, and a computer program stored in the memory 50 and executable on the processor 25. When the computer program is executed by the processor 52, the following method steps are implemented:

[0119] S1. Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to perform the first phase of training on the spatial feature extraction backbone network to extract the spatial features of natural images and obtain a spatial feature extractor.

[0120] S2. Using labeled hyperspectral images in heterogeneous source and target domains, we use the small sample size classification method (FSL) to perform the second-stage training of a spatial-spectral feature extraction network consisting of a spatial feature extractor and a spectral feature extractor. This method obtains the spatial-spectral features of the domain-adaptive hyperspectral image (HSI) and the resulting spatial-spectral feature extractor.

[0121] S3. Classify the unlabeled hyperspectral image in the target domain through the spatial-spectral feature extractor.

[0122] Device Example 2

[0123] An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program for implementing information transmission. When the program is executed by the processor 52, the following method steps are implemented:

[0124] S1. Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to perform the first phase of training on the spatial feature extraction backbone network to extract the spatial features of natural images and obtain a spatial feature extractor.

[0125] S2. Using labeled hyperspectral images in heterogeneous source and target domains, we use the small sample size classification method (FSL) to perform the second-stage training of a spatial-spectral feature extraction network consisting of a spatial feature extractor and a spectral feature extractor. This method obtains the spatial-spectral features of the domain-adaptive hyperspectral image (HSI) and the resulting spatial-spectral feature extractor.

[0126] S3. Classify the unlabeled hyperspectral image in the target domain through the spatial-spectral feature extractor.

[0127] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk, or optical disk.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-domain small sample classification method for hyperspectral images based on heterogeneous data, characterized by: include: Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to train the spatial feature extraction backbone network in the first stage to extract the spatial features of natural images and obtain a spatial feature extractor. The metric-based meta-learning method performs the first-stage training of the spatial feature extraction backbone network as follows: In each meta-training task, a set of labeled natural images from the source domain Randomly select N classes from the dataset, and then randomly select K labeled samples from the N classes as the support set , randomly select T labeled data from the remaining labeled samples of the selected category to build the query set ; Calculate the sample by formula 1 Probability of belonging to class k: Formula 1: in, Extract backbone network for spatial features, Is the support set The centroid of the kth class in , d(·) represents the Euclidean distance between two vectors; The first stage of training uses a small sample classification loss function based on cross entropy. The cross entropy-based FSL loss function is calculated using Formula 2: Formula 2: and Refers to the support set and query set constructed from natural image data labeled in heterogeneous source domains; Using labeled hyperspectral images in heterogeneous source and target domains, a small sample size classification method (FSL) is used to perform the second-stage training of a spatial-spectral feature extraction network consisting of a spatial feature extractor and a spectral feature extractor. This method obtains the spatial-spectral features of domain-adaptive hyperspectral images (HSI) and a spatial-spectral feature extractor. The small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI. The specific method is as follows: Combined with the spatial feature extractor described in the first stage, a 2D convolution layer and a 2D batch normalization layer are added to the front of it to reduce the number of channels of the input data to 3 channels, thus obtaining the spatial feature extractor of the second stage; A spectral feature extraction branch is designed using 1D convolution, and combined with the spatial feature extractor in the second stage to obtain a dual-branch spatial-spectral feature extractor; Taking a square block centered on each pixel of the HSI as input data, processing the input data, and uniformly mapping the dimensions of the input data to 100 through a mapping layer constructed by a two-dimensional convolutional neural network; Based on the processed input data, training the dual-branch spatial-spectral feature extractor using a metric-based meta-learning method; The spatial and spectral features of the hyperspectral image are extracted through the trained dual-branch spatial-spectral feature extractor; By connecting the extracted spatial features and spectral features, the embedded spatial spectral features are obtained; Unlabeled hyperspectral images in the target domain are classified by the proposed spatial-spectral feature extractor.

2. The method according to claim 1, characterized in that The method further comprises: The rotation prediction auxiliary task is introduced to enhance the robustness of the backbone network to different orientations of objects in HSI. Specifically: The set of labeled natural images in the source domain The image in the four directions is rotated , using a prediction head consisting of an activation function and a fully connected layer To predict the score of the rotation category, the resulting cross entropy loss is calculated by formula 3: Formula 3: in, The extended dataset is obtained by rotating the image in four directions; express The rotated image of the original image x in .

3. The method according to claim 1, characterized in that The specific method of using the metric-based meta-learning method to train the dual-branch spatial-spectral feature extractor is as follows: N-way K-shot is used to construct the corresponding support set and query set from the labeled hyperspectral images in the source domain and the hyperspectral images in the target domain, respectively. Formula 4 is used as the small sample classification loss function for the second stage training: Formula 4: in and are the FSL losses in SD and TD of HSI, respectively.

4. The method according to claim 1, wherein The method further includes adopting a contrastive learning method based on category matching to promote similar representations of similar categories across domains and alleviate domain shift: and Denotes the mean of the spatial spectral features of the kth class in the HSI source domain and the target domain, respectively. For each class in the hyperspectral image in the target domain, its best matching class is found in the hyperspectral image in the source domain according to the cosine similarity. The matching class with the highest similarity in the two domains is regarded as a positive sample, and the remaining unmatched classes are regarded as negative samples. The contrast loss based on class matching is calculated using Formula 5: Formula 5: in, , is the indicator function and τ is the temperature parameter.

5. The method according to claim 4, characterized in that The method further includes employing a self-supervised contrastive loss to promote consistent representation of similar samples within a class: In the hyperspectral image of the target domain, the labeled data is expanded by data augmentation, and the self-supervised contrast loss is calculated using Formula 6: Formula 6: in, represents the Kullback-Leibler divergence, H(·) represents the entropy of a specific probability distribution, B represents the batch size, and represents the original view and the enhanced view, and represents their class distributions, which are obtained by inputting spatial spectral features into the softmax classifier; The total training loss for the second stage is calculated using Formula 7: Formula 7: Among them, λ and η are two parameters used to balance different terms.

6. A hyperspectral image cross-domain small sample classification system based on heterogeneous data, applied to the hyperspectral image cross-domain small sample classification method based on heterogeneous data according to claim 1, characterized in that: include: The first-stage training module is used to use labeled natural images in heterogeneous source domains and perform the first-stage training of the spatial feature extraction backbone network based on the metric meta-learning method to extract the spatial features of natural images and obtain the spatial feature extractor; The second stage training module is used to use the hyperspectral images labeled in the heterogeneous source domain and the target domain, and adopt the small sample classification method FSL to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor, so as to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the hyperspectral image classification model; Image classification module: used to classify unlabeled hyperspectral images in the target domain through the hyperspectral image classification model.

7. An electronic device, characterized in that: include: processor; as well as, A memory arranged to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the steps of the cross-domain small sample classification method for hyperspectral images based on heterogeneous data according to any one of claims 1 to 5.

8. A storage medium, characterized in that: Used to store computer-executable instructions, which, when executed, implement the steps of the cross-domain small sample classification method for hyperspectral images based on heterogeneous data as described in any one of claims 1 to 5.

Images