Hyperspectral image cross-domain small sample classification method and system based on heterogeneous data

Abstract

The embodiment of the invention provides a hyperspectral image cross-domain small sample classification method and system based on heterogeneous data, and the method comprises the steps: carrying out the first-stage training of a spatial feature extraction backbone network through a natural image marked in a heterogeneous source domain and a metrics-based meta-learning method, extracting the spatial features of the natural image, and carrying out the first-stage training of the spatial feature extraction backbone network; obtaining a spatial feature extractor; performing second-stage training on a spatial spectral feature extraction network consisting of a spatial feature extractor and a spectral feature extractor by using the hyperspectral images marked in the heterogeneous source domain and the target domain and adopting a small sample classification method FSL to obtain spatial spectral features of the domain-adaptive hyperspectral image HSI, and obtaining a spatial spectral feature extractor; and classifying the unmarked hyperspectral images in the target domain through the spatial spectral feature extractor.

Description

Technical Field

[0001] This document relates to the field of image classification technology, and in particular to a cross-domain small sample classification method and system for hyperspectral images based on heterogeneous data. Background Art

[0002] Hyperspectral images (HSI) capture rich spectral information in hundreds of spectral bands and can better distinguish ground cover than traditional multispectral images. In HSI interpretation, classification, as a basic task, has received extensive attention. Recent HSI classification methods mainly use deep learning technology and have achieved remarkable performance in classification. Specific representative methods include convolutional neural network (CNN)-based methods, graph convolutional neural network-based methods, and the recent Transformer-based methods. However, these methods usually require a large amount of labeled data to train the model.

[0003] Few-shot learning (FSL) is an effective technique to alleviate the problem of scarce training data, which is inspired by the human cognitive process of new categories. FSL usually follows a meta-learning framework, including meta-training and meta-testing stages. The former obtains useful meta-knowledge from different meta-training tasks, which can be generalized to the classification of new unseen categories in the meta-testing stage.

[0004] Applying few-shot learning (FSL) to hyperspectral image (HSI) classification has achieved excellent performance with limited training data. However, existing HSI few-shot classification methods often suffer from the problem of insufficient domain transfer knowledge learning, which either comes only from natural images or only from HSI, resulting in insufficient learning of useful meta-knowledge for the target domain. Summary of the invention

[0005] One or more embodiments of this specification provide a cross-domain small sample classification method for hyperspectral images based on heterogeneous data, including:

[0006] Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to train the spatial feature extraction backbone network in the first stage to extract the spatial features of natural images and obtain a spatial feature extractor.

[0007] Using the labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor.

[0008] Unlabeled hyperspectral images in the target domain are classified by the proposed spatial-spectral feature extractor.

[0009] Furthermore, the metric-based meta-learning method performs the first-stage training of the spatial feature extraction backbone network as follows:

[0010] In each meta-training task, we collect labeled natural images D from the source domain sn Randomly select N classes from the dataset, and then randomly select K labeled samples from the N classes as the support set. Randomly select T labeled data from the remaining labeled samples of the selected category to construct the query set

[0011] Calculate the sample x by formula 1 j Probability of belonging to class k:

[0012]

[0013] Among them, F s a (·) is the backbone network for spatial feature extraction, n k is the support set S s The centroid of the kth class in , d(·) represents the Euclidean distance between two vectors;

[0014] The first stage of training uses a small sample classification loss function based on cross entropy, and the cross entropy-based FSL loss function is calculated by Formula 2:

[0015]

[0016] S sn and Q sn Refers to the support set and query set constructed from natural image data labeled in heterogeneous source domains.

[0017] Furthermore, the method further comprises:

[0018] The rotation prediction auxiliary task is introduced to enhance the robustness of the backbone network to different directions of objects in HSI. Specifically:

[0019] The labeled natural image set D in the source domain sn The image in is rotated in four directions R = {0°, 90°, 180°, 270°}, using a prediction head consisting of an activation function and a fully connected layer To predict the score of the rotation category, the resulting cross entropy loss is calculated by formula 3:

[0020]

[0021] in, The extended data set obtained after rotating the image in four directions; x r (r∈R) means The rotated image of the original image x in .

[0022] Furthermore, the small sample classification method FSL is used to perform the second-stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI. The specific method is:

[0023] Combined with the spatial feature extractor in the first stage, a 2D convolution layer and a 2D batch normalization layer are added to the front of the spatial feature extractor to reduce the number of channels of the input data to 3 channels, thereby obtaining the spatial feature extractor in the second stage;

[0024] A spectral feature extraction branch is designed using 1D convolution, and combined with the spatial feature extractor in the second stage to obtain a dual-branch spatial-spectral feature extractor;

[0025] Taking a square block centered on each pixel of the HSI as input data, processing the input data, and uniformly mapping the dimensions of the input data to 100 through a mapping layer constructed by a two-dimensional convolutional neural network;

[0026] Based on the processed input data, training the dual-branch spatial-spectral feature extractor using a metric-based meta-learning method;

[0027] The spatial and spectral features of the hyperspectral image are extracted through the trained dual-branch spatial-spectral feature extractor;

[0028] By connecting the extracted spatial features and spectral features, the embedded spatial-spectral features are obtained.

[0029] Furthermore, the specific method of using the metric-based meta-learning method to train the dual-branch spatial-spectral feature extractor is:

[0030] N-way K-shot is used to construct the corresponding support set and query set from the labeled hyperspectral images in the source domain and the hyperspectral images in the target domain, respectively, and Formula 4 is used as the small sample classification loss function for the second stage training:

[0031]

[0032] in and are the FSL losses in SD and TD of HSI, respectively.

[0033] Furthermore, the method further includes adopting a contrastive learning method based on category matching to promote similar categories across domains to generate similar representations and alleviate domain shift:

[0034] and Represent the mean of the spatial spectral features of the kth class in the HSI source domain and the target domain respectively. For each class in the hyperspectral image in the target domain, its best matching class is found in the hyperspectral image in the source domain according to the cosine similarity. The matching class with the highest similarity in the two domains is regarded as a positive sample, and the remaining unmatched classes are regarded as negative samples. The contrast loss based on class matching is calculated using Formula 5:

[0035]

[0036] in, || is the indicator function and τ is the temperature parameter.

[0037] Furthermore, the method further includes using a self-supervised contrast loss to promote consistent representation of similar samples within a class:

[0038] In the hyperspectral image of the target domain, the labeled data is extended by data augmentation, and the self-supervised contrast loss is calculated using Formula 6:

[0039]

[0040] Among them, D KL (·||·) represents the Kullback-Leibler divergence, H(·) represents the entropy of a specific probability distribution, B represents the batch size, and represents the original view and the enhanced view, P i 1 and P i 2 represents their class distribution, which is obtained by inputting spatial spectral features into the softmax classifier;

[0041] The total training loss for the second stage is calculated using Formula 7:

[0042]

[0043] Among them, λ and η are two parameters used to balance different terms.

[0044] One or more embodiments of this specification provide a hyperspectral image cross-domain small sample classification system based on heterogeneous data, including:

[0045] The first stage training module is used to use the labeled natural images in the heterogeneous source domain to perform the first stage training on the spatial feature extraction backbone network based on the metric meta-learning method to extract the spatial features of the natural images and obtain the spatial feature extractor;

[0046] The second stage training module is used to use the hyperspectral images labeled in the heterogeneous source domain and the target domain, and adopt the small sample classification method FSL to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor, so as to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the hyperspectral image classification model;

[0047] Image classification module: used to classify unlabeled hyperspectral images in the target domain through the hyperspectral image classification model.

[0048] One or more embodiments of the present specification provide an electronic device, including:

[0049] processor; and,

[0050] A memory is arranged to store computer executable instructions, which, when executed, cause the processor to implement the steps of the above-mentioned method for cross-domain small sample classification of hyperspectral images based on heterogeneous data.

[0051] One or more embodiments of the present specification provide a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the above-mentioned method for cross-domain small sample classification of hyperspectral images based on heterogeneous data.

[0052] By adopting the embodiment of the present invention, full use is made of different types of labeled images in heterogeneous source domains, and complementary advantages are achieved through two-stage training to improve the comprehensiveness and accuracy of feature extraction. The obtained spatial spectral feature extractor is used to efficiently and specifically classify the unlabeled hyperspectral images in the target domain, which helps to improve the classification accuracy under limited labeled data and expand the application scenarios of hyperspectral images.

[0053] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1A flowchart of a method for cross-domain small sample classification of hyperspectral images based on heterogeneous data provided by one or more embodiments of this specification;

[0056] Figure 2 A schematic diagram of the overall framework of a hyperspectral image cross-domain small sample classification network based on heterogeneous data provided for one or more embodiments of this specification;

[0057] Figure 3 The classification results of different classification methods compared in the specific embodiments of this specification on the IP data set;

[0058] Figure 4 The classification results of different classification methods compared in the specific embodiments of this specification on the HT data set;

[0059] Figure 5 It is a visualization diagram of the matching degree between the classification diagram and the true value of different methods on IP in the specific embodiments of this specification;

[0060] Figure 6 The effects of different losses on the performance of the method in the specific embodiments of this specification;

[0061] Figure 7 A schematic diagram of the composition of a hyperspectral image cross-domain small sample classification system based on heterogeneous data provided for one or more embodiments of this specification;

[0062] Figure 8 A schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0064] Method Embodiment

[0065] According to an embodiment of the present invention, a cross-domain small sample classification method for hyperspectral images based on heterogeneous data is provided. Figure 1 A flowchart of a method for cross-domain small sample classification of hyperspectral images based on heterogeneous data is provided for one or more embodiments of this specification, such as Figure 1As shown, the cross-domain small sample classification method of hyperspectral images based on heterogeneous data according to an embodiment of the present invention specifically includes:

[0066] S1. Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to perform the first stage training of the spatial feature extraction backbone network to extract the spatial features of natural images and obtain a spatial feature extractor.

[0067] The overall framework of the cross-domain small sample classification network for hyperspectral images based on heterogeneous data is as follows: Figure 2 As shown in the figure, it includes two stages, namely the bone intervention training stage of spatial feature extraction and the cross-domain small sample learning stage. Let D sn and D sh denote the labeled natural images and HSI in the source domain, D t Represents the hyperspectral data in the target domain. The classes in the source domain are taken as base classes, and the classes in the target domain are taken as new classes. The number of base classes is usually greater than the number of new classes. sn and D sh Rich information to support D t The classification of the new category, among which D t Contains only a small amount of labeled data.

[0068] The specific method of the first stage training of the spatial feature extraction backbone network based on the metric meta-learning method is as follows:

[0069] In each meta-training task, we collect labeled natural images D from the source domain sn Randomly select N classes from the dataset, and then randomly select K labeled samples from the N classes as the support set. Randomly select T labeled data from the remaining labeled samples of the selected category to construct the query set

[0070] In the first stage, a metric-based meta-learning method is used to train the spatial extraction backbone network, which makes samples belonging to the same class closer in the feature space and samples belonging to different classes farther away.

[0071] Formula 1 calculates sample x j Probability of belonging to class k:

[0072]

[0073] Among them, F s a (·) is the backbone network for spatial feature extraction, n k is the support set S s is the centroid of the kth class in , and d(·) represents the Euclidean distance between two vectors.

[0074] The first stage of training uses a small sample classification loss function based on cross entropy, and the cross entropy-based FSL loss function is calculated by Formula 2:

[0075]

[0076] S sn and Q sn Refers to the support set and query set constructed from natural image data labeled in heterogeneous source domains.

[0077] Objects in remote sensing usually have different directions, while the directions of objects in natural images are relatively consistent. In order to enhance the spatial feature extractor F s a (·) To improve the robustness of HSI objects in different directions, we introduce a rotation prediction auxiliary task. Specifically:

[0078] The labeled natural image set D in the source domain sn The image in is rotated in four directions R = {0°, 90°, 180°, 270°}, using a prediction head consisting of an activation function and a fully connected layer To predict the score of the rotation category, the resulting cross entropy loss is calculated by formula 3:

[0079]

[0080] in, The extended data set obtained after rotating the image in four directions; x r (r∈R) means The rotated image of the original image x in .

[0081] S2. Using the labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor.

[0082] Combined with the spatial feature extractor in the first stage, a 2D convolution layer and a 2D batch normalization layer are added to the front of the spatial feature extractor to reduce the number of channels of the input data to 3 channels, thereby obtaining the spatial feature extractor in the second stage; a spectral feature extraction branch is designed using 1D convolution, and combined with the spatial feature extractor in the second stage, a dual-branch spatial-spectral feature extractor is obtained; a square block centered on each pixel of the HSI is used as input data, the input data is processed, and the dimension of the input data is uniformly mapped to 100 through a mapping layer constructed by a two-dimensional convolutional neural network; based on the processed input data, the dual-branch spatial-spectral feature extractor is trained using a metric-based meta-learning method.

[0083] The specific method of training the dual-branch spatial-spectral feature extractor based on the metric meta-learning method is as follows:

[0084] N-way K-shot is used to construct the corresponding support set and query set from the labeled hyperspectral images in the source domain and the hyperspectral images in the target domain, respectively, and Formula 4 is used as the small sample classification loss function for the second stage training:

[0085]

[0086] in and are the FSL losses in SD and TD of HSI, respectively.

[0087] The spatial features and spectral features of the hyperspectral image are extracted by a trained dual-branch spatial-spectral feature extractor; the extracted spatial features and spectral features are connected to obtain embedded spatial-spectral features.

[0088] Due to the influence of different changes in categories, land cover, sensors, noise and degradation, the HSI of the source domain and the target domain usually show a large inter-domain gap. Therefore, in order to alleviate the domain shift, a contrastive learning method based on category matching is adopted to promote similar representations of similar classes across domains. Specifically:

[0089] A contrastive learning method based on category matching is used to promote similar representations of similar classes across domains and alleviate domain shift:

[0090] and Represent the mean of the spatial spectral features of the kth class in the HSI source domain and the target domain respectively. For each class in the hyperspectral image in the target domain, its best matching class is found in the hyperspectral image in the source domain according to the cosine similarity. The matching class with the highest similarity in the two domains is regarded as a positive sample, and the remaining unmatched classes are regarded as negative samples. The contrast loss based on class matching is calculated using Formula 5:

[0091]

[0092] in, ‖ is the indicator function and τ is the temperature parameter.

[0093] In the target HSI domain, hyperspectral data usually exhibit large intra-class variations due to noise and spectral mixing. The limited training data in the target domain makes it difficult for the model to provide a consistent representation for intra-class data, which degrades the subsequent classification performance. To alleviate this problem and obtain a smooth classification map, self-supervised learning techniques are used to promote consistent representation of similar samples. In HSI TD, data augmentation is used to expand the labeled data. Let and represents the original view and the enhanced view, P i 1 and P i 2 Represent their class distribution, which is obtained by inputting spatial spectral features into the softmax classifier, and introducing self-supervised contrast loss to promote consistent representation of similar samples within the class:

[0094] In the hyperspectral image of the target domain, the labeled data is extended by data augmentation, and the self-supervised contrast loss is calculated using Formula 6:

[0095]

[0096] Among them, D KL (·‖·) represents the Kullback-Leibler divergence, H(·) represents the entropy of a specific probability distribution, B represents the batch size, and represents the original view and the enhanced view, P i 1 and P i 2 represents their class distribution, which is obtained by inputting spatial spectral features into the softmax classifier;

[0097] The total training loss for the second stage is calculated using Formula 7:

[0098]

[0099] Among them, λ and η are two parameters used to balance different terms.

[0100] S3. Classify the unlabeled hyperspectral image in the target domain through the spatial-spectral feature extractor.

[0101] The trained spatial spectral feature extractor is used to extract the spatial spectral embedding features of the samples, and the features and labels of a small number of labeled samples are used to fit the neural network classifier, and the unlabeled hyperspectral images are classified through the classifier.

[0102] The following is a specific example of cross-domain small sample classification of hyperspectral images based on heterogeneous data:

[0103] We use the natural image dataset MiniImageNet and the HSI dataset Chikusei as heterogeneous source domain data. MiniImageNet contains 100 classes, and the size of each image is 84×84×3; the size of Chikusei is 2517×2335×128, and contains 19 classes. We select Indian Pines (IP) and Houston13 (HT) as the target domain HSI, which contain new classes to be classified. The data size of IP is 145×145×200, containing 16 classes, and the size of HT is 349×1905×155, with 15 classes.

[0104] In this embodiment, VGG16 is used as the backbone network for the first stage of spatial feature extraction. The input block size of natural images is 33×33×3. In each meta-training of FSL, N is set to the number of target HSI categories, K is set to 1, and the number of samples per class in the support set and query set is 1 and 19 respectively. t In , 5 labeled samples are selected for each class of FSL.

[0105] From D sh 80 labeled samples are randomly selected for each class of FSL. For the target domain contrast loss, D is obtained by adding random Gaussian noise. t The optimizer is Adam with a learning rate of 0.001; in the second stage, we set the number of iterations to 1000. For IP and HT, λ is set to 1, η is set to 0.2 and 0.5 respectively, and the batch size is 128. The evaluation indicators include overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ). The compared methods include the traditional deep learning-based supervised classification method SSRN and four recent HSI few-shot classification methods DCFSL, Gia-CFSL

[28] , and FSCF-SSL. Figure 3 The average results of ten runs for all methods are shown.

[0106] Depend on Figure 3 It can be seen that this embodiment performs best in OA, AA and κ. Compared with the traditional supervised method SSRN, all methods based on small sample learning achieved better performance, proving the effectiveness of FSL for HSI classification with limited training data. Compared with DCFSL using only HSI as source domain data, a 14.02% OA improvement was obtained; compared with FSCF-SSL using only natural images as source domain data, a 1.9% OA improvement was obtained; compared with DCFSL and FSCF-SSL, the performance improvement proves the benefit of using both natural images and HSI as source domain data. This method achieved the top 2 category-specific accuracies on 13 categories. Figure 5The visualization results in show that the classification map has the highest match with the true value and is less contaminated by impulse noise than other classification maps.

[0107] Figure 4 The results of different methods on HT are shown in Figure 1. All FSL-based methods achieve better results than the supervised method SSRN. In addition, the results show that the method of this embodiment outperforms DCFSL and FSCF-SSL, thereby promoting discriminative spatial spectral feature extraction.

[0108] Figure 3 and Figure 4 The effectiveness of FSL and fusing heterogeneous data in the source domain has been verified. Here, the impact of different losses on the performance of this method is analyzed. Figure 6 In the above, “Baseline” represents a simplified version of the above method, which uses D sn and D sh , and the model is trained using FSL loss. "Cross-domain" and "Target-domain" are the corresponding cross-domain and target domain contrast losses. The results show that, in general, these two losses can improve the performance on different datasets. Compared with Scheme 2, OA improves by 3.74% and 2.32% on the HT dataset; compared with Baseline, Scheme 4 improves OA by 3.84% on IP and 2.47% on HT.

[0109] The beneficial effects of the present invention are as follows:

[0110] The present invention makes full use of different types of labeled images in heterogeneous source domains, realizes complementary advantages through two-stage training, improves the comprehensiveness and accuracy of feature extraction, and uses the obtained spatial spectral feature extractor to efficiently and specifically classify unlabeled hyperspectral images in the target domain, which helps to improve classification accuracy under limited labeled data and expand the application scenarios of hyperspectral images.

[0111] System Example

[0112] According to an embodiment of the present invention, a hyperspectral image cross-domain small sample classification system based on heterogeneous data is provided. Figure 3 A schematic diagram of a cross-domain small sample classification system for hyperspectral images based on heterogeneous data provided in one or more embodiments of this specification is shown in FIG. Figure 3 As shown, the cross-domain small sample classification system of hyperspectral images based on heterogeneous data according to an embodiment of the present invention specifically includes:

[0113] The first stage training module is used to use the labeled natural images in the heterogeneous source domain to perform the first stage training on the spatial feature extraction backbone network based on the metric meta-learning method to extract the spatial features of the natural images and obtain the spatial feature extractor;

[0114] The second stage training module is used to use the hyperspectral images labeled in the heterogeneous source domain and the target domain, and adopt the small sample classification method FSL to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor, so as to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the hyperspectral image classification model;

[0115] Image classification module: used to classify unlabeled hyperspectral images in the target domain through the hyperspectral image classification model.

[0116] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0117] Device Example 1

[0118] An embodiment of the present invention provides an electronic device, such as Figure 5 As shown, it includes: a memory 50, a processor 52, and a computer program stored in the memory 50 and executable on the processor 25. When the computer program is executed by the processor 52, the following method steps are implemented:

[0119] S1. Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to perform the first-stage training of the spatial feature extraction backbone network to extract the spatial features of natural images and obtain a spatial feature extractor;

[0120] S2. Using the labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor;

[0121] S3. Classify the unlabeled hyperspectral image in the target domain through the spatial-spectral feature extractor.

[0122] Device Example 2

[0123] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by the processor 52, the following method steps are implemented:

[0124] S1. Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to perform the first-stage training of the spatial feature extraction backbone network to extract the spatial features of natural images and obtain a spatial feature extractor;

[0125] S2. Using the labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor;

[0126] S3. Classify the unlabeled hyperspectral image in the target domain through the spatial-spectral feature extractor.

[0127] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-domain small sample classification method for hyperspectral images based on heterogeneous data, characterized in that: include: Using labeled natural images in heterogeneous source domains, a metric-based meta-learning method is used to train the spatial feature extraction backbone network in the first stage to extract the spatial features of natural images and obtain a spatial feature extractor. Using the labeled hyperspectral images in heterogeneous source and target domains, the small sample classification method FSL is used to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the spatial-spectral feature extractor. Unlabeled hyperspectral images in the target domain are classified by the proposed spatial-spectral feature extractor.

2. The method according to claim 1, characterized in that The metric-based meta-learning method performs the first-stage training of the spatial feature extraction backbone network as follows: In each meta-training task, we collect labeled natural images D from the source domain sn Randomly select N classes from the dataset, and then randomly select K labeled samples from the N classes as the support set. Randomly select T labeled data from the remaining labeled samples of the selected category to construct the query set Calculate the sample x by formula 1 j Probability of belonging to class k: in, is the backbone network for spatial feature extraction, n k is the support set S s The centroid of the kth class in , d(·) represents the Euclidean distance between two vectors; The first stage of training uses a small sample classification loss function based on cross entropy. The cross entropy-based FSL loss function is calculated by Formula 2: S sn and Q sn Refers to the support set and query set constructed from natural image data labeled in heterogeneous source domains.

3. The method according to claim 2, characterized in that The method further comprises: The rotation prediction auxiliary task is introduced to enhance the robustness of the backbone network to different directions of objects in HSI. Specifically: The labeled natural image set D in the source domain sn The image in is rotated in four directions R = {0°, 90°, 180°, 270°}, using a prediction head consisting of an activation function and a fully connected layer To predict the score of the rotation category, the resulting cross entropy loss is calculated by formula 3: in, The extended data set obtained after rotating the image in four directions; x r (r∈R) means The rotated image of the original image x in .

4. The method according to claim 1, characterized in that: The small sample classification method FSL is used to perform the second-stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI. The specific method is: Combined with the spatial feature extractor in the first stage, a 2D convolution layer and a 2D batch normalization layer are added to the front of the spatial feature extractor to reduce the number of channels of the input data to 3 channels, thereby obtaining the spatial feature extractor in the second stage; A spectral feature extraction branch is designed using 1D convolution, and combined with the spatial feature extractor in the second stage to obtain a dual-branch spatial-spectral feature extractor; Taking a square block centered on each pixel of the HSI as input data, processing the input data, and uniformly mapping the dimensions of the input data to 100 through a mapping layer constructed by a two-dimensional convolutional neural network; Based on the processed input data, training the dual-branch spatial-spectral feature extractor using a metric-based meta-learning method; The spatial and spectral features of the hyperspectral image are extracted through the trained dual-branch spatial-spectral feature extractor; By connecting the extracted spatial features and spectral features, the embedded spatial-spectral features are obtained.

5. The method according to claim 1, characterized in that The specific method of using the metric-based meta-learning method to train the dual-branch spatial-spectral feature extractor is: N-way K-shot is used to construct the corresponding support set and query set from the labeled hyperspectral images in the source domain and the hyperspectral images in the target domain, respectively. Formula 4 is used as the small sample classification loss function for the second stage training: in and are the FSL losses in SD and TD of HSI, respectively.

6. The method according to claim 1, characterized in that The method further includes adopting a contrastive learning method based on category matching to promote similar categories across domains to generate similar representations and alleviate domain shift: and Represent the mean of the spatial spectral features of the kth class in the HSI source domain and the target domain respectively. For each class in the hyperspectral image in the target domain, its best matching class is found in the hyperspectral image in the source domain according to the cosine similarity. The matching class with the highest similarity in the two domains is regarded as a positive sample, and the remaining unmatched classes are regarded as negative samples. The contrast loss based on class matching is calculated using Formula 5: in, || is the indicator function and τ is the temperature parameter.

7. The method according to claim 6, characterized in that The method further includes using a self-supervised contrastive loss to promote consistent representation of similar samples within a class: In the hyperspectral image of the target domain, the labeled data is extended by data augmentation, and the self-supervised contrast loss is calculated using Formula 6: Among them, D KL (·||·) represents the Kullback-Leibler divergence, H(·) represents the entropy of a specific probability distribution, B represents the batch size, and represents the original view and the enhanced view, and represents their class distribution, which is obtained by inputting spatial spectral features into the softmax classifier; The total training loss for the second stage is calculated using Formula 7: Among them, λ and η are two parameters used to balance different terms.

8. A cross-domain small sample classification system for hyperspectral images based on heterogeneous data, characterized in that: include: The first stage training module is used to use the labeled natural images in the heterogeneous source domain to perform the first stage training on the spatial feature extraction backbone network based on the metric meta-learning method to extract the spatial features of the natural images and obtain the spatial feature extractor; The second stage training module is used to use the hyperspectral images labeled in the heterogeneous source domain and the target domain, and adopt the small sample classification method FSL to perform the second stage training on the spatial-spectral feature extraction network composed of the spatial feature extractor and the spectral feature extractor, so as to obtain the spatial-spectral features of the domain-adaptive hyperspectral image HSI and obtain the hyperspectral image classification model; Image classification module: used to classify unlabeled hyperspectral images in the target domain through the hyperspectral image classification model.

9. An electronic device, characterized in that: include: processor; as well as, A memory arranged to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the steps of the cross-domain small sample classification method for hyperspectral images based on heterogeneous data as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: Used to store computer executable instructions, which, when executed, implement the steps of the cross-domain small sample classification method for hyperspectral images based on heterogeneous data as described in any one of claims 1 to 7.

Images