A hyperspectral small sample classification method, system, device and medium

Abstract

The present application relates to the technical field of hyperspectral remote sensing image classification, and discloses a hyperspectral small sample classification method, system, device and medium. The method comprises: obtaining a source domain data set and a target domain data set of a hyperspectral image; constructing a multi-level spatial spectral feature fusion network (including a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module), extracting spatial spectral local fusion features in the multi-level spatial spectral feature encoding module, obtaining global features in the adaptive hierarchical feature fusion module, and constructing a domain distribution feature loss function; calculating feature distances using a feature set matching module, and constructing a multi-level feature matching loss function; constructing a total loss function by combining these two types of loss functions to train the multi-level spatial spectral feature fusion network and obtain an initial classification model; inputting the target domain data set into the model for adaptive adjustment to achieve accurate prediction of hyperspectral small sample classification.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral remote sensing image classification, and in particular to a hyperspectral small sample classification method, system, equipment and medium. Background Art

[0002] As an important branch of remote sensing technology, hyperspectral image (HSI) classification technology makes full use of spatial and spectral information to make the identification of different objects more accurate. In actual production, hyperspectral image classification technology is widely used in many fields, such as crop classification in precision agriculture, vegetation identification in forestry monitoring, and mineral identification in mineral exploration. With the advancement of science and technology, especially the rapid development of remote sensing technology, hyperspectral image classification technology has become an important part of modern intelligent identification. Due to its high information richness and fine recognition ability, hyperspectral image classification technology has received widespread attention and research.

[0003] Traditional machine learning methods face many challenges when processing hyperspectral image data, such as large data volume, high dimensionality, and difficulty in sample labeling. However, with the rapid development of deep learning technology, its application in the field of hyperspectral remote sensing image classification has achieved certain results. Deep learning constructs a neural network structure to automatically extract features from images, effectively solving the problem of manual feature extraction in traditional machine learning methods. In addition, deep learning can also process large-scale hyperspectral image data, improving the accuracy and efficiency of classification.

[0004] Although deep learning has achieved certain results in hyperspectral image classification, hyperspectral image classification based on small sample learning (FSL) still faces major challenges due to the limited number of hyperspectral labeled samples and the domain gap problem between heterogeneous images. First, the acquisition and processing costs of hyperspectral images are high, resulting in a limited number of labeled samples, which limits the training effect of deep learning models. Second, there are large differences in hyperspectral images from different regions and under different environmental conditions, that is, the domain gap problem between heterogeneous images, which affects the generalization ability of the model and makes it difficult to achieve high-precision and strong generalization hyperspectral image classification. Summary of the invention

[0005] Based on this, this application proposes a hyperspectral small sample classification method, system, equipment and medium, among which a new multi-level spatial-spectral feature fusion network (HSSFF-Net) is proposed. This network structure improves the expression ability of features by fusing spatial information and spectral information, and improves the generalization ability of the model by learning small samples. It provides new ideas and methods for solving the hyperspectral image classification problem in actual production, and can achieve high-precision and strong generalization hyperspectral image classification.

[0006] The first aspect of the present application provides a hyperspectral small sample classification method, the method comprising:

[0007] Acquire a source domain dataset and a target domain dataset of a hyperspectral image, wherein the source domain dataset includes a support set and a query set, and the target domain dataset includes a target domain support set, a labeled query set, and an unlabeled query set, wherein the target domain support set and the labeled query set constitute a target domain training dataset, and the unlabeled query set serves as a test dataset;

[0008] A multi-level spatial spectral feature fusion network is constructed, wherein the multi-level spatial spectral feature fusion network includes a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module, wherein:

[0009] Inputting the support set and the query set into the multi-level spatial-spectral feature encoding module to extract the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set;

[0010] Inputting the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set into the adaptive hierarchical feature fusion module to obtain global features, and constructing a domain distribution feature loss function according to the global features;

[0011] Inputting the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set into the feature set matching module to obtain feature distances, and constructing a multi-level feature matching loss function according to the feature distances;

[0012] Constructing a total loss function according to the domain distribution feature loss function and the multi-level feature matching loss function; based on the total loss function, training the multi-level spatial spectral feature fusion network using the support set and the query set to obtain an initial hyperspectral small sample classification model;

[0013] Inputting the target domain training data set into the initial hyperspectral small sample classification model for adaptive adjustment to obtain a target hyperspectral small sample classification model;

[0014] The test data set is input into the target hyperspectral small sample classification model to obtain a hyperspectral small sample prediction classification result.

[0015] As an optional implementation manner of the first aspect, the step of inputting the support set and the query set into the multi-level spatial-spectral feature encoding module to extract the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set includes: obtaining first spatial data and first spectral data according to the support set, and obtaining second spatial data and second spectral data according to the query set; the multi-level spatial-spectral feature encoding module includes a spatial local feature extraction unit and a spectral local feature extraction unit, the spatial local feature extraction unit is used to extract first spatial local features and second spatial local features of different levels according to the first spatial data and the second spatial data, respectively, and the spectral local feature extraction unit is used to extract first spatial local features and second spatial local features of different levels according to the first spatial data and the second spatial data, and the spectral local feature extraction unit is used to extract first spatial local features and second spatial local features of different levels according to the query set. The first spectral data and the second spectral data respectively extract the first spectral local features and the second spectral local features at different levels; the first spatial local features and the first spectral local features are concatenated and input into the NetVLAD network to obtain the first spatial spectral local fusion features, and the second spatial local features and the second spectral local features are also concatenated and input into the NetVLAD network to obtain the second spatial spectral local fusion features; the first spatial spectral local fusion features and the second spatial spectral local fusion features are respectively input into the multi-layer perceptron for deep feature fusion to obtain the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set at different levels.

[0016] As an optional implementation of the first aspect, the spatial local feature extraction unit is used to extract first spatial local features and second spatial local features of different levels according to the first spatial data and the second spatial data, respectively: the spatial data includes the first spatial data and the second spatial data, the spatial local features include the first spatial local features and the second spatial local features, and the spatial local feature extraction unit is a four-layer spatial residual module, wherein the spatial data is input into the four-layer spatial residual module to obtain four different levels of spatial local features respectively. F 1 , F 2 , F 3 and F 4 .

[0017] As an optional implementation of the first aspect, the spectral local feature extraction unit is used to extract the first spectral local features and the second spectral local features of different levels according to the first spectral data and the second spectral data respectively in the step: the spectral data includes the first spectral data and the second spectral data, the spectral local features include the first spectral local features and the second spectral local features, and the spectral local feature extraction unit is a four-layer spectral residual module, wherein the spectral data is input into the four-layer spectral residual module to obtain four different levels of spectral local features respectively. G 1 , G 2 , G 3 and G 4 .

[0018] As an optional implementation of the first aspect, the step of inputting the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set into the adaptive hierarchical feature fusion module to obtain the global features comprises: F 1 , F 2 , F 3 and F 4 and four different levels of spectral local features G 1 , G 2 , G 3 and G 4 Feature fusion is performed at the same level to obtain four different levels of spatial-spectral local fusion features. , , and ; Local fusion features of four different levels of spatial spectrum , , and Concatenate into local feature sets ; Through the first MLP and Softmax function, the concatenated Perform saliency detection to obtain the saliency results corresponding to the local fusion features of the spatial spectrum at each level; use the second MLP to analyze the local feature set Perform feature reconstruction to obtain a reconstruction result; multiply the saliency result and the reconstruction result to obtain a fusion feature; and combine the fusion feature with the local feature set. After addition, the global feature is obtained by passing through the third MLP .

[0019] As an optional implementation manner of the first aspect, the expression of the domain distribution feature loss function is: ,in, represents the total number of query set samples, k Representation sample The real label, represents the Euclidean distance.

[0020] As an optional implementation of the first aspect, the step of constructing a multi-level feature matching loss function according to the feature distance includes: calculating the corresponding predicted label probability according to the feature distance, and constructing a multi-level feature matching loss function according to the predicted label probability, wherein the expression of the multi-level feature matching loss function is: ,in, , indicating the sample Belongs to the true label k The predicted label probability of represents the local feature set extracted by the multi-level spatial-spectral feature encoding module, represents the feature distance between two feature sets, represents the feature set of the kth category, C represents the number of categories, Represents a sample, Representation and The corresponding category label.

[0021] A second aspect of the present application provides a hyperspectral small sample classification system, the system comprising:

[0022] A data acquisition module is used to acquire a source domain dataset and a target domain dataset of a hyperspectral image, wherein the source domain dataset includes a support set and a query set, and the target domain dataset includes a target domain support set, a labeled query set, and an unlabeled query set, wherein the target domain support set and the labeled query set constitute a target domain training dataset, and the unlabeled query set serves as a test dataset;

[0023] Constructing a preliminary model overall module, including a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module, wherein: the support set and the query set are input into the multi-level spatial spectral feature encoding module to extract the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set; the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set are input into the adaptive hierarchical feature fusion module to obtain global features, and a domain distribution feature loss function is constructed according to the global features; the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set are input into the feature set matching module to obtain feature distance, and a multi-level feature matching loss function is constructed according to the feature distance;

[0024] A preliminary model training module is used to construct a total loss function according to the domain distribution feature loss function and the multi-level feature matching loss function; based on the total loss function, the multi-level spatial spectral feature fusion network is trained using the support set and the query set to obtain an initial hyperspectral small sample classification model;

[0025] A model adjustment module, used for inputting the target domain training data set into the initial hyperspectral small sample classification model for adaptive adjustment to obtain a target hyperspectral small sample classification model;

[0026] The small sample prediction module is used to input the test data set into the target hyperspectral small sample classification model to obtain a hyperspectral small sample prediction classification result.

[0027] The third aspect of the present application provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the above-mentioned hyperspectral small sample classification method.

[0028] A fourth aspect of the present application provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned hyperspectral small sample classification method.

[0029] Compared with the prior art, the present application provides a method for hyperspectral small sample classification, which realizes the feature extraction and fusion of the hyperspectral source domain data set and the target domain data set through a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module, thereby improving the accuracy of hyperspectral small sample classification. Specifically, by inputting the support set and the query set into the multi-level spatial spectral feature encoding module, the support set spatial spectral local fusion features and the query set spatial spectral local fusion features are extracted, which provides a basis for subsequent feature fusion and classification. Then, the support set spatial spectral local fusion features and the query set spatial spectral local fusion features are input into the adaptive hierarchical feature fusion module to obtain global features, and a domain distribution feature loss function is constructed based on the global features to ensure that the model has better adaptability to features of different domains. At the same time, the support set spatial spectral local fusion features and the query set spatial spectral local fusion features are input into the feature set matching module to obtain the feature distance, and a multi-level feature matching loss function is constructed based on the feature distance to promote the model to match and classify features between different samples. Finally, the total loss function was constructed based on the domain distribution feature loss function and the multi-level feature matching loss function, and the multi-level spatial spectral feature fusion network was trained using the support set and query set to obtain the initial hyperspectral small sample classification model. By inputting the target domain training dataset into the initial hyperspectral small sample classification model for adaptive adjustment, the target hyperspectral small sample classification model was obtained, which improved the model's generalization ability and adaptability to new data. The test dataset was input into the target hyperspectral small sample classification model, and the hyperspectral small sample prediction classification results were obtained, verifying the effectiveness and accuracy of the model.

[0030] Additional aspects and advantages of the present application will be given in part in the following description, and in part will become apparent from the following description, or will be understood through the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a hyperspectral small sample classification method proposed in the first embodiment of the present application;

[0032] Figure 2 This is a network structure diagram of a multi-level spatial spectral feature fusion network (HSSFF-Net) in the first embodiment of the present application;

[0033] Figure 3 This is a network structure diagram of the spatial residual module in the first embodiment of the present application;

[0034] Figure 4 This is a network structure diagram of the spectral residual module in the first embodiment of the present application;

[0035] Figure 5This is a network structure diagram of the spatial-spectral local feature fusion (SSLFF) in the first embodiment of the present application;

[0036] Figure 6 This is a network framework diagram corresponding to different feature similarity measurement methods in the first embodiment of the present application;

[0037] Figure 7 This is a comparison diagram of the classification obtained by this method and other methods on the Indian Pines dataset in the second embodiment of this application;

[0038] Figure 8 This is a comparison chart of the classification obtained by this method and other methods on the Salinas dataset in the second embodiment of this application;

[0039] Fig. 9 This is a comparison diagram of the classification obtained by this method and other methods on the University of Pavia dataset in the second embodiment of this application;

[0040] Fig.10 This is a comparison diagram of the classification obtained by this method and other methods on the Pavia Center dataset in the second embodiment of this application;

[0041] Fig.11 This is a schematic diagram of the structure of a hyperspectral small sample classification system proposed in the third embodiment of the present application.

[0042] The following specific implementation methods will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0045] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0046] Example 1

[0047] See also Figure 1 , which is a flow chart of a hyperspectral small sample classification method proposed in the first embodiment of the present application, and the proposed method includes S01 to S08.

[0048] S01: Obtain a hyperspectral source domain dataset and a target domain dataset. The source domain dataset includes a support set and a query set. The target domain dataset includes a target domain support set, a labeled query set, and an unlabeled query set. The target domain support set and the labeled query set constitute a target domain training dataset, and the unlabeled query set is used as a test dataset.

[0049] For ease of understanding, the terms in this application are explained as follows:

[0050] 1. The source domain dataset is the initial dataset used to train the model, which contains rich spectral and spatial information. In transfer learning and domain adaptation, the source domain dataset usually comes from one domain or task and is used to help solve data analysis problems in another different but related target domain or task. The source domain dataset provides initial knowledge and patterns for building models and algorithms.

[0051] 2. The target domain dataset is the dataset of the actual problem that needs to be solved. The target domain dataset is usually used to test and optimize the model to ensure that the model performs well in a new and unknown environment. In the context of transfer learning, the target domain dataset usually contains data without labels or only partially labeled, and needs to be labeled or classified using the knowledge in the source domain dataset.

[0052] 3. The support set is an important data subset in the small sample learning task. It contains a small number of labeled samples, which are used to provide category information and help the model learn the feature representation of each category. For example, in a n -way k -shot small sample learning tasks ( n represents the number of categories, k represents the number of samples in each category), the support set contains n categories, each with k A labeled sample.

[0053] 4. The query set is another subset of data used to evaluate model performance. It also contains samples whose labels may be known (used to evaluate model performance in the training and validation phases) or unknown (simulating real prediction scenarios in the testing phase). The samples in the query set and the samples in the support set come from the same category distribution, but usually have no intersection with the support set.

[0054] In this step, Few-Shot Learning (FSL) is a task-based learning method that usually contains two datasets with different data distributions, namely the source domain dataset and the (Include object categories) and target domain datasets (Include In order to ensure the diversity of the training sample set, Usually greater than . According to whether they are labeled or not, they are divided into two different data sets, namely the labeled data set ( ) and unlabeled datasets ( ),and ∪ = .

[0055] Training data set construction: from Randomly select C classes from the dataset, and select K labeled samples from each class to build a support set Then, randomly select N objects from the same C class that have Constructing query sets from different labeled samples .therefore, Each sample in Finding the exact corresponding category in is a small sample classification problem of C class K samples, which is called an episode. Samples in different episodes are randomly selected and dynamically changed.

[0056] It should be noted that, in this embodiment, for the training of the source domain dataset, the same network structure is used to train and Extract features from samples and obtain The accurate label of each sample in The training process is iteratively performed in different episodes until convergence. The process of training tasks in The process of training tasks is the same as in . However, due to The samples in the target domain are limited and cannot generate enough training sets for the target domain. Therefore, data augmentation methods are needed to Add random Gaussian noise to increase the number of samples in each category. For a given training instance , a new extended sample set can be generated by random Gaussian noise technique using formula (1) :

[0057]

[0058] in, represents the transformation function, γ is the random Gaussian noise weight n control factor.

[0059] S02: Construct a multi-level spatial spectral feature fusion network, which includes a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module.

[0060] For details, please refer to Figure 2 , which is a network structure diagram of the multi-level spatial spectral feature fusion network (HSSFF-Net) proposed in the first embodiment of the present application.

[0061] S03: Input the support set and the query set into a multi-level spatial-spectral feature encoding module to extract the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set.

[0062] It should be noted that hyperspectral remote sensing images contain rich spatial and spectral information. In order to fully exploit these two types of feature information, a hierarchical spatial spectral feature encoding module (HSSFE) is designed to extract spatial and spectral features at different levels in the local area, providing more discriminative feature expressions for subsequent feature set matching.

[0063] like Figure 2 As shown in Figure 2, due to the inconsistency between the source domain dataset and the target domain dataset, convolution kernels of different sizes are used to adaptively align the input samples to , c is the number of bands of the hyperspectral image (c is 100 in this method). The aligned samples are split into spatial data and spectral data The spatial data includes first spatial data and second spatial data, and the spectral data includes first spectral data and second spectral data.

[0064] In some embodiments, first, the first spatial data and the first spectral data can be obtained according to the support set, and the second spatial data and the second spectral data can be obtained according to the query set. A spatial local feature extraction unit and a spectral local feature extraction unit are designed in the multi-level spatial spectral feature encoding module. The spatial local feature extraction unit is used to extract the first spatial local features and the second spatial local features of different levels according to the first spatial data and the second spatial data, and the spatial local features include the first spatial local features and the second spatial local features. The spectral local feature extraction unit is used to extract the first spectral local features and the second spectral local features of different levels according to the first spectral data and the second spectral data, and the spectral local features include the first spectral local features and the second spectral local features.

[0065] Specifically, the spatial local feature extraction unit is a four-layer spatial residual module, wherein the spatial data is input into the four-layer spatial residual module to obtain four different levels of spatial local features. F 1 , F 2 , F 3 and F 4 The spectral local feature extraction unit is a four-layer spectral residual module, in which the spectral data is input into the four-layer spectral residual module to obtain four different levels of spectral local features. G 1 , G 2 , G 3 and G 4 .

[0066] For easier understanding, see Figure 3 , is the network structure diagram of the spatial residual module. The data processing process of the spatial residual module includes: input spatial data Then through two branches, branch one undergoes three 2D convolutions (each convolution result needs to go through batch normalization BN and ReLU activation function) to obtain the feature f 1 ; Branch 2 only uses 2D convolution for feature extraction to obtain features f 2 ,Will f 1 and f 2 Add and input the results into the maximum pooling and ReLU activation function in turn to obtain spatial local features .

[0067] For easier understanding, see Figure 4, is the network structure diagram of the spectral residual module. The data processing process of the spectral residual module includes: input spectral data After passing through two branches, branch one undergoes two 1D convolutions (each convolution result needs to go through batch normalization BN and ReLU activation function) to obtain the feature g 1 ; Branch 2 only uses 1D convolution for feature extraction to obtain features g 2 ,Will g 1 and g 2 Add them together and input the result of the addition into the ReLU activation function to obtain the local features of the spectrum .

[0068] Secondly, the first spatial local feature and the first spectral local feature are concatenated and input into the NetVLAD network to obtain the first spatial-spectral local fusion feature. The second spatial local feature and the second spectral local feature are also concatenated and input into the NetVLAD network to obtain the second spatial-spectral local fusion feature. The first spatial-spectral local fusion feature and the second spatial-spectral local fusion feature are respectively input into the multi-layer perceptron for deep feature fusion to obtain the spatial-spectral local fusion features of the support set at different levels and the spatial-spectral local fusion features of the query set.

[0069] For easier understanding, see Figure 5 , which is the network structure diagram of spatial-spectral local feature fusion (SSLFF). In this step, a multi-layer perceptron (MLP, with 10, 5, and D neurons) is finally used for deep fusion processing to obtain the first i The spatial-spectral local fusion features at different levels Specifically, four different levels of spatial local features are F 1 , F 2 , F 3 and F 4 and four different levels of spectral local features G 1 , G 2 , G 3 and G 4 Feature fusion is performed at the same level to obtain four different levels of spatial-spectral local fusion features. , , and .

[0070] Furthermore, in the network structure of the NetVLAD network, the data processing process of the NetVLAD network includes: given an N-dimensional local image descriptor (a first spatial local feature or a first spectral local feature) As input, NetVLAD assigns these local descriptors to k cluster centers and calculates the sum of the difference vectors between these cluster centers and the descriptors assigned to them. Therefore, the VLAD feature is a K×D dimensional feature vector. Assume Indicates the local descriptor j Features, represents the kth cluster center j features, then relative to the kth cluster j The NetVLAD features can be described as:

[0071]

[0072] in, It is the feature expression output by NetVLAD (first spatial-spectral local fusion feature or second spatial-spectral local fusion feature); yes Belongs to class k The probability of allocation; Use the Softmax function for calculation:

[0073]

[0074] S04: Input the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set into the adaptive hierarchical feature fusion module to obtain the global features, and construct the domain distribution feature loss function according to the global features.

[0075] It should be noted that there is a domain gap problem between different domain data, which makes it difficult for the network to construct commonalities between features from different source data, which is not conducive to the subsequent acquisition of high-quality feature matching results, thereby affecting the final hyperspectral remote sensing image classification accuracy. Therefore, this application designs an adaptive hierarchical feature fusion module to solve the problem of inconsistent domain feature distribution.

[0076] In this step, firstly, the four different levels of spatial spectrum local fusion features output by the spatial spectrum feature fusion module are , , and Concatenate into local feature sets Since the contribution of fused features at different levels to classification is inconsistent, the significance of features at different levels needs to be considered when performing feature fusion.

[0077] Specifically, the first MLP and Softmax function are connected in series. Perform saliency detection to obtain the saliency results corresponding to the local fusion features of the spatial spectrum at each level; use the second MLP to analyze the local feature set Perform feature reconstruction to obtain the reconstruction result; multiply the saliency result and the reconstruction result to obtain the fusion feature; combine the fusion feature with the local feature set After addition, the global features are obtained through the third MLP. .

[0078] The detailed calculation formula of this module is as follows:

[0079]

[0080]

[0081] in, represents a hyperspectral input image patch; Represents the local feature set extracted using the multi-level spatial-spectral feature encoding module; Represents the global features of the output image patch.

[0082] Secondly, in order to reduce the impact of domain differences, this application uses the cross entropy formula to calculate the domain distribution feature loss :

[0083]

[0084] in, represents the total number of query set samples, represents the global features obtained by aggregating local features, k Representation sample The real label, represents the Euclidean distance.

[0085] S05: Input the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set into the feature set matching module to obtain the feature distance, and construct a multi-level feature matching loss function according to the feature distance.

[0086] It should be noted that the spatial spectrum local fusion features of the input image block are obtained through the multi-level spatial spectrum feature encoding module. In order to obtain high-precision classification results, the query set Each feature and support set To achieve high-quality matching of features of similar objects, this application adopts a feature set matching strategy based on Hausdoff distance. Compared with the Euclidean distance method, it can more accurately measure the overall similarity between features by integrating different levels of feature information of HSI pixel blocks.

[0087] Matching metric based on Hausdorff distance: Hausdorff distance (HD) is a maximum-minimum distance metric used to measure the similarity between two sets. and , HD is calculated according to formula (7):

[0088]

[0089] in, and They represent the one-way Hausdorff distance from set A to set B and vice versa; Represents a point in set A The Euclidean distance between and point b in set B.

[0090] This application proposes an improved method based on the one-way Hausdorff distance to improve the accuracy and robustness of the feature matching process. Formula (8) is used to calculate the sum of the maximum m distances of the nearest points:

[0091]

[0092] in, Indicates selecting the maximum sum of m feature distances; express Samples in The local feature set of i Features express Samples in The local feature set of j Features express and The Euclidean distance between individual features of .

[0093] In this step, if Figure 6 As shown in the figure, it is the network framework diagram corresponding to different feature similarity measurement methods, where: Figure 6 (a) in the figure is a Siamese Network (SN). Figure 6 (b) is the relationship network (RN). Figure 6(c) is the Prototypical Network (PN). Figure 6 (d) in the above is the similarity measurement strategy proposed in this application. The feature distance measurement method proposed in this application is similar to Figure 7 Different from the other three classical FSL-based methods shown in , the advantages of the network are the feature extraction ability and the full use of features at different levels for comprehensive matching. However, the twin network (SN), prototype network (PN) and relational network (RN) can only use the features of the last layer for similarity measurement.

[0094] Furthermore, constructing a multi-level feature matching loss function according to the feature distance includes: calculating the corresponding predicted label probability according to the feature distance, and then constructing a multi-level feature matching loss function according to the predicted label probability.

[0095] Among them, the expression for calculating the probability of predicted labels is:

[0096]

[0097] in, represents the local feature set extracted by the multi-level spatial-spectral feature encoding module (including four levels of local fusion features), represents the distance measure between two feature sets, express The feature set of the kth category in , C represents the number of categories, express The samples in Representation and The corresponding category label.

[0098] The expression of the multi-level feature matching loss function is:

[0099]

[0100] in, represents the total number of samples in the query set, Representation sample Belongs to the true label k The predicted label probability.

[0101] S06: Construct a total loss function based on the domain distribution feature loss function and the multi-level feature matching loss function; based on the total loss function, use the support set and the query set to train the multi-level spatial spectral feature fusion network to obtain the initial hyperspectral small sample classification model.

[0102] It should be noted that in order to guide the network to solve the domain gap problem and ensure the accuracy of feature set matching, this application designs two loss functions: multi-level feature matching loss and domain distribution feature loss.

[0103] Specifically, the expression of the total loss function is:

[0104]

[0105] in, represents the multi-level feature matching loss, λ It is a custom hyperparameter set to 0.2 to balance the supervision effects of the two loss functions on the network.

[0106] S07: Input the target domain training data set into the initial hyperspectral small sample classification model for adaptive adjustment to obtain the target hyperspectral small sample classification model.

[0107] It should be noted that since the data distribution of the target domain may be different from the data during training, the model needs to be fine-tuned to adapt to the new data distribution. This can be achieved by continuing to train the model or using a specific adaptive learning method. For example, methods such as fine-tuning network parameters, adding new layers, or using domain adaptation techniques can be used. After adaptive adjustment, a hyperspectral small sample classification model that adapts to the target domain data is obtained.

[0108] S08: Input the test data set into the target hyperspectral small sample classification model to obtain the hyperspectral small sample prediction classification result.

[0109] In summary, in the first embodiment of the present application, firstly, through the multi-level spatial spectral feature fusion network, the spatial spectral local fusion features of the hyperspectral source domain data set and the target domain data set can be extracted, so as to make full use of the rich spatial information and spectral information of the hyperspectral image. The network includes a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module, which can gradually extract features at different levels, and perform deep feature fusion and adaptive hierarchical feature fusion to obtain a more discriminative feature expression. Secondly, by constructing a domain distribution feature loss function and a multi-level feature matching loss function, the network can be guided to solve the domain gap problem while ensuring the accuracy of feature set matching. These two loss functions can supervise the learning process of the network, so that the network can better adapt to the target domain data and improve the accuracy of hyperspectral small sample classification. In addition, by adaptively adjusting the network parameters and feature expression, the model can adapt to different data distributions and task requirements. After adaptive adjustment on the target domain data set, the target hyperspectral small sample classification model obtained can better adapt to the target domain data and improve the classification performance. Finally, through the method of the present application, high classification accuracy and robustness can be achieved in the hyperspectral small sample classification task. Since the spatial and spectral information of the hyperspectral image is fully utilized and combined with the domain adaptation technology, the model has good generalization ability on the target domain data. Therefore, the first embodiment of the present application provides an efficient and accurate hyperspectral small sample classification method, which can be widely used in the fields of remote sensing image processing, geographic information system, environmental monitoring, resource exploration and other fields of hyperspectral small sample classification tasks. For example, in remote sensing image processing, it can be used for tasks such as ground object classification and target recognition; in the field of geographic information system, it can be used for map updating, land use planning, etc.; in the field of environmental monitoring, it can be used for pollutant identification, ecological assessment, etc.

[0110] Example 2

[0111] Based on the first embodiment proposed in the present application, in the second embodiment of the present application, the hyperspectral small sample classification result is compared with the ground truth, and the accuracy evaluation is performed. The comparison result and the accuracy evaluation result can be used as a feedback mechanism to further optimize the hyperspectral small sample classification method.

[0112] First, we obtain the hyperspectral images to be classified from the Indian Pines dataset, the Salinas dataset, the University of Pavia dataset, and the Pavia Center dataset, and compare the true value images of the hyperspectral images with the classified images optimized by the method of this application. The comparison results are shown in Figure 7 , Figure 8 , Fig. 9 and Fig.10 ,in, Figure 7This is a classification comparison diagram of this method and other methods on the IndianPines dataset. Figure 8 This is a classification comparison diagram of this method and other methods on the Salinas dataset. Fig. 9 This is a classification comparison chart obtained by this method and other methods on the University of Pavia dataset. Fig.10 This is a classification comparison chart of this method and other methods on the Pavia Center dataset.

[0113] Furthermore, this application uses three indicators, namely, Overall Accuracy (OA), Average Accuracy (AA) and Kappa coefficient, to evaluate the hyperspectral image classification results, where the OA calculation formula is as follows:

[0114]

[0115] in, represents the total number of correctly classified pixels, Indicates the total number of pixels.

[0116]

[0117] Where C is the total number of categories, Indicates that it is correctly classified k The number of pixels of the class, express k The total number of class pixels.

[0118]

[0119] in, is the observed agreement ratio, representing OA, represents the expected agreement ratio under accidental circumstances, which can be calculated using formula (15):

[0120]

[0121] in, Indicates the confusion matrix i Row and j The elements at the intersection of the columns represent the elements that are misclassified as j Class instead of i The number of pixels in the class.

[0122] Specifically, please refer to the accuracy evaluation results in Tables 1 to 5 below, where the compared classification methods mainly include: HybridSN, DFSL, RN-FSL, DCFSL, HFSL, FSCF-SSL and FDFSL. It should be noted that:

[0123] The HybridSN method is a hybrid convolutional neural network for hyperspectral image classification, which integrates the advantages of three-dimensional convolution and two-dimensional convolution. The three-dimensional convolution part is responsible for extracting joint spatial-spectral feature expressions from continuous spectral bands, while the two-dimensional convolution part further learns more abstract high-level spatial expressions based on three-dimensional convolution. The DFSL method uses a deep residual three-dimensional convolutional neural network to extract the spectral-spatial features of the image, and at the same time learns a metric space so that samples of the same category are more similar in the metric space and enhance the differences between samples of different categories. Finally, the nearest neighbor classifier is used to classify the test samples in the learned metric space. The RN-FSL method is a deep classification model based on relational networks and meta-learning ideas. The feature learning module of the network design extracts deep features from hyperspectral image samples, while the relation learning module performs relational learning by comparing the similarities between different samples, that is, the relationship score between samples of the same category is high, and the relationship score between samples of different categories is low. The DCFSL method is a deep cross-domain small-sample learning method that aims to solve the domain shift problem in hyperspectral image classification. It uses the adaptive strategy of conditional adversarial domains to overcome the domain shift problem and achieve domain distribution alignment. In addition, FSL is performed in both the source and target classes, which can not only discover transferable knowledge in the source class, but also learn the discriminative encoding model of the target class. The HFSL method learns transferable knowledge from the mini-ImageNet dataset, and then fine-tunes on hyperspectral data to extract more discriminative spectral spatial features and domain knowledge, thereby improving the accuracy of HSI classification. The FSCF-SSL method designs a self-supervised learning module with geometric transformation based on HFSL. By setting the rotation label as supervision, it extracts low-level features that can better represent different directions, and then performs self-supervised learning and small-sample learning on the base class to learn transferable spatial meta-knowledge. The FDFSL method is a hyperspectral image classification method based on feature decoupling. From the perspective of decoupled expression learning, it aims to solve the domain shift problem in hyperspectral image classification and reduce the bias of source data on expression, so that the model can implicitly focus on the inherent knowledge of the target domain.

[0124] Table 1 Accuracy of HSSFF-Net on four datasets

[0125]

[0126] Table 2 Comparison of classification results of Indian Pines dataset

[0127]

[0128] Table 3 Comparison of classification results of Salinas dataset

[0129]

[0130] Table 4 Comparison of classification results of University of Pavia dataset

[0131]

[0132] Table 5 Comparison of classification results of Pavia Center dataset

[0133]

[0134] Example 3

[0135] See also Fig.11 , which is a schematic diagram of the structure of a hyperspectral small sample classification system proposed in the third embodiment of the present application, the system includes:

[0136] The data acquisition module 100 is used to acquire a source domain dataset and a target domain dataset of a hyperspectral image, wherein the source domain dataset includes a support set and a query set, and the target domain dataset includes a target domain support set, a labeled query set, and an unlabeled query set, wherein the target domain support set and the labeled query set constitute a target domain training dataset, and the unlabeled query set serves as a test dataset;

[0137] Constructing a preliminary model overall module 200, including a multi-level spatial spectral feature encoding module 201, an adaptive hierarchical feature fusion module 202 and a feature set matching module 203, wherein: the support set and the query set are input into the multi-level spatial spectral feature encoding module to extract the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set; the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set are input into the adaptive hierarchical feature fusion module to obtain global features, and a domain distribution feature loss function is constructed according to the global features; the spatial spectral local fusion features of the support set and the spatial spectral local fusion features of the query set are input into the feature set matching module to obtain feature distances, and a multi-level feature matching loss function is constructed according to the feature distances;

[0138] A training preliminary model module 300 is used to construct a total loss function according to the domain distribution feature loss function and the multi-level feature matching loss function; based on the total loss function, the multi-level spatial spectral feature fusion network is trained using the support set and the query set to obtain an initial hyperspectral small sample classification model;

[0139] A model adjustment module 400 is used to input the target domain training data set into the initial hyperspectral small sample classification model for adaptive adjustment to obtain a target hyperspectral small sample classification model;

[0140] The small sample prediction module 500 is used to input the test data set into the target hyperspectral small sample classification model to obtain a hyperspectral small sample prediction classification result.

[0141] On the other hand, the present application also proposes an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the above-mentioned hyperspectral small sample classification method.

[0142] On the other hand, the present application further proposes a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned hyperspectral small sample classification method.

[0143] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0145] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A hyperspectral small sample classification method, characterized in that: The method comprises: Acquire a source domain dataset and a target domain dataset of a hyperspectral image, wherein the source domain dataset includes a support set and a query set, and the target domain dataset includes a target domain support set, a labeled query set, and an unlabeled query set, wherein the target domain support set and the labeled query set constitute a target domain training dataset, and the unlabeled query set serves as a test dataset; A multi-level spatial spectral feature fusion network is constructed, wherein the multi-level spatial spectral feature fusion network includes a multi-level spatial spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module, wherein: Inputting the support set and the query set into the multi-level spatial-spectral feature encoding module to extract the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set, wherein the multi-level spatial-spectral feature encoding module includes a spatial-local feature extraction unit, a spectral local feature extraction unit and a spatial-spectral local feature fusion network, wherein the spatial-local feature extraction unit is a four-layer spatial residual module, and the spectral local feature extraction unit is a four-layer spectral residual module. The four-layer spatial residual module includes two branches, an addition layer, a maximum pooling layer and a ReLU activation function layer, branch one includes three 2D convolutional layers, three batch normalization BN layers and three ReLU activation function layers, branch two includes a 2D convolutional layer, the four-layer spectral residual module includes two branches, an addition layer and a ReLU activation function layer, branch one includes two 1D convolutional layers, two batch normalization BN layers and two ReLU activation function layers, branch two includes a 1D convolutional layer, and the spatial-spectral local feature fusion network includes a concatenation layer, a NetVLAD network layer, two 1D convolutional layers and an MLP layer; Inputting the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set into the adaptive hierarchical feature fusion module to obtain global features, and constructing a domain distribution feature loss function according to the global features, wherein the adaptive hierarchical feature fusion module includes three MLP layers, a Softmax function layer, an addition layer and a multiplication layer; The spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set are input into the feature set matching module, and the feature set matching module is used to obtain the feature distance, and a multi-level feature matching loss function is constructed according to the feature distance. The calculation formula of the feature distance is: , where Indicates selecting the maximum sum of m feature distances; Represents a sample in the query set The i-th feature in the local feature set; Represents the samples in the support set The jth feature in the local feature set of ; Representation sample and samples The Euclidean distance between individual features of ; Constructing a total loss function according to the domain distribution feature loss function and the multi-level feature matching loss function; based on the total loss function, training the multi-level spatial spectral feature fusion network using the support set and the query set to obtain an initial hyperspectral small sample classification model; Inputting the target domain training data set into the initial hyperspectral small sample classification model for adaptive adjustment to obtain a target hyperspectral small sample classification model; The test data set is input into the target hyperspectral small sample classification model to obtain a hyperspectral small sample prediction classification result.

2. A hyperspectral small sample classification method according to claim 1, characterized in that: The step of inputting the support set and the query set into the multi-level spatial spectrum feature encoding module to extract the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set comprises: Obtain first spatial data and first spectral data according to the support set, and obtain second spatial data and second spectral data according to the query set; The multi-level spatial-spectral feature encoding module includes a spatial local feature extraction unit and a spectral local feature extraction unit, wherein the spatial local feature extraction unit is used to extract first spatial local features and second spatial local features of different levels according to the first spatial data and the second spatial data, respectively, and the spectral local feature extraction unit is used to extract first spectral local features and second spectral local features of different levels according to the first spectral data and the second spectral data, respectively; The first spatial local feature and the first spectral local feature are concatenated and input into the NetVLAD network to obtain a first spatial-spectral local fusion feature, and the second spatial local feature and the second spectral local feature are also concatenated and input into the NetVLAD network to obtain a second spatial-spectral local fusion feature; The first spatial-spectral local fusion feature and the second spatial-spectral local fusion feature are respectively input into a multi-layer perceptron for deep feature fusion to obtain spatial-spectral local fusion features of the support set and spatial-spectral local fusion features of the query set at different levels.

3. A hyperspectral small sample classification method according to claim 2, characterized in that: The spatial local feature extraction unit is used to extract first spatial local features and second spatial local features of different levels according to the first spatial data and the second spatial data respectively: The spatial data includes the first spatial data and the second spatial data, the spatial local features include the first spatial local features and the second spatial local features, and the spatial local feature extraction unit is a four-layer spatial residual module, wherein the spatial data is input into the four-layer spatial residual module to obtain four different levels of spatial local features respectively. F 1. F 2. F 3 and F 4.

4. A hyperspectral small sample classification method according to claim 3, characterized in that: The spectral local feature extraction unit is used to extract first spectral local features and second spectral local features of different levels according to the first spectral data and the second spectral data respectively: The spectral data includes the first spectral data and the second spectral data, the spectral local features include the first spectral local features and the second spectral local features, and the spectral local feature extraction unit is a four-layer spectral residual module, wherein the spectral data is input into the four-layer spectral residual module to obtain four different levels of spectral local features respectively. G 1. G 2. G 3 and G 4.

5. A hyperspectral small sample classification method according to claim 4, characterized in that: The step of inputting the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set into the adaptive hierarchical feature fusion module to obtain the global features comprises: Four different levels of spatial local features F 1. F 2. F 3 and F 4 and four different levels of spectral local features G 1. G 2. G 3 and G 4. Feature fusion is performed at the same level to obtain four different levels of spatial spectrum local fusion features , , and ; The spatial spectrum of four different levels is locally fused. , , and Concatenate into local feature sets ; Through the first MLP and Softmax function, the concatenated Perform saliency detection to obtain the saliency results corresponding to the local fusion features of the spatial spectrum at each level; The second MLP is used to analyze the local feature set Perform feature reconstruction and obtain reconstruction results; Multiplying the significance result and the reconstruction result to obtain a fusion feature; The fusion feature and the local feature set After addition, the global feature is obtained by passing through the third MLP .

6. A hyperspectral small sample classification method according to claim 5, characterized in that: The expression of the domain distribution feature loss function is: ， in, represents the total number of query set samples, k Representation sample The real label, represents the Euclidean distance.

7. A hyperspectral small sample classification method according to claim 6, characterized in that: The step of constructing a multi-level feature matching loss function according to the feature distance includes: The corresponding predicted label probability is calculated according to the feature distance, and a multi-level feature matching loss function is constructed according to the predicted label probability. The expression of the multi-level feature matching loss function is: ， in, , indicating the sample Belongs to the true label k The predicted label probability of represents the local feature set extracted by the multi-level spatial-spectral feature encoding module, represents the feature distance between two feature sets, represents the feature set of the kth category, C represents the number of categories, Represents a sample, Representation and The corresponding category label.

8. A hyperspectral small sample classification system, characterized in that: The system comprises: A data acquisition module is used to acquire a source domain dataset and a target domain dataset of a hyperspectral image, wherein the source domain dataset includes a support set and a query set, and the target domain dataset includes a target domain support set, a labeled query set, and an unlabeled query set, wherein the target domain support set and the labeled query set constitute a target domain training dataset, and the unlabeled query set serves as a test dataset; Construct a preliminary model overall module for constructing a multi-level spatial-spectral feature fusion network, the multi-level spatial-spectral feature fusion network includes a multi-level spatial-spectral feature encoding module, an adaptive hierarchical feature fusion module and a feature set matching module, wherein: the support set and the query set are input into the multi-level spatial-spectral feature encoding module to extract the spatial-spectral local fusion features of the support set and the spatial-spectral local fusion features of the query set, the multi-level spatial-spectral feature encoding module includes a spatial local feature extraction unit, a spectral local feature extraction unit and a spatial-spectral local feature fusion network, the spatial local feature extraction unit is a four-layer spatial residual module, the spectral local feature extraction unit is a four-layer spectral residual module, the four-layer spatial residual module includes two branches, an addition layer, a maximum pooling layer and a ReLU activation function layer, branch one includes three 2D convolution layers, three batch normalization BN layers and three ReLU activation function layers, branch two includes a 2D convolution layer, and the four-layer spectral residual module includes two branches, addition layers and ReLU activation function layers, branch one includes two 1D convolution layers, two batch normalization BN layers and two ReLU activation function layers, branch two includes a 1D convolution layer, and the spatial spectrum local feature fusion network includes a concatenation layer, a NetVLAD network layer, two 1D convolution layers and an MLP layer; the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set are input into the adaptive hierarchical feature fusion module to obtain global features, and a domain distribution feature loss function is constructed according to the global features. The adaptive hierarchical feature fusion module includes three MLP layers, a Softmax function layer, an addition layer and a multiplication layer; the spatial spectrum local fusion features of the support set and the spatial spectrum local fusion features of the query set are input into the feature set matching module, and the feature set matching module is used to obtain feature distances, and a multi-level feature matching loss function is constructed according to the feature distances. The calculation formula of the feature distance is: , where Indicates selecting the maximum sum of m feature distances; Represents a sample in the query set The i-th feature in the local feature set; Represents the samples in the support set The jth feature in the local feature set of ; Representation sample and samples The Euclidean distance between individual features of ; A preliminary model training module is used to construct a total loss function according to the domain distribution feature loss function and the multi-level feature matching loss function; based on the total loss function, the multi-level spatial spectral feature fusion network is trained using the support set and the query set to obtain an initial hyperspectral small sample classification model; A model adjustment module, used for inputting the target domain training data set into the initial hyperspectral small sample classification model for adaptive adjustment to obtain a target hyperspectral small sample classification model; The small sample prediction module is used to input the test data set into the target hyperspectral small sample classification model to obtain a hyperspectral small sample prediction classification result.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement a hyperspectral small sample classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the hyperspectral small sample classification method as described in any one of claims 1 to 7.

Images