Cross-domain small sample learning hyperspectral image classification method

Abstract

A cross-domain small sample learning hyperspectral image classification method comprises a data classification step, a hyperspectral image preprocessing step, a deep three-dimensional residual network spatial spectrum feature extraction step, a small sample learning step, a domain alignment step and a network training step. According to the method, challenges of insufficient feature extraction, low resolution, inter-domain data difference and the like in small-sample hyperspectral image classification are effectively processed, the classification precision of an algorithm under the condition that marked samples are limited is improved through Euclidean distance measurement and image information aggregation, and the classification precision of hyperspectral images under the cross-domain condition is remarkably improved; the method effectively deals with data distribution differences among different domains, provides a new theory and tool for solving the problem of scarcity of hyperspectral image labeled samples, and provides a new thought and method for improving and optimizing hyperspectral image classification tasks.

Description

Technical Field

[0001] The present invention relates to the field of image processing algorithms, and more specifically, to a cross-domain small-shot learning hyperspectral image classification method based on a three-dimensional residual network and graph information aggregation (3D Graph Information Aggregation Cross-Domain Few-Shot Learning, 3DGia-CFSL), which can effectively solve the problem of scarcity of labeled samples of hyperspectral images and improve and optimize hyperspectral image classification tasks. Background Art

[0002] With the rapid development of hyperspectral image technology, hyperspectral images play an important role in many fields such as geological exploration, environmental monitoring and agriculture. However, a major problem faced by existing hyperspectral image classification algorithms is the scarcity of labeled samples. In practical applications, the acquisition area of ​​hyperspectral images is usually complex and varied, including complex environments such as swamps, mountainous areas and rainforests. This makes it difficult to collect labeled samples of hyperspectral images in the field. The extraction process of deep convolutional features is data-dependent, and a large amount of high-quality training data is usually required to obtain excellent model performance. However, due to the complexity and potential danger of the hyperspectral image acquisition area, hyperspectral images must be recognized and processed under limited labeled samples. This increases the difficulty of hyperspectral image analysis. In addition, due to the limitations of different scenes and devices, the dataset may cover data from different scenes and devices. Therefore, it is necessary to design a cross-domain hyperspectral image classification model. Here, cross-domain refers to migrating the model from one domain (source domain) to another domain (target domain) and accurately classifying it in the target domain.

[0003] Therefore, how to effectively solve the problem of scarcity of labeled samples in hyperspectral images and realize cross-domain small sample classification of hyperspectral images has become a technical problem that needs to be urgently solved in existing technologies. Summary of the invention

[0004] The purpose of the present invention is to propose a cross-domain small sample learning hyperspectral image classification method, which is based on a three-dimensional residual network and graph information aggregation, and can effectively solve the problem of scarce labeled samples of hyperspectral images and improve and optimize the hyperspectral image classification task.

[0005] A cross-domain small sample learning hyperspectral image classification method includes the following steps:

[0006] Data classification step S110:

[0007] Input hyperspectral data and corresponding labels. The input data is divided into source domain data and target domain data.

[0008] Hyperspectral image preprocessing step S120:

[0009] Use the principal component analysis method to reduce the dimension of the input data set to obtain the feature map after dimensionality reduction, and then use the neighborhood pixel extraction method to further divide the feature map after dimensionality reduction, cut and divide the data set, and select various types of samples from it as support sets and query sets for subsequent small sample learning;

[0010] Deep three-dimensional residual network spatial spectrum feature extraction step S130:

[0011] This step includes feature mapping and deep three-dimensional residual network spatial spectrum feature extraction, wherein the feature mapping includes a source domain mapping layer M s and the target domain mapping layer M t , through the two-dimensional CNN, the data of the source domain and the target domain are mapped from the original spectral dimension to a common mapping dimension dmap, so that the input dimension of the source domain is equal. The deep three-dimensional residual network spatial spectrum feature extraction includes two low-resolution three-dimensional spatial spectrum feature processing modules, two three-dimensional residual blocks, two three-dimensional pooling layers and a three-dimensional convolution layer, through the spatial spectrum attention mechanism;

[0012] Small sample learning step S140:

[0013] Small sample learning is performed simultaneously on the source domain and the target domain. In each domain, the support set and the query set are input into femb(Ms(·)) to obtain embedded features. In each episode, the small sample learning task is performed by calculating the Euclidean distance between the embedded features of the query set and the prototype of each class, while minimizing the prediction loss of the query set. Finally, the Euclidean distance is calculated by the softmax function to predict the probability of the query sample, and the cross entropy loss function is used for training.

[0014] Domain alignment step S150:

[0015] Convert the classified data into graph data, where the edges contain the spatial distribution information of the data and the nodes contain the spectral dimension information of the data. Specifically for the source domain and the target domain, use the intra-domain feature extraction IFE-BLOCK to extract the respective features of the source domain and the target domain respectively. In each update, calculate the WD distance at the feature level and the distribution level and the GWD distance for edge matching, and minimize the GOT loss to reduce the difference between domains. Then, use the inter-domain similarity measurement module to capture the similarity and distribution similarity of cross-domain features, and use attention aggregation to make them work together to reduce cross-domain differences.

[0016] Network training step S160:

[0017] In summary, the above steps obtain the total loss function. After training a certain number of times, the total loss function is minimized to train the method.

[0018] The present invention also discloses a storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed by a processor, execute the above-mentioned cross-domain small sample learning hyperspectral image classification method.

[0019] In summary, the present invention provides a new theory and tool to solve the problem of scarcity of labeled samples in hyperspectral images, and provides a new idea and method for improving and optimizing hyperspectral image classification tasks, which has the following advantages:

[0020] 1. Through three-dimensional residual network and neighborhood pixel extraction, the challenges of insufficient feature extraction, low resolution and inter-domain data differences in small sample hyperspectral image classification are effectively addressed.

[0021] 2. The small sample learning step and domain alignment step are introduced, and the classification accuracy of the algorithm is improved when there are limited labeled samples through Euclidean distance measurement and graph information aggregation.

[0022] 3. The domain alignment step is introduced to significantly improve the classification accuracy of hyperspectral images in cross-domain situations and effectively address the differences in data distribution between different domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of a cross-domain small sample learning hyperspectral image classification method according to a specific embodiment of the present invention;

[0024] Figure 2 It is a framework diagram of a cross-domain small sample learning hyperspectral image classification method according to a specific embodiment of the present invention;

[0025] Figure 3 is a mapping layer structure diagram of a cross-domain small sample learning hyperspectral image classification method according to a specific embodiment of the present invention;

[0026] Figure 4 It is a flowchart of the steps of extracting spatial spectrum features of a deep three-dimensional residual network of a cross-domain small sample learning hyperspectral image classification method according to a specific embodiment of the present invention;

[0027] Figure 5 is a structural diagram of a LoR-3DSSFPM according to a specific embodiment of the present invention;

[0028] Figure 6 It is a structural diagram of intra-domain feature extraction according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] This paper proposes a cross-domain small sample learning hyperspectral image classification method based on three-dimensional residual network and graph information aggregation, which is as follows:

[0031] (1) Aiming at the situation that hyperspectral image samples are rich in spatial-spectral information, a deep three-dimensional residual network spectral-spatial feature extraction module is proposed. Through three-dimensional convolution and residual network structure, the spatial-spectral information of hyperspectral images is fully mined, thereby improving the cross-domain small sample hyperspectral image classification performance.

[0032] (2) To address the problem of low spatial resolution of hyperspectral images, a three-dimensional spatial-spectral feature processing step focusing on low-resolution hyperspectral images is proposed. Through image feature reconstruction and enhancement technology, the resolution and feature expression capabilities of the image are improved. This helps the model better understand the subtle changes and details in the hyperspectral images, thereby improving the classification results.

[0033] (3) To address the problem of insufficient number of labeled samples in small-sample hyperspectral image classification, a small-sample learning mechanism is used to improve the performance of the network in small-sample hyperspectral image classification. The feature distribution of hyperspectral images in different domains is obtained through image feature aggregation, and the domain alignment method is used to reduce the feature differences between different domains, thereby achieving cross-domain small-sample classification of hyperspectral images.

[0034] For details, see Figure 1 , Figure 2 , shows a flowchart and module diagram of a cross-domain small sample learning hyperspectral image classification method according to a specific embodiment of the present invention.

[0035] Data classification step S110:

[0036] Input hyperspectral data and corresponding labels. The input data is divided into source domain data (Source Data, SD) and target domain data (Target Data, TD).

[0037] In the subsequent steps, relevant processing is performed on the source domain data and the target domain data respectively.

[0038] Hyperspectral image preprocessing step S120:

[0039] The principal component analysis method is used to reduce the dimensionality of the input data set to obtain the feature map after dimensionality reduction, remove redundant channels in the data, and improve the classification performance of the network. Then the neighborhood pixel extraction method is used to further divide the feature map after dimensionality reduction, cut and divide the data set, and select various types of samples from it as support sets and query sets for subsequent small sample learning.

[0040] This step preprocesses the source and target domain data, maps them to the same dimension, and extracts neighborhood pixels to reduce the data distribution differences between different domains.

[0041] Among them, dimensionality reduction through principal component analysis can effectively reduce the dimension of data, remove redundant information, highlight the main features, solve the problem of collinearity, and improve data visualization capabilities, thereby improving computing efficiency, reducing the impact of noise, and helping to improve the performance and generalization ability of classification and regression models.

[0042] Neighborhood pixel extraction enhances the spatial information of an image by capturing the spatial relationship around pixels, which helps improve the performance of tasks such as classification and target detection. This is because it can retain the contextual information of objects and improve the ability to understand complex scenes. Through neighborhood pixel extraction, the relationship between pixels and their surroundings can be more comprehensively considered, making image processing more robust and accurate.

[0043] Suppose a hyperspectral image H has a length of l, a width of w, and a depth of d, that is, H∈R(l,w,d), and all its label data is represented by T, then T∈R(l,w). Considering the high-dimensional characteristics of hyperspectral image data, there is a significant information correlation between adjacent dimensions in the original data, resulting in high data redundancy. In order to deal with this problem, a suitable dimensionality reduction method is needed.

[0044] Specifically, the principal component analysis selects the PCA method to achieve data dimensionality reduction, reducing the number of channels to k dimensions while keeping the spatial dimension unchanged, specifically including:

[0045] First, the hyperspectral image data is organized into a matrix X, where each row represents a sample (pixel) and each column represents a band (feature). Assuming there are n samples and m bands, we get an n×m data matrix X.

[0046] In order to ensure that each band is equally important, the data needs to be standardized, usually by making each band zero-mean (mean is 0) and unit-variance (variance is 1). This step ensures that the contribution of each band is equal. Refer to formula (1) to obtain the centered data matrix:

[0047]

[0048] Among them, X centered is the centralized data matrix, X is the input hyperspectral data matrix, is the mean of the input hyperspectral data matrix.

[0049] Secondly, calculate the covariance matrix Cov of the data matrix X. The covariance matrix Cov describes the correlation between the various bands in the data. The calculation formula of the covariance matrix is ​​as follows:

[0050]

[0051] Among them, X T represents the transposed matrix, and N represents the number of samples.

[0052] Next, the covariance matrix Cov is decomposed to obtain the eigenvalues ​​λ1,λ2,…,λ M and the corresponding eigenvectors v1,v2,…,v M , arranged in descending order of eigenvalue. The eigenvalue represents the variance in the data, and the eigenvector represents the projection of the data in different directions. The number of principal components to be retained, that is, the number of dimensions after dimensionality reduction, is selected by analyzing the size of the eigenvalue. The number of principal components to be retained can be a subset of the original data dimensions, for example, the first k principal components can be retained.

[0053] Finally, project the data onto the selected principal components to obtain new data after dimensionality reduction. For each sample, multiply it with the selected principal component to obtain the sample after dimensionality reduction. Assuming that the first k principal components are selected, the data matrix after dimensionality reduction is Y, then: Y = XV k , here, V k is a matrix containing the first k eigenvectors.

[0054] The domain extraction method is to transform the original pixel point as a sample unit into a spectral curve block as a sample unit for training. Its function is to reconstruct the unit sample to create a suitable sample set for the deep learning network, avoiding the problem that PCA dimensionality reduction has already reduced the spectral dimension of the hyperspectral image and reduced the redundant information of the hyperspectral image, but a single spectral curve cannot effectively utilize the rich spatial domain information of the hyperspectral image.

[0055] Specifically, for a given central pixel a, the pixel and its neighboring pixels of size L×L are extracted to form a three-dimensional cuboid data A, that is, A∈R(L,L,D), where L represents the length and width of the cuboid, and D represents the depth. In this way, a data set P consisting of sample data is obtained, where each sample consists of a three-dimensional cuboid data and a corresponding label value, expressed as P=(X n ,Y n ), X n is the three-dimensional cuboid data of the nth sample, Y n Represents the label value of the nth sample.

[0056] After the data preprocessing is completed, the source domain and target domain data are divided. The small sample learning task T of each domain includes a support set S and a query set Q, where S is a labeled set and Q is an unlabeled set used to evaluate the performance of the learned classifier. The support set S consists of C categories, each category has K samples, so the small sample learning task is called C-way K-shot, usually called an episode. The query set Q contains samples of the same category as the support set S in the meta-task.

[0057] Deep three-dimensional residual network spatial spectrum feature extraction step S130:

[0058] This step includes feature mapping and deep three-dimensional residual network spatial spectrum feature extraction, wherein the feature mapping includes a source domain mapping layer M s and the target domain mapping layer M t , through two-dimensional CNN, the data of the source domain and the target domain are mapped from the original spectral dimension to a common mapping dimension dmap, so that the input dimensions of the source domain (128 bands of the Chikusei dataset) and the target domain (such as the 103 bands of the Pavia University dataset) are equal. In this way, the difference between the source domain data (Source Data, SD) and the target domain data (Target Data, TD) is reduced; the deep three-dimensional residual network spatial spectrum feature extraction includes two low-resolution three-dimensional spatial spectrum feature processing modules (Low Resolution 3D Space Spectrum Feature Processing Module, LoR-3DSSFPM), two three-dimensional residual blocks (Three-dimensional residualblock, Res-3D), two three-dimensional pooling layers (Three-Dimensional Max Pooling, 3DMaxPool) and a three-dimensional convolutional layer. For the low resolution of the neighborhood pixel stereo block, the spatial and spectral features are efficiently extracted through the spatial spectrum attention mechanism, further improving the generalization ability of the algorithm.

[0059] For details, see Figure 3 , showing the feature mapping layer structure diagram.

[0060] Assume that the input hyperspectral image is H∈R W×H×B , where W×H represents the spatial dimension, B represents the spectral dimension, and H′∈R W×H×100 Represents the data set after mapping, and the mapping transformation operation is T∈R B×100 , see formula (3) for details:

[0061] H′=H×T (3)

[0062] Where H′ is the data after mapping, each mapping layer contains B×100 learnable parameters, for the source domain mapping layer M s There are 128×100 parameters for the target domain mapping layer M t There are 103×100 parameters

[0063] After mapping, the data sets of the two domains have the same dimension. Considering that the hyperspectral image data is a three-dimensional data, the use of three-dimensional data processing technology is more conducive to image processing. Therefore, this step has a deep three-dimensional residual network spatial spectrum feature extraction step.

[0064] For details, see Figure 4 The order of the deep three-dimensional residual network spatial spectral features is: low-resolution three-dimensional spatial spectral feature processing, three-dimensional residual block, three-dimensional pooling layer, low-resolution three-dimensional spatial spectral feature processing, three-dimensional residual block, three-dimensional pooling layer, and then the extracted features are obtained by deformation using three-dimensional convolution.

[0065] The transformation is the Rshape operation in deep learning, for example, transforming a 3*4*5 matrix into a 2*6*5 matrix

[0066] See also Figure 5 The low-resolution three-dimensional spatial spectrum feature processing LoR-3DSSFPM includes: space-depth feature conversion and a non-cross-row convolution.

[0067] The spatial-depth feature conversion extracts a series of sub-feature maps from the intermediate feature map A of size L×L×C1. The specific operation is shown in formula (4):

[0068] f i,j,d =A[i:R:2,j:R:2,d:R:2] (4)

[0069] Where R is the scale factor, which is set to 2 in the present invention, i, j, d are the starting positions of the sub-graph in three dimensions when it is generated, and sub-graph f i,j,d, which is composed of X divided by the scale factor. Therefore, each time a sub-feature map is generated, the length, width, and number of channels of the sample will become half of the original, and 8 sub-feature maps f will also be obtained. 0,0,0 ,f 0,1,0 ,f 0,0,1 ,f 1,0,0 ,f 1,1,0 ,f 1,0,1 ,f 0,1,1 ,f 1,1,1 ,Each sub-image is obtained by downsampling the original feature map by twice in three dimensions.

[0070] At the same time, these sub-feature maps are connected along the channel dimension to obtain a new feature map X', the spatial dimension of the new feature map becomes half of the original, and the channel dimension becomes eight times the original. In other words, the spatial-depth feature conversion converts the feature map A(L,L,C1) into an intermediate feature map A'(L / 2,L / 2,8*C1).

[0071] Finally, after the spatial-depth feature conversion, a non-stride convolution is added. The non-stride convolution is a non-stride 3D convolution layer. The number of channels output by the 3D convolution is C2<8C1, and A'(L / 2,L / 2,8C1) is further converted to A" (L / 2,L / 2,C2). Using non-stride convolution can retain all discriminative feature information as much as possible and avoid information loss. For example, using a 3×3 convolution kernel with a stride of 3, the feature map will be "compressed", but each pixel will be sampled only once; if the stride is 2, there will be asymmetric sampling, that is, the even rows and columns will be sampled different times than the odd rows and columns. In general, a stride greater than 1 will result in information loss, although on the surface, it also appears to convert the feature map A(L,L,C1) to A" (L / 2,L / 2,C2).

[0072] The data of the 3D spatial spectrum feature processing module of the low-resolution hyperspectral image is input into the 3D residual block, which consists of three 3D convolutions and one RELU. The feature data extracted by the three 3D convolutions are added to the original data through the residual connection and activated by RELU to finally obtain the extracted features. The extracted features are then passed through the 3D pooling layer to prevent fitting.

[0073] After another three-dimensional spatial spectrum feature processing of the low-resolution hyperspectral image, three-dimensional residual feature extraction and three-dimensional maximum pooling, the hyperspectral image has been fully extracted. After that, the extracted features have been obtained through deformation and three-dimensional convolution.

[0074] This step can learn the importance weights of hyperspectral image data in different bands and different spaces, so that the network can focus more on important features or areas, thereby reducing network complexity and computational burden; image feature reconstruction and enhancement technology is realized through downsampling and splicing operations, improving the image resolution and feature expression ability. This helps the model better understand the subtle changes and detailed information in the hyperspectral image.

[0075] This step retains all the information in the channel dimension, and extracts spatial-spectral features through a three-dimensional deep residual network, which can capture the differences between different bands and emphasize the important information in the data, thereby helping to identify and understand targets, objects or other key features in the image; it can learn the importance weights of hyperspectral image data in different bands and different spaces, so that the network can focus more on important features or areas, thereby reducing network complexity and computational burden, helping to optimize subsequent data analysis and processing steps, and improving the performance of hyperspectral images in classification, detection or monitoring tasks.

[0076] Small sample learning step S140:

[0077] After the hyperspectral data has gone through the spatial-spectral feature extraction step of the deep three-dimensional residual network, the spatial-spectral features are fully mined. Next, the small sample learning step is used to achieve accurate classification of cross-domain small sample hyperspectral images.

[0078] This step includes: performing small sample learning on the source domain and the target domain respectively. In each domain, the support set and the query set are input into femb(Ms(·) to obtain embedded features. In each episode, the small sample learning task is performed by calculating the Euclidean distance between the embedded features of the query set and the prototype of each class (i.e., the mean of the embedded features of the support set), while minimizing the prediction loss of the query set. Finally, the Euclidean distance is calculated by the softmax function to predict the probability of the query sample, and the training is performed by the cross entropy loss function.

[0079] Specifically, in the source domain, the support set and the query set are input into femb(Ms(·)) to obtain the embedded features Zs. In each episode, the small sample learning task is performed by calculating the Euclidean distance between the embedded features of the query set and the prototype of each class (i.e., the mean of the embedded features of the support set), while minimizing the prediction loss of the query set. Finally, the Euclidean distance is calculated by the softmax function to predict the query sample x sm The probability of , the process is shown in formula (4).

[0080]

[0081] Where ED(·) represents the Euclidean distance function, is the query set embedding feature, is the mean of the embedding features of the c-th support set, also called the c-th prototype (c∈C s ), the FSL loss of the query sample is calculated by the cross entropy loss, as shown in formula (5)

[0082]

[0083] The calculation process is the same as the former, but it should be noted that Based on the use The dataset is obtained by constructing a meta-task after data enhancement, where: Represents the target domain dataset.

[0084] The computational method of small sample learning in the target domain is the same as that in the source domain, but the target set The dataset is obtained by constructing a meta-task after data augmentation.

[0085] This step uses technical means to leverage existing knowledge to help the model better generalize to new, relatively less labeled scenarios. This helps improve model performance in hyperspectral image processing, enhances adaptability to new data, and reduces reliance on large amounts of labeled data.

[0086] Domain alignment step S150:

[0087] The classified data is converted into graph data, in which the edges contain the spatial distribution information of the data, and the nodes contain the spectral dimension information of the data. Specifically for the source domain and the target domain, the intra-domain feature extraction IFE-BLOCK is used to extract the respective features of the source domain and the target domain respectively. The WD distance (Wasserstein distance) at the feature level and the distribution level and the GWD distance (Gromov-Wasserstein distance, a generalization of the Wasserstein distance) for edge matching are calculated in each update, and the GOT loss is minimized to reduce the differences between domains. Then, the inter-domain similarity measurement module (CSM-BLOCK) is used to capture the similarity and distribution similarity of cross-domain features, and attention aggregation is used to make them work together to further reduce the cross-domain differences.

[0088] The combination of the two modules enables the model to effectively reduce the distribution difference between the source domain data and the target domain data after mapping, so that the model has good generalization ability in the target domain.

[0089] For details, see Figure 6 , the intra-domain feature extraction IFE-BLOCK includes:

[0090] Generate a node feature graph based on the dataset and a spatial distribution feature map Then the two images are aggregated and updated. First, the feature embedding of all samples is extracted through the convolution backbone. The feature embedding is used to calculate the instance similarity relationship. Then the instance similarity relation is passed to build the distribution map In the distribution graph, the node feature V l d according to The position order in is summarized to initialize, and the edge features Represents the node feature V l p The distribution similarity between is passed to To build node V l p The generation and update of IFE-BLOCK are briefly introduced as follows: Where l represents the lth generation.

[0091] WD distance and GWD distance measure the distance between nodes and edges of two graphs. The fusion of these two distances effectively considers the information of nodes and edges to achieve better graph alignment. In order to achieve the best integration of WD and GWD, considering the transformation matrix T shared by WD and GWD, the GOT distance is defined as shown in formula (6):

[0092]

[0093] Where T ij and is the transformation matrix of two pairs of nodes, is the cross-domain cost matrix obtained by cosine distance, To evaluate two pairs of nodes The cost function of the graph structure similarity between them. Finally, the GOT loss definition of cross-domain graph alignment is calculated using formula (7):

[0094]

[0095] Where f represents the feature map and d represents the distribution map.

[0096] The main purpose of intra-domain feature extraction IFE-BLOCK is to characterize and summarize the non-local relationships between samples in the source domain and the target domain, as well as to align feature and distribution-level graphs. However, reducing the impact of domain shift by minimizing the GOT loss alone is not enough in the absence of cross-domain information interaction. The present invention further calculates the inter-domain similarity measurement module (CSM-BLOCK) to generate feature-level attention and distribution-level high-order attention between the source domain and the target domain, respectively capturing cross-domain feature similarity and distribution similarity, and making them work together through attention aggregation to reduce cross-domain differences together with the IFE-BLOCK block.

[0097] Specifically, the calculation of the inter-domain similarity measurement module (CSM-BLOCK) specifically includes: when calculating in the source domain,

[0098] First, the feature-level attention is calculated. In generating the feature-level attention of the source domain, the node feature map generated by the source domain is Node feature graph with target domain Perform linear self-representation and calculate the feature-level attention of the source domain relative to the target domain as shown in formula (8):

[0099]

[0100] Where C(·) represents a one-dimensional convolution. Then, the distribution feature maps of the extracted source domain and target domain are and To calculate the distribution-level attention, an MLP consisting of two Conv1d-BN-ReLU blocks is used to increase the dimension of the distribution feature map from C×K to 128, and a one-dimensional convolution is used for self-representation. The calculation process is shown in formula (9):

[0101]

[0102] Finally, the feature-level attention is used as the value, and the distribution-level high-order attention map is applied to obtain the final attention feature. The calculation process is shown in formula (10):

[0103]

[0104] Since the node feature map and the distribution feature map are mutually generated in the IDE-block, the distribution-level high-order attention map works together with the above feature-level attention map to capture the characteristics and distribution similarity of the source domain relative to the target domain. The attention feature generated by the (l-1) layer is m s(l-1) , can be combined with the attention aggregation features of the (l-1) layer Propagate to obtain the attention aggregation features of the lth layer The calculation process is shown in formula (11):

[0105]

[0106] The CSM-BLOCK similarity measurement block corresponding to the target domain is the same as the above process.

[0107] Furthermore, during attention aggregation, the linear MMD distance is used to align the outputs of the source domain and the target domain. The specific process is shown in formula (12):

[0108]

[0109] Where N = C × K + T, the average difference is calculated in the middle of Hilbert's space.

[0110] Network training step S160:

[0111] In summary, according to the above steps, the total loss function is obtained according to formula (5), formula (7) and formula (12), as shown in formula (13):

[0112]

[0113] Among them, λ1 and λ2 represent the hyperparameters of the corresponding parameters.

[0114] After training a certain number of times, minimize the total loss function L total , wherein the certain number of times may be 1000 times as an example.

[0115] The present invention further discloses a storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed by a processor, execute the above-mentioned cross-domain small sample learning hyperspectral image classification method.

[0116] Example:

[0117] In this embodiment, a server host with 32GB RAM and RTX2080Ti GPU is used to train the cross-domain small sample learning hyperspectral image classification method of the present invention. The programming language used is Python version 3.7.

[0118] The present invention conducts experiments on four hyperspectral datasets, namely Chikusei, Indian Pines, Pavia University and Salinas Scene, wherein Chikusei is used as the source domain dataset, and the other three datasets are used as target domain datasets for three experiments. In the field of hyperspectral image classification, the size of the input sample needs to be adjusted for different datasets. The size of the input sample of the Chikusei dataset is set to 9×9×128, the size of the input sample of the Indian Pines dataset is set to 9×9×200, the size of the input sample of the Pavia University dataset is set to 9×9×103, and the input sample of the Salinas Scene dataset is set to 9×9×204. The optimizer used in this experiment is the Adam optimizer, and its learning rate is set to 0.001. All convolutional layers and linear layers are initialized using Xavier normalization, and the batch normalization layer uses the normal distribution N(1,0.02). The default value of the 2-norm regularization for feature extraction is 0, and the weight decay of other modules is set to 1e-4. The number of training iterations is set to 10,000 to prevent too few training epochs from fully learning the prototypes of each class.

[0119] In order to evaluate the performance of the proposed 3DGia-CFSL in the case of SD and TD from different sensors and containing different categories, related algorithms were used for comparison, including SVM, 3DCNN, PToP CNN, DAAN, DSAN, DPGN, DFSL, RN-FSC and DCFSL. The training sample settings are as follows: SVM, 3DCNN and PToP CNN are regarded as supervised methods, requiring training and test samples with the same dimension and the same category, so only five labeled samples of each category in TD are selected as training sets, and the rest are used as test sets. For the unsupervised domain adaptation methods DAAN and DSAN, the projection layer consistent with 3DGia-CFSL is added, and all labeled SD data and unlabeled TD data are used for training. For DPGN, DFSL, RN-FSC and DCFSL developed based on FSL, the selection of training samples is the same as 3DGia-CFSL. Specifically, DPGN only uses SD to construct meta-tasks for training, and does not involve TD, DFSL and training samples of the same dimension and the same category, so only five labeled samples of each category in TD are selected as training sets, and the rest are used as test sets. For the unsupervised domain adaptation methods DAAN and DSAN, the projection layer consistent with 3DGia-CFSL is added, and all labeled SD data and unlabeled TD data are used for training. The training sample selection of DPGN, DFSL, RN-FSC and DCFSL developed based on FSL is the same as that of 3DGia-CFSL. Specifically, DPGN only uses SD to construct meta-tasks for training, and does not involve TD, DFSL, and training samples of the same dimension and the same category. Therefore, only five labeled samples of each category in TD are selected as the training set, and the rest are used as the test set.

[0120] In order to comprehensively evaluate the performance of the algorithm, this embodiment randomly selects five labeled samples from each category as the supervision samples of the target domain, and performs ten experiments to calculate the average accuracy and standard deviation. The accuracy evaluation uses three indicators: overall accuracy OA, average accuracy AA and Kappa coefficient.

[0121] In order to quantitatively analyze the classification effect of the algorithm, experiments were conducted on three different target datasets. Tables 1, 2, and 3 show the classification results of different algorithms on the Pavia University, Indian Pines, and Salinas Scene datasets.

[0122] Table 1: Classification results of 9 cross-domain deep learning algorithms and 3DGia-CFSL algorithm on Pavia University dataset with 5 labeled samples per class

[0123]

[0124]

[0125] For all classification methods, 5 labeled samples are randomly selected from each category as supervised samples in TD, and the experimental average accuracy and standard deviation of 10 groups of different random seeds for each algorithm are calculated. Table 2 is an analysis of the statistical table of these three classification results:

[0126] Table 2 Classification results of 9 cross-domain deep learning algorithms and 3DGia-CFSL algorithm on Indian Pines dataset with 5 labeled samples per class

[0127]

[0128]

[0129] Table 3 Classification results of 9 cross-domain deep learning algorithms and 3DGia-CFSL algorithm on the Salinas dataset with 5 labeled samples per class

[0130]

[0131] It can be seen that:

[0132] 1) Compared with SVM, 3DCNN and PToPCNN trained with only a small number of target domain TD samples, DAAN, DSAN, DFSL, RN-FSC, DCFSL and 3DGia-CFSL, which utilize source domain SD knowledge and design domain alignment modules, improve OA by 5%-10%. This shows that when there are fewer samples in TD, transferring SD knowledge to TD helps improve the classification accuracy of the model on TD.

[0133] 2) The performance of DFSL, RN-FSC, DCFSL and 3DGia-CFSL methods based on FSL is 4%-8% higher than that of unsupervised data processing methods such as DAAN and DSAN. This is because the learning method of constructing meta-tasks for scenario training in FSL has stronger generalization ability and learns more complete SD knowledge. This shows that the FSL method trained with meta-learning ideas can better solve the problem of fewer labeled samples.

[0134] 3) Due to the domain alignment module design, 3DGia-CFSL and DCFSL outperform DPGN, DFSL, and RN-FSC, which only focus on FSL. In addition, an IFE-BLOCK block for spatial feature and node feature alignment is developed in 3DGia-CFSL, assisted by a CSM-BLOCK block that can measure inter-domain similarity to reduce cross-domain differences. This domain alignment method outperforms the conditional adversarial strategy in DCFSL, and overall the global accuracy OA of the experiments conducted on these three datasets is improved by about 2%.

[0135] The experimental results on three target data show that the algorithm adopted by the present invention can achieve better results than the previous algorithms.

[0136] In summary, the present invention provides a new theory and tool to solve the problem of scarcity of labeled samples in hyperspectral images, and provides a new idea and method for improving and optimizing hyperspectral image classification tasks, which has the following advantages:

[0137] 1. Through three-dimensional residual network and neighborhood pixel extraction, the challenges of insufficient feature extraction, low resolution and inter-domain data differences in small sample hyperspectral image classification are effectively addressed.

[0138] 2. The small sample learning step and domain alignment step are introduced, and the classification accuracy of the algorithm is improved when there are limited labeled samples through Euclidean distance measurement and graph information aggregation.

[0139] 3. The domain alignment step is introduced to significantly improve the classification accuracy of hyperspectral images in cross-domain situations and effectively address the differences in data distribution between different domains.

[0140] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. It cannot be determined that the specific embodiments of the present invention are limited to this. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention determined by the submitted claims.

Claims

1. A cross-domain small sample learning hyperspectral image classification method, characterized in that: The steps include: Data classification step S110: Input hyperspectral data and corresponding labels. The input data is divided into source domain data and target domain data. Hyperspectral image preprocessing step S120: Use the principal component analysis method to reduce the dimension of the input data set to obtain the feature map after dimensionality reduction, and then use the neighborhood pixel extraction method to further divide the feature map after dimensionality reduction, cut and divide the data set, and select various types of samples from it as support sets and query sets for subsequent small sample learning; Deep three-dimensional residual network spatial spectrum feature extraction step S130: This step includes feature mapping and deep three-dimensional residual network spatial spectrum feature extraction, wherein the feature mapping includes a source domain mapping layer M s and the target domain mapping layer M t , through the two-dimensional CNN, the data of the source domain and the target domain are mapped from the original spectral dimension to a common mapping dimension dmap, so that the input dimension of the source domain is equal. The deep three-dimensional residual network spatial spectrum feature extraction includes two low-resolution three-dimensional spatial spectrum feature processing modules, two three-dimensional residual blocks, two three-dimensional pooling layers and a three-dimensional convolution layer, through the spatial spectrum attention mechanism; Small sample learning step S140: Small sample learning is performed simultaneously on the source domain and the target domain. In each domain, the support set and the query set are input into femb(Ms(·)) to obtain embedded features. In each episode, the small sample learning task is performed by calculating the Euclidean distance between the embedded features of the query set and the prototype of each class, while minimizing the prediction loss of the query set. Finally, the Euclidean distance is calculated by the softmax function to predict the probability of the query sample, and the cross entropy loss function is used for training. Domain alignment step S150: Convert the classified data into graph data, where the edges contain the spatial distribution information of the data and the nodes contain the spectral dimension information of the data. Specifically for the source domain and the target domain, use the intra-domain feature extraction IFE-BLOCK to extract the respective features of the source domain and the target domain respectively. In each update, calculate the WD distance at the feature level and the distribution level and the GWD distance for edge matching, and minimize the GOT loss to reduce the difference between domains. Then, use the inter-domain similarity measurement module to capture the similarity and distribution similarity of cross-domain features, and use attention aggregation to make them work together to reduce cross-domain differences. Network training step S160: In summary, the above steps obtain the total loss function. After training a certain number of times, the total loss function is minimized to train the method.

2. The hyperspectral image classification method according to claim 1, characterized in that: In the hyperspectral image preprocessing step S120, The principal component analysis method selects the PCA method, including: First, the hyperspectral image data is organized into a matrix X, where each row represents a sample and each column represents a band. Assuming there are n samples and m bands, an n×m data matrix X is obtained; Each band is zero-mean and unit-varianced, see formula (1), and the centered data matrix is ​​obtained: Among them, X centered is the centralized data matrix, X is the input hyperspectral data matrix, is the mean of the input hyperspectral data matrix; Secondly, calculate the covariance matrix Cov of the data matrix X. The covariance matrix Cov describes the correlation between the various bands in the data. The calculation formula of the covariance matrix is ​​as shown in formula (2): Among them, X T represents the transposed matrix, N represents the number of samples, Next, the covariance matrix Cov is decomposed to obtain the eigenvalues ​​λ1,λ2,…,λ M and the corresponding eigenvectors v1,v2,…,v M , arrange them in descending order according to the eigenvalues, and select the first k principal components to be retained. Finally, project the data onto the selected principal components to obtain new reduced-dimensional data. Assuming that the first k principal components are selected, the reduced-dimensional data matrix is ​​Y, then: Y = XV k , here, V k is a matrix containing the first k eigenvectors; The field extraction method is specifically as follows: For a given central pixel a, the pixel and its neighboring pixels of size L×L are extracted to form a three-dimensional rectangular parallelepiped data A, that is, A∈R(L,L,D), where L represents the length and width of the rectangular parallelepiped, and D represents the depth. A data set P consisting of sample data is obtained, where each sample consists of a three-dimensional rectangular parallelepiped data and a corresponding label value, expressed as P=(X n ,Y n ), X n is the three-dimensional cuboid data of the nth sample, Y n Represents the label value of the nth sample.

3. The hyperspectral image classification method according to claim 2, characterized in that: Furthermore, in the hyperspectral image preprocessing step S120, After completing the data preprocessing, the source domain and target domain data are divided. The small sample learning task T of each domain includes a support set S and a query set Q, where S is a labeled set and Q is an unlabeled set. The support set S consists of C categories, each category has K samples. The small sample learning task is called C-way K-shot or an episode. The query set Q contains samples of the same category as the support set S in the meta-task.

4. The hyperspectral image classification method according to claim 1, characterized in that: In the deep three-dimensional residual network spatial spectrum feature extraction step S130: The feature map is: Assume that the input hyperspectral image is H∈R W×H×B , where W×H represents the spatial dimension, B represents the spectral dimension, and H′∈R W ×H×100 Represents the data set after mapping, and the mapping transformation operation is T∈R B×100 , see formula (3) for details: H′=H×T (3) Where H′ is the data after mapping, each mapping layer contains B×100 learnable parameters, for the source domain mapping layer M s There are 128×100 parameters for the target domain mapping layer M t There are 103×100 parameters.

5. The hyperspectral image classification method according to claim 4, characterized in that: In the deep three-dimensional residual network spatial spectrum feature extraction step S130 The deep three-dimensional residual network spatial spectrum feature sequence is: low-resolution three-dimensional spatial spectrum feature processing, three-dimensional residual block, three-dimensional pooling layer, low-resolution three-dimensional spatial spectrum feature processing, three-dimensional residual block, three-dimensional pooling layer, and then three-dimensional convolution is used to obtain the extracted features through deformation; The low-resolution three-dimensional spatial spectrum feature processing includes: space-depth feature conversion and a non-cross-row convolution, The spatial-depth feature conversion extracts a series of sub-feature maps from the intermediate feature map A of size L×L×C1. The specific operation is shown in formula (4): f i,j,d =A[i:R:2,j:R:2,d:R:2] (4) Among them, R is the scale factor, i, j, d are the starting positions of the sub-graph in the three dimensions when it is generated, and the sub-graph f i,j,d , which is composed of X divided by the scale factor. Therefore, each time a sub-feature map is generated, the length, width, and number of channels of the sample will become half of the original, and 8 sub-feature maps f will also be obtained. 0,0,0 ,f 0,1,0 ,f 0,0,1 ,f 1,0,0 ,f 1,1,0 ,f 1,0,1 ,f 0,1,1 ,f 1,1,1 , At the same time, these sub-feature maps are connected along the channel dimension to obtain a new feature map X′, the spatial dimension of the new feature map becomes half of the original one, and the channel dimension becomes eight times the original one, that is, the spatial-depth feature conversion converts the feature map A(L,L,C1) into an intermediate feature map A'(L / 2,L / 2,8*c1); Finally, after the spatial-depth feature conversion, a non-stride convolution is added. The non-stride convolution is a three-dimensional convolution layer with no stride. The number of channels output by the three-dimensional convolution is c2<8C1, and A'(L / 2,L / 2,8C1) is further converted to A”(L / 2,L / 2,C2). The data of the three-dimensional spatial spectrum feature processing module of the low-resolution hyperspectral image is input into the three-dimensional residual block, which is composed of three three-dimensional convolutions and one RELU. The feature data extracted by the three three-dimensional convolutions are added to the original data through the residual connection and activated by RELU, and finally the extracted features are obtained. The extracted features are then passed through the three-dimensional pooling layer; After another three-dimensional spatial spectrum feature processing of the low-resolution hyperspectral image, three-dimensional residual feature extraction and three-dimensional maximum pooling, the hyperspectral image has been fully extracted. After that, the extracted features have been obtained through deformation and three-dimensional convolution.

6. The hyperspectral image classification method according to claim 1, characterized in that: In the small sample learning step S140, In the source domain, the support set and the query set are input into femb(Ms(·)) to obtain the embedded features Zs. In each episode, the small sample learning task is performed by calculating the Euclidean distance between the embedded features of the query set and the prototype of each class, while minimizing the prediction loss of the query set. Finally, the Euclidean distance is calculated by the softmax function to predict the query sample x. sm The probability of , the process is shown in formula (4). Where ED(·) represents the Euclidean distance function, is the query set embedding feature, is the mean of the embedding features of the c-th support set, also called the c-th prototype (c∈C s ), the FSL loss of the query sample is calculated by the cross entropy loss, as shown in formula (5) The calculation process of is the same as the former, but Based on the use The dataset is obtained by constructing a meta-task after data enhancement, where: represents the target domain dataset, The computational method of small sample learning in the target domain is the same as that in the source domain, but the target set The dataset is obtained by constructing a meta-task after data augmentation.

7. The hyperspectral image classification method according to claim 1, characterized in that: In the domain alignment step S150, The intra-domain feature extraction IFE-BLOCK includes: Generate a node feature graph based on the dataset and a spatial distribution feature map Then the two images are aggregated and updated. First, the feature embedding of all samples is extracted through the convolution backbone. The feature embedding is used to calculate the instance similarity relationship. Then the instance similarity relation is passed to build the distribution map In the distribution graph, the node feature V l d according to The position order in is summarized to initialize, and the edge features Represents the node feature V l p The distribution similarity between is passed to To build node V l p A more distinguishable representation of , and the above process is repeated generation by generation; In order to achieve the best integration of WD and GWD, considering the transformation matrix T shared by WD and GWD, the GOT distance is defined as shown in formula (6): Where T ij and is the transformation matrix of two pairs of nodes, is the cross-domain cost matrix obtained by cosine distance, To evaluate two pairs of nodes The cost function of the graph structure similarity between them. Finally, the GOT loss definition of cross-domain graph alignment is calculated using formula (7): Where f represents the feature map and d represents the distribution map.

8. The hyperspectral image classification method according to claim 7, characterized in that: The inter-domain similarity measurement module calculation specifically includes: When calculating in the source domain, First, the feature-level attention is calculated. In generating the feature-level attention of the source domain, the node feature map generated by the source domain is Node feature graph with target domain Perform linear self-representation and calculate the feature-level attention of the source domain relative to the target domain as shown in formula (8): Where C(·) represents a one-dimensional convolution. Then, the distribution feature maps of the extracted source domain and target domain are and To calculate the distribution-level attention, an MLP consisting of two Conv1d-BN-ReLU blocks is used to increase the dimension of the distribution feature map from C×K to 128, and a one-dimensional convolution is used for self-representation. The calculation process is shown in formula (9): Finally, the feature-level attention is used as the value, and the distribution-level high-order attention map is applied to obtain the final attention feature. The calculation process is shown in formula (10): Since the node feature map and the distribution feature map are generated mutually in the IDE-block, the distribution-level high-order attention map works together with the above feature-level attention map to capture the characteristics and distribution similarity of the source domain relative to the target domain. The attention feature generated by the (l-1)th layer is m s(l-1) , can be combined with the attention aggregation features of the (l-1) layer Propagate to obtain the attention aggregation features of the lth layer The calculation process is shown in formula (11): The CSM-BLOCK similarity measurement block corresponding to the target domain is the same as the above process; During attention aggregation, the linear MMD distance is used to align the outputs of the source domain and the target domain. The specific process is shown in formula (12): Where N = C × K + T, the average difference is calculated in the middle of Hilbert's space.

9. The hyperspectral image classification method according to claim 1, characterized in that: In the network training step S160: According to formula (5), formula (7) and formula (12), the total loss function is obtained as shown in formula (13): After training a certain number of times, minimize the total loss function L total , where the certain number of times is 1000 times.

10. A storage medium for storing computer executable instructions, characterized in that: When the computer executable instructions are executed by the processor, the cross-domain small sample learning hyperspectral image classification method described in any one of claims 1 to 9 is performed.

Images