Open set domain adaptation remote sensing image small sample classification method based on multi-graph convolutional network

Abstract

A small sample classification method for remote sensing images based on multi-graph convolutional network open set domain adaptation includes: S100: reading M samples from the task data set and the auxiliary data set respectively; S200: extracting the features of all samples using the feature extraction network Conv4; S300: calculating the adjacency matrix A based on the extracted features; S400: updating the sample features using the obtained adjacency matrix A and multi-graph convolution operation; S500: predicting the labels of the unlabeled samples of the task data set using the updated features; S600: training the network by back propagation based on the predicted sample labels; S700: classifying the unlabeled samples of the task data set using the trained network to obtain the classification results. The classification accuracy is improved by reducing the inter-domain differences of the common class parts, the similarity of the graph modeling samples in the feature space is used, and the aggregation of the same category samples is enhanced through the graph convolution operation, so as to further improve the classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to an open set domain adaptive remote sensing image small sample classification method based on a multi-graph convolutional network. Background Art

[0002] Remote sensing scene classification is an important method in remote sensing scene understanding. Its purpose is to classify remote sensing images into different categories according to their content. Remote sensing scene classification has a wide range of uses in urban planning, land prediction, environmental protection, etc. Deep learning has achieved great success in the field of remote sensing image classification. Deep learning generally requires a large number of labeled samples for training, but in many cases it is difficult to obtain enough labeled samples. In this regard, the small sample learning method is applied to remote sensing image scene classification to solve the remote sensing image scene classification problem when there are only a few labeled samples for each type of image.

[0003] Metric learning is one of the mainstream methods to solve the problem of small sample classification. The main idea of ​​metric learning is to quantify the similarity between any pairs of samples by training and learning a similarity metric. The method is to learn a metric network with certain generalization performance through auxiliary data sets. For this metric network, sample pairs of the same category obtain higher similarity, and sample pairs of different categories obtain lower similarity. Then, when there are only a few labeled samples in each category, the category of the unlabeled samples in the task dataset can be predicted based on the similarity between the unlabeled samples and the labeled samples. Metric learning was first introduced into the field of natural image processing to solve the problem of small sample classification of natural images. In recent years, small sample classification methods have been introduced into the field of remote sensing image processing to solve the problem of small sample classification of remote sensing images.

[0004] Definition of the small sample classification problem: Given a task dataset D that is specific to task T and contains a small amount of supervised information T and an auxiliary dataset D that is not related to task T A , here D A There are a large number of labeled samples in D T There are only a few labeled samples in D A and D T Usually the sources are different. Small sample learning is to use D T There is little supervision information and D A In this case, “not relevant to task T” means D T The category set Y in T and D A The category set Y in AThe intersection is empty. However, in reality, the task data and auxiliary data labels may overlap, that is, Y A ∩Y T ≠φ and Y A ∩Y T ≠Y T , the small sample classification task in this case is called open set small sample classification, Y A ∩Y T =φ is called closed set small sample classification. The parts with the same categories in the two datasets are called common classes, and the rest are private classes. Due to the differences in the distribution of image features in different datasets, for common class data, although they have the same labels, the model trained on the auxiliary dataset often cannot obtain high accuracy on the task dataset. Domain adaptation is a method to reduce the difference in feature distribution between different datasets. Its principle is to quantify the feature differences between datasets into domain loss functions, and to reduce the feature distribution differences by reducing the domain loss function during model training. If the domain adaptation method can be used to reduce the difference in feature distribution between auxiliary data and task data, the classification accuracy of small samples can be improved. In addition, existing small sample classification methods rarely consider the relationship characteristics of data. If the similarity between model samples is modeled using graphs, the classification accuracy of small samples can be further improved by learning features on the graph.

[0005] The prototype network is a small sample classification method based on metric learning. The method mainly consists of two parts: feature extraction network and metric prediction. The main idea is to use the feature extraction network to map the image to the feature space, find the mean of each class of labeled samples as the prototype center of the class, and train the features of the same class of images to be closest to the prototype center of this category and farther away from the prototype centers of other classes. Then, the unlabeled samples are predicted using the distance between the unlabeled samples and the prototype center in the feature space. The classification accuracy of the prototype network is negatively correlated with the degree of aggregation of the features of each class of samples. Domain adaptation can improve the degree of aggregation of the common classes in the task dataset, and graph convolution operations can simultaneously improve the degree of aggregation of the common and private class datasets. Summary of the invention

[0006] In order to solve the above problems, the present disclosure provides an open set domain adaptation remote sensing image small sample classification method based on a multi-graph convolutional network (MGCN), which comprises the following steps:

[0007] S100: Read M samples from the task dataset and the auxiliary dataset respectively. Among the M samples from the task dataset, there are N categories, each category has K labeled samples, and the rest are unlabeled samples. The samples from the auxiliary dataset are all labeled samples.

[0008] S200: Extract features of all samples using feature extraction network Conv4;

[0009] S300: Based on the extracted features, the similarity relationship between the images is modeled using the K nearest neighbor and radial basis function method, and the adjacency matrix A is calculated;

[0010] S400: Update sample features using the obtained adjacency matrix A and multi-graph convolution operation;

[0011] S500: Using the updated features, obtain the prototype center of each category, calculate the distance from the unlabeled sample to the prototype center of each category, and predict the label of the unlabeled sample of the task data set;

[0012] S600: According to the predicted sample labels, sample features of the same category as the task dataset and the auxiliary dataset are selected, a domain loss function based on variance weighting is calculated, and the network is trained by back propagation;

[0013] S700: Use the trained network to classify the unlabeled samples of the task data set to obtain the classification results.

[0014] The beneficial effects of this program are:

[0015] (1) In the classification of remote sensing images, domain adaptation and graph convolutional networks are introduced into the small sample classification task. An open set domain adaptation small sample classification method based on multi-graph convolutional networks is proposed. This is the first method specifically proposed to solve the open set small sample classification problem.

[0016] (2) A variance-weighted domain adaptation method is proposed. This method assigns different weights to domain loss functions of different categories according to the strength of feature aggregation of different categories. This method focuses on difficult categories, which can effectively improve the efficiency of domain adaptation and classification accuracy.

[0017] (3) Use graphs to model the similarity between image features and utilize multi-graph convolution operations to enhance the aggregation of features of each category, further improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a small sample classification method for remote sensing images based on an open set domain adaptation and multi-graph convolutional network provided in one embodiment of the present disclosure;

[0019] Figure 2 is a schematic diagram of the Conv4 structure provided in one embodiment of the present disclosure;

[0020] Figure 3 It is an open set domain adaptive small sample classification model based on a multi-graph convolutional neural network in one embodiment of the present disclosure;

[0021] Figure 4 It is a flow chart of open set domain adaptation small sample classification training based on multi-graph convolutional network in one embodiment of the present disclosure;

[0022] Figure 5 It is a comparison chart of the classification accuracy of various methods when the number of labeled samples of each category in the task data set in one embodiment of the present disclosure is 1;

[0023] Figure 6 This is a comparison chart of the classification accuracy of various methods when there are 5 labeled samples of each category in the task data set in an embodiment of the present disclosure;

[0024] Figure 7 is the classification accuracy and α in one embodiment of the present disclosure c Curve chart with category changes;

[0025] Figure 8 is a curve diagram showing the change of classification accuracy with P when the labeled sample is 1 in one embodiment of the present disclosure;

[0026] Fig. 9 It is a curve chart showing the change of classification accuracy with P when the number of labeled samples is 5 in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In one embodiment, Figure 1 As shown, the present disclosure provides an open set domain adaptation remote sensing image small sample classification method based on a multi-graph convolutional network, which includes the following steps:

[0028] S100: Read M samples from the task dataset and the auxiliary dataset respectively. Among the M samples from the task dataset, there are N categories, each category has K labeled samples, and the rest are unlabeled samples. The samples from the auxiliary dataset are all labeled samples; wherein the task dataset and the auxiliary dataset are both datasets of remote sensing images;

[0029] S200: Extract features of all samples using feature extraction network Conv4;

[0030] S300: Based on the extracted features, the similarity relationship between the images is modeled using the K nearest neighbor and radial basis function method, and the adjacency matrix A is calculated;

[0031] S400: Update sample features using the obtained adjacency matrix A and multi-graph convolution operation;

[0032] S500: Using the updated features, obtain the prototype center of each category, calculate the distance from the unlabeled sample to the prototype center of each category, and predict the label of the unlabeled sample of the task data set;

[0033] S600: According to the predicted sample labels, sample features of the same category as the task dataset and the auxiliary dataset are selected, a domain loss function based on variance weighting is calculated, and the network is trained by back propagation;

[0034] S700: Use the trained network to classify the unlabeled samples of the task data set to obtain the classification results.

[0035] As far as this embodiment is concerned, in order to solve the problem that the auxiliary data and task data categories may overlap in the remote sensing image small sample classification task, this method proposes an open set domain adaptation small sample image classification method based on a multi-graph convolutional network. This method uses graphs to model the similarity relationship of images in the feature space, improves the aggregation degree of similar samples through graph convolution operations, and finally classifies the unlabeled samples according to the distance between the labeled samples and the unlabeled samples in the feature space. At the same time, in order to reduce the difference in feature distribution between the auxiliary data set and the data set of the same category, a domain adaptation method based on intra-class variance weighting is proposed. This method assigns a larger weight to the domain loss item of the category with a lower degree of aggregation during the training process, and assigns a smaller weight to the domain loss item of the category with a higher degree of aggregation, so as to more effectively enhance the aggregation degree of similar features. Compared with the original domain adaptation method, this method can further improve the classification accuracy. The multi-graph convolution model and the domain adaptation method based on variance weighting proposed in this method can be extended to other open set small sample classification tasks.

[0036] In another embodiment, the step S200 further includes the following steps:

[0037] S201: Input image X i The size of the image is uniformly adjusted to 80×80 pixels.

[0038] S202: After Conv4, a feature map of size 5×5×64 is obtained;

[0039] S203: Concatenate the feature maps into a 1600-dimensional feature vector as the input image X i The initial feature h i .

[0040] In this embodiment, the feature extraction part uses a four-layer convolutional neural network Cony4, and each layer of the convolutional neural network is followed by a normalization layer, a ReLU activation layer, and a maximum pooling layer. The structure of Conv4 is as follows: Figure 2 shown.

[0041] In another embodiment, the step S300 further comprises the following steps: i , radial basis function is used to calculate the adjacency matrix.

[0042] For this embodiment, the graph convolutional neural network is a deep learning network defined on a graph. First, the definition of a graph is given: a graph is a nonlinear data structure consisting of nodes and edges, usually used to describe a one-to-many data relationship in a non-Euclidean space. The graph can be represented as G = (V, E, A), where v is a set of nodes, E is a set of edges, and A∈R M×M is the adjacency matrix. In the figure, v i ∈V represents a node in the graph, i∈{1,…,M}, and M represents the number of nodes. ij =(v i , v j )∈E represents an edge on the graph, if e ij ∈E,A ij >0, otherwise A ij =0.

[0043] Graph convolution operation GCN:

[0044] H (l+1) =σ(5H (l) W (l) ) (1)

[0045] The method is used in the semi-supervised classification task of graph nodes. is the normalized augmented adjacency matrix, is the adjacency matrix of an undirected graph with self-loops, is the degree matrix, and there is l represents the number of layers in the graph convolutional network, W (l) is a trainable weight matrix, σ(·) is the activation function, usually ReLU(·)=max(0,.), H (l) ∈R M×D It is a matrix composed of the feature vectors of M nodes on the first layer of the graph convolution network, and D is the dimension of the feature. The process of graph convolution operation is to input A and H (l) In the case of , each node on the graph aggregates the features of neighboring nodes and updates its own features. Through graph spectrum analysis, it is pointed out that multiplying the feature matrix H of the node by S on the left in the graph convolution operation is equivalent to low-pass filtering H in the frequency domain. The effect of low-pass filtering is to smooth the features of the nodes on the graph, so that nodes with the same label have more similar features. When solving the semi-supervised classification problem of graph nodes, graph convolution can be used to effectively improve the classification accuracy.

[0046] Based on the above analysis, this method uses the "K nearest neighbor + radial basis function" method to model the similarity relationship between images, and then smoothes the image features through graph convolution operations to enhance the aggregation characteristics of similar node features to improve the classification accuracy of small image samples.

[0047] The M samples read from the data set are constructed into a graph convolutional network. Each node V in the graph convolutional network i , i∈{1,…,M} represents an image, where the number of labeled nodes is N×K, N represents the number of categories in the task data, K represents the number of labeled nodes in each category, and the initial features of the nodes are It is extracted by the feature extraction network Conv4, and then the adjacency matrix is ​​calculated using the radial basis function (RBF):

[0048]

[0049] N(i) refers to node V i K nearest neighbors, in the experiment γ = 0.2. Through the above method, in most cases, a larger weight will be assigned to nodes of the same category, and the weights between nodes of different categories are very small or 0. In this way, during the graph convolution operation, the features of nodes of the same category will quickly become similar, while the features of nodes of different categories have little influence on each other, which increases the distinguishability of node features and thus improves classification accuracy.

[0050] In another embodiment, step S400 further includes the following step: using a plurality of adjacency matrices of different powers for linear combination to obtain an adjacency matrix used for the final graph convolution.

[0051] In this embodiment, the features of the nodes are updated through graph convolution operations, and the aggregation characteristics of the features of similar nodes are enhanced to improve the classification accuracy of small image samples. In order to improve the effectiveness of graph convolution operations, this method uses a linear combination of several adjacency matrices of different powers to obtain the adjacency matrix used by the final graph convolution: and Here P represents the highest power of the adjacency matrix, α j is the weight coefficient to be learned, S j Refers to S to the jth power. After the graph is generated, a convolution operation is performed on the graph, and S in formula (1) is replaced by B. The final mathematical expression of graph convolution is as follows:

[0052] H=σ(BH 0 W) (3)

[0053] Experiments show that when P = 2, the model can achieve the highest accuracy in most cases, so when classifying small samples of remote sensing images, it is recommended that P = 2.

[0054] In another embodiment, the step S500 further includes the following steps:

[0055] S501: Use the feature vector of each class of labeled samples to calculate the prototype center c of each class k ;

[0056] S502: Calculate the distance of the prototype center of the kth category of the unlabeled sample in the feature space using the Euclidean distance;

[0057] S503: Use the classification probability function softmax to calculate the probability that the sample belongs to a certain class. The classification probability function softmax is expressed as:

[0058]

[0059] Among them, h i refers to the sample characteristics, p(y i =k|h i ) refers to the probability that the sample belongs to the kth class.

[0060] In this embodiment, after the graph convolution operation, the features output by the graph neural network are sent to the metric prediction part for prediction, and the class probability of the predicted node is output. First, the feature vector of each class of labeled nodes is used to calculate the prototype center of each class:

[0061]

[0062] In the above formula, h i and i Respectively represent the node v i The corresponding feature vector and label, N is the number of categories, S(k) represents the set of images with labels of the kth category in the input M images. After calculating the prototype center, use the Euclidean distance to calculate the distance between the unlabeled data and the prototype center of each category in the feature space, and then substitute it into the softmax to calculate the probability of the category:

[0063]

[0064] In another embodiment, Figure 3 As shown in the figure, the model structure based on this method mainly consists of three parts: (1) feature extraction network, which is used to extract image features; (2) multi-graph convolutional network, which is used to model the similarity relationship between image features and update node features through convolution operations on the graph; (3) metric prediction module, which realizes the prediction of unlabeled samples based on the metric distance between labeled samples and unlabeled samples in the feature space.

[0065] In another embodiment, the step S600 further includes the following steps:

[0066] S601: Select sample features in the task data set whose prediction probability is greater than a certain threshold, use the prediction results as pseudo labels of the samples, and calculate the ratio α of the intra-class variance and the nearest neighbor class distance of each category c ;

[0067] S602: Using α c The domain loss function and the classification loss function are calculated based on the features of the task dataset and the auxiliary dataset, and the entire model is trained by minimizing the domain loss function and the classification loss function.

[0068] As for this embodiment, during the training process of the model, in order to reduce the feature distribution difference between the auxiliary data set and the task data set, domain adaptation is introduced into the training process. The purpose of reducing the feature distribution difference is achieved by reducing the domain loss, and a domain adaptation method based on variance weighting is proposed.

[0069] This method solves the problem of open set domain adaptation small sample classification, that is, small sample classification when the auxiliary data set and the task data set have overlapping categories. For the common class data in the task data set, although the auxiliary data contains data of the same category, due to the difference in feature distribution of different data sets, the model trained on the auxiliary data set often cannot achieve high classification accuracy on the target data set. To this end, this method introduces a domain adaptation method to reduce the difference between the features of the auxiliary data and the task data. Use formula (6) to calculate the inter-domain difference:

[0070]

[0071] In the above formula, the left side of “+” measures the difference in the overall feature distribution of the two data sets, which is called the marginal distribution difference. The right side of “+” measures the difference in the feature distribution between the corresponding categories of the two data sets, which is called the conditional distribution difference. (6) In the formula, μ is a constant, x A 、x T Refer to the samples in the auxiliary dataset and the task dataset respectively, The superscript c indicates that the sample belongs to the cth class, f(·) refers to the feature extraction network, refers to the reproducing kernel Hilbert space, and C refers to the number of categories.

[0072] In the prototype network, domain adaptation can improve the aggregation of similar samples in the feature space. This method proposes to use the intra-class variance S of the sample c The square of the distance to the nearest neighbor The ratio between To measure the degree of aggregation of the c-th class samples in the feature space, where:

[0073]

[0074] Dc is the matrix formed by concatenating the characteristic vectors of the c-th type of data, 1 is a full 1 vector, n c Refers to the number of samples in the cth class. c The larger the value is, the more dispersed the distribution of the c-th class samples in the feature space is, the closer they are to other classes, and the lower the degree of aggregation is; conversely, the higher the degree of aggregation is. Intuitively speaking, α c The larger the category, the more difficult it is to classify. Experiments have shown that the classification accuracy of different categories is related to α c Therefore, in order to improve the effectiveness of domain adaptation, domain adaptation should be focused on difficult-to-distinguish categories. During model training, α should be given c The domain loss corresponding to the smaller category is assigned a smaller value, giving α c The domain loss of larger categories is assigned a larger value. The final domain loss function is modified to:

[0075]

[0076] in: C is the number of common classes. During the model training process, by minimizing L d In order to enhance the aggregation of similar data and improve the classification accuracy.

[0077] In another embodiment, the training of the above model can be divided into two stages: the first stage is the pre-training of the multi-graph convolutional network, and the second stage is the domain adaptation stage.

[0078] For this embodiment, the entire model training process is as follows: Figure 4 As shown. The loss function used in the first stage is the classification loss given by formula (9), and the loss function used in the second stage is the loss function given by formula (11).

[0079]

[0080]

[0081] L=L A +L T +L D (11)

[0082] In another embodiment, the first stage uses an auxiliary dataset to pre-train the multi-graph convolutional network MGCN to obtain initial parameters of the multi-graph convolutional network MGCN.

[0083] In another embodiment, the second stage uses a pre-trained multi-graph convolutional network MGCN to perform coarse classification of the task dataset, and selects NT data predicted to be a common class with a confidence level greater than a threshold to perform domain adaptation with the common class data in the auxiliary dataset.

[0084] In another embodiment, in order to verify the effectiveness of this method in open set domain adaptation to small sample classification, this method was used together with the existing ProtoNet and TPN to conduct experiments on three public datasets and the collected Qinghai-Tibet Plateau dataset. The similarities and differences of the three methods are shown in Table 1:

[0085]

[0086] Table 1

[0087] Remote sensing datasets:

[0088] The datasets used in the experiment include three public datasets: AID, OPTIMAL-31, and RSI-CB256, as well as the Qinghai-Tibet Plateau dataset collected from the Landsat satellite. The AID dataset is an aerial image dataset with 30 scene categories and a total of about 10,000 images, each of which is 600×600 pixels in size. The OPTIMAL-31 dataset contains 31 types of images collected from Google Maps, each of which consists of 60 images, with a size of 256×256 pixels, and a total of 1,860 images. RSI-CB256 is a large remote sensing image dataset that contains six major categories, including agricultural land, construction land, transportation and facilities, water conservancy facilities, forest land, and other land, 35 minor categories, and a total of 24,000 images. The size of each image is 256×256 pixels. In the experiment, the categories of the 35 minor categories are used as image labels. The Qinghai-Tibet Plateau dataset is a collection of remote sensing scene images of the Qinghai-Tibet Plateau taken by Landsat5 and Landsat8 satellites in 1990 and 2020 respectively. Each image is 256×256 pixels in size and contains six categories: bare land, forest, grassland, lake, river and snow-capped mountain, with about 500 images. Parameter settings:

[0089] The experiment uses the PyTorch deep learning framework and uses the NVIDIA GeForce RTX 3090 GPU for acceleration. In the experiment, the size of all remote sensing images was resized to 80×80 pixels, and all methods used Conv4 to extract image features. The weight matrix W∈R in the multi-graph convolutional network 1600×1000 Obtained through training. The training uses the Adma optimizer with a training step size of 10 -4 , the maximum number of cycles max_epoch = 20.

[0090] In order to evaluate the effectiveness of this method, experiments were conducted on three public datasets using this method and the existing methods ProtoNet and TPN, and the classification accuracy of each method on different datasets before and after domain adaptation was compared.

[0091] In order to construct the open set domain adaptation classification task, we first select data with the same category from any two datasets as common class data, and then select several categories from different categories of data as private class data, and use the remaining data as the validation set. For the AID and OPTIMAL-31 datasets, there are 9 categories of the same category. From the remaining categories of each dataset, 11 categories are selected in sequence as private classes, and combined with the 9 common categories as auxiliary datasets and task datasets, respectively. The selected categories are shown in Table 2:

[0092]

[0093] Table 2

[0094] Similarly, for the two combinations of AID and RSI-CB256, and OPTIMAL-31 and RSI-CB256, the selection of data set categories is shown in Table 3 and Table 4 respectively:

[0095]

[0096] Table 3

[0097]

[0098]

[0099] Table 4

[0100] Tables 5 and 6 respectively give the classification accuracy (%) of this method and the comparison method on different datasets when the number of labeled samples in each category is 1 and 5. The subscript "noDA" means that this method is directly tested on the task data without using domain adaptation after training on the auxiliary dataset. The subscript "DA" means that this method uses the domain adaptation method to train the model after pre-training on the auxiliary dataset. The subscript "α" means that this method uses the domain adaptation method proposed in this method. "A", "O", and "R" in the table are abbreviations of the three datasets AID, OPTIMAL-31, and RSI-CB256, respectively. The left side of "->" represents the auxiliary data, and the right side represents the task data.

[0101]

[0102] Table 5

[0103]

[0104] Table 6

[0105] From Table 5 and Table 6, we can see that the three methods have improved their classification accuracy after training with domain adaptation. At the same time, compared with ProtoNet, the proposed MGCN adds a multi-graph convolutional network on the basis of ProtoNet, and the feature extraction network is completely consistent. Therefore, it can be said that domain adaptation for common classes and convolution operations for graph networks constructed for all classes can play a role in improving classification accuracy. In addition, compared with MGCN DA and MGCN α It can be seen that compared with the existing domain adaptation method, the improved domain adaptation method of this method can further improve the classification accuracy by about 1%.

[0106] Table 7 shows the average α output by the five methods after training on three datasets. c From the data in the table, it can be concluded that both domain adaptation and graph convolution can reduce α c The value of α c reflects the degree of aggregation of sample features, α c The smaller the value, the higher the degree of aggregation of samples of the same category in the feature space, and the more accurate the classification result using the metric function. Therefore, the reason why domain adaptation and graph convolutional networks can improve the classification accuracy of the prototype network is that they improve the degree of aggregation of samples of the same category in the feature space.

[0107]

[0108] Table 7

[0109] Tables 8 and 9 give the training time and test time of different methods. It can be seen from the above two tables that the training time will increase to a certain extent after adding multi-graph convolution and domain adaptation. However, in the test, the average time for this method to classify an image is less than 2ms, which meets the needs of most applications.

[0110]

[0111] Table 8

[0112]

[0113] Table 9

[0114] In another embodiment, the MGCN method of the present invention is tested on a collected Qinghai-Tibet Plateau dataset with two methods, ProtoNet and TPN. The Qinghai-Tibet Plateau dataset has 6 categories, including bare land, forest, river, grassland, lake, and snowmountain. In the experiment, the auxiliary dataset selected data from 20 categories in AID, including bare land, forest, river, airport, baseball field, beach, bridge, center, church, commercial, dense land, desert, farmland, industrial, meadow, stadium, storage tanks, and viaduct. There are 3 categories in common in the two datasets: bare land, forest, and river.

[0115]

[0116]

[0117] Table 10

[0118] Table 10 shows the common class classification accuracy, private class classification accuracy and total classification accuracy of the proposed method and the comparison method on the Qinghai-Tibet Plateau dataset when the number of labeled samples in each category is 1 and 5. It can be concluded from Table 10 that after using domain adaptation, the common class classification accuracy of each method has been significantly improved, and the total classification accuracy has also been improved. The proposed domain adaptation method can achieve higher accuracy than the original domain adaptation method, which shows the effectiveness of the proposed domain adaptation method. Compared with PrOtONet noDA and MGCN noDA It can be seen that when the number of labeled samples per class is 1, adding a graph network can improve the accuracy by about 10%, and when the number of labeled samples is 5, the accuracy can be improved by about 5%. Using multi-graph convolution can further improve the classification accuracy. Compared with the comparison method, this method can also achieve the highest classification accuracy on the collected Qinghai-Tibet Plateau dataset.

[0119] In another embodiment, in order to explore the influence of domain adaptation and graph convolution operation on the prediction accuracy of common class data, the common class data given in Tables 2 to 4 are used as training and test data for training and testing respectively, and the classification results are as follows: Figure 5 , Figure 6 shown.

[0120] from Figure 5 , Figure 6It can be seen that ProtoNet, TPN and the MGCN proposed by this method have significantly improved their accuracy after using the domain adaptation method. This shows that in small sample classification, domain adaptation of the same category of data can significantly improve the classification accuracy of the metric method. After using the improved domain adaptation method, the accuracy can be further improved. At the same time, compared with ProtONet noDA ,MGCN noDA The method has an additional graph convolution operation, and the other parts are the same. When there is only 1 labeled sample per class, the accuracy is improved by 8%-15%, and when there are 5 labeled samples, the accuracy is improved by 2%-5%. Therefore, it can be concluded that by modeling the similarity relationship between nodes through a graph network and then updating the node features through convolution operations on the graph, the classification accuracy of small sample learning can also be improved.

[0121] In another embodiment, in order to explore the classification accuracy and α of each class in the prototype network c The relationship between them is analyzed. The common class data in OPTIMAL-31 and RSICB256 are selected. The prototype network is trained on one of the datasets and then tested on the other dataset. The classification accuracy of each class is arranged in order from small to large, and the curve of classification accuracy changing with category and the α of each class are drawn. c The curves that vary with category are as follows Figure 7 shown.

[0122] from Figure 7 It can be seen that α c It is negatively correlated with the overall classification accuracy of the category. c It reflects the degree of aggregation of a certain type of samples in the feature space, α c The larger the value, the larger the variance of samples in this category is in the feature space and the closer they are to other categories. The lower the degree of feature aggregation is, the lower the corresponding classification accuracy is. c The smaller it is, the higher the degree of aggregation is, and the corresponding classification accuracy is also higher.

[0123] In another embodiment, in order to explore the influence of the order of the adjacency matrix on the accuracy in the multi-graph convolution operation, the classification accuracy of the proposed MGCN method on three different datasets was tested when the values ​​of p in formula (3) were 1, 2, 3, 4, and 5 respectively without using domain adaptation. The results are shown in Figure 2. Figure 8 , Fig. 9 shown.

[0124] Depend on Figure 8 , Fig. 9It can be seen that when the number of labeled samples in each category is 1 or 5, P = 2 can achieve a 0.5%-3% improvement compared to P = 1. When P is greater than 2, the classification accuracy of some data sets increases slightly, while other parts decrease slightly. Therefore, in the experiment, the value of P is 2.

[0125] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A small sample classification method for remote sensing images based on open set domain adaptation and multi-graph convolutional network, comprising the following steps: S100: Read M samples from the task dataset and the auxiliary dataset respectively. Among the M samples from the task dataset, there are N categories, each category has K labeled samples, and the rest are unlabeled samples. The samples from the auxiliary dataset are all labeled samples; Both the mission dataset and the auxiliary dataset are datasets of remote sensing images; S200: Extract features of all samples using feature extraction network Conv4; S300: Based on the extracted features, the similarity relationship between the images is modeled using the K nearest neighbor and radial basis function method, and the adjacency matrix A is calculated; S400: Update sample features using the obtained adjacency matrix A and multi-graph convolution operation; S5 00: Use the updated features to obtain the prototype center of each category, calculate the distance from the unlabeled sample to the prototype center of each category, and predict the label of the unlabeled sample in the task dataset; S600: According to the predicted sample labels, sample features of the same category as the task dataset and the auxiliary dataset are selected, a domain loss function based on variance weighting is calculated, and the network is trained by back propagation; S700: Use the trained network to classify the unlabeled samples of the task data set to obtain the classification results; The step S400 further includes the following steps: using a plurality of adjacency matrices of different orders to perform linear combination to obtain the adjacency matrix used for the final graph convolution , and , P represents the highest power of the adjacency matrix, is the weight coefficient that needs to be learned, Refers to S to the jth power; then a convolution operation is performed on the graph to update the sample features. The operation expression for updating the sample features is: , where H 0 is the original feature of the sample, H is the updated sample feature, is the ReLU activation function; The feature extraction network Conv4 is a four-layer convolutional neural network, and each layer of the convolutional neural network is followed by a normalization layer, a ReLU activation layer, and a maximum pooling layer; The variance-weighted domain loss function is a domain loss function obtained by correcting the ratio between the intra-class variance of the sample and the square of the nearest neighbor class distance. The calculation formula of the domain loss function is: ； Where: μ is a constant, , Refer to the samples in the auxiliary dataset and the task dataset respectively, The superscript c indicates that the sample belongs to the cth class. refers to the feature extraction network, It means to find the mathematical expectation. refers to the reproducing kernel Hilbert space, is a weighting coefficient, It is the ratio of the variance within the c-th class sample to the square of the nearest neighbor class distance, where C refers to the number of classes.

2. The method according to claim 1, wherein: The step S200 further includes the following steps: S201: Input image x i The size of the RGB image is uniformly adjusted to 80×80 pixels; S202: After Conv4, a feature map of size 5×5×64 is obtained; S203: Concatenate the feature maps into a 1600-dimensional feature vector as the input image x i The initial feature h i .

3. The method according to claim 1, wherein: The step S500 further includes the following steps: S501: Calculate the prototype center of each class using the feature vector of each class of labeled samples ; S502: Calculate the distance of the prototype center of the kth category of the unlabeled sample in the feature space using the Euclidean distance; S503: Use the classification probability function softmax to calculate the probability that the sample belongs to a certain class. The classification probability function softmax is expressed as: ； in, Refers to the sample characteristics, Refers to the probability that the sample belongs to the kth class.

4. The method according to claim 1, wherein: The step S600 further includes the following steps: S601: Select sample features in the task data set whose prediction probability is greater than a certain threshold, use the prediction results as pseudo labels of the samples, and calculate the ratio α of the intra-class variance and the nearest neighbor class distance of each category c ; S602: Using α c The domain loss function and classification loss function are calculated based on the features of the task dataset and the auxiliary dataset, and the entire model is trained by minimizing the loss function.

5. The method according to claim 1, wherein: The model on which this method is based consists of the following three parts: (1) a feature extraction network, which is used to extract image features; (2) a multi-graph convolutional network, which is used to model the similarity relationship between image features and update sample features through convolution operations on the graph; and (3) a metric prediction module, which predicts unlabeled samples based on the metric distance between labeled samples and unlabeled samples in the feature space.

6. The method according to claim 5, wherein: A variance-weighted domain adaptation method is introduced during the training process of the above model.

7. The method according to claim 6, wherein: The training of the above model is divided into two stages: the first stage is the pre-training of the multi-graph convolutional network, and the second stage is the domain adaptation stage.

8. The method according to claim 7, wherein: In the first stage, the auxiliary data set is used to pre-train the multi-graph convolutional network MGCN to obtain the initial parameters of the multi-graph convolutional network MGCN.

9. The method according to claim 7, wherein: The second stage uses the pre-trained multi-graph convolutional network MGCN to perform coarse classification of the task dataset, selects data predicted as common classes with a confidence level greater than a certain threshold, and performs domain adaptation with the common class data in the auxiliary dataset.

Images