Small sample target classification method for domain-independent meta-training

Abstract

The invention relates to a small sample target classification method for domain-independent meta-training, and belongs to the technical field of image information processing. According to the method, small sample learning tasks of different source domains are adapted through internal circulation, the domain adaptive capacity of parameters is tested, the element network parameters are updated through the updating direction obtained through the internal circulation domain adaptive tasks in external circulation, and the domain adaptive capacity is migrated to the element network. Through training of the classification method, model parameters cannot obtain an optimal solution, the model cannot be over-fitted with source domain data, and the model obtains the domain generalization ability through learning a domain adaptation task every time.

Description

Technical Field

[0001] The invention relates to a method for classifying cross-domain small sample remote sensing image targets, and belongs to the technical field of image information processing. Background Art

[0002] At present, the technologies related to the present invention include two aspects: the first is a small sample target classification method; the second is a cross-domain small sample target classification method.

[0003] The small sample target classification algorithms based on deep learning are mainly divided into three categories: the first method is the small sample target classification algorithm based on metric learning, which projects the small sample target into the feature space by learning a specific feature space and classifies it by feature similarity. Snell et al. proposed in the paper "Prototypical Networks for Few-shot Learning" to achieve small sample classification by learning the category prototype vector and using the Euclidean distance to measure the similarity between the sample and the prototype. Vinyals et al. proposed in the paper "Matching Networks for One Shot Learning" to achieve small sample classification by learning a metric network and using the network to calculate the similarity between the query sample and the support set sample. The second type of small sample target classification algorithm based on meta-learning improves the learning ability of the model in small sample scenarios by learning how to quickly learn new tasks from multiple related tasks. Finn et al. used a series of related "meta-tasks" as training units in the paper "Model-AgnosticMeta-Learning for Fast Adaptation of Deep Networks". Through training, the model can learn a parameter initialization that can quickly adapt to new tasks, so that the model can learn new tasks with very few samples. Karaali et al. introduced a local adaptive scheme and a new filter in the paper "Edge-based defocus blur estimation with adaptive scale selection" to obtain better defocus images to train the deblurring network. The third type of small sample target classification algorithm based on transfer learning mainly uses the features learned from the base class for new categories through fine-tuning or knowledge distillation, and adjusts the pre-trained model used for the source task to the target task. Mangla et al. introduced the Mixup image mixing technology in the paper "Charting the Right Manifold: Manifold Mixup for Few-shot Learning" to help fine-tune the model, so that the model is robust when there are slight changes in the data distribution, thereby improving the performance of the model on small sample tasks.

[0004] Further cross-domain small sample target classification methods have also received widespread attention this year. Cross-domain small sample target classification algorithms can be roughly divided into two categories: enhancement-based methods and transfer learning-based methods. On the one hand, enhancement-based methods enhance the robustness of the model to domain shift by enhancing the domain diversity of data at the task, feature, pixel and other levels. In the paper StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-ShotLearning, Fu et al. used adversarial attack methods to change the mean and variance of features, and maximized the distribution of differences to make the model robust to domain shift. On the other hand, transfer learning-based methods make the model pay more attention to domain-independent information in the data through fine-tuning or learning of specific subtasks, thereby improving the model's generalization ability for unseen domains. In the paper Revisiting Prototypical Network for Cross Domain Few-Shot Learning, Zhou et al. cropped image blocks and assisted model learning through image block classification tasks, making the model pay more attention to category-related local features and improving the model's domain generalization ability.

[0005] However, most of the enhanced cross-domain small sample algorithms improve the domain generalization ability of the model by enhancing the domain diversity of single source domain data. However, the fitting ability of a single source domain for the actual domain offset is often limited. In addition, in the training and domain adaptation stages, there is a large difference between the amount of source domain data and the amount of target domain data in algorithms based on enhanced or transfer learning, which makes it easy for the model to overfit the source domain. Summary of the invention

[0006] Aiming at the problem that the cross-domain small sample target classification of remote sensing images has limited ability to fit domain offset and is easy to overfit the source domain, the present invention proposes a domain-independent meta-training small sample target classification method, which adapts to small sample learning tasks in different source domains through an inner loop, tests the domain adaptability of parameters, and updates the meta-network parameters in the outer loop through the update direction obtained by the inner loop domain adaptation task, so as to transfer the domain adaptability to the meta-network. Through the training of the above classification method, the model parameters will not obtain the optimal solution, and the model will not overfit the source domain data, but the model will acquire the domain generalization ability by learning the domain adaptation task each time.

[0007] The domain-independent meta-training small sample target classification method of the present invention is special in that it includes the following steps:

[0008] Step 1: Load the meta-network and twin network structures and initial network parameters into the graphics card, where the meta-network and twin network have the same network structure and initial parameters. In subsequent steps, the meta-network and twin network share network parameters, where the meta-network parameters are θ m , the twin network parameter is θ s ,The network structure is composed of residual convolution blocks and fully connected layers. After the graphics card loads the initial values ​​of the network parameters, it enters the inner loop step of meta-training;

[0009] Step 2: In the meta-training phase of the inner loop, small sample classification tasks are sampled from multiple source domains as the basic learning unit to train the model. The support set of each small sample classification task contains N categories, and each category contains K images. The Ubuntu-based server loads the images into the classification model on the graphics card for calculation, and updates the twin network parameters through back propagation of the loss.

[0010] Step 3: Through learning of small sample tasks in multiple different source domains, the updated twin network parameters are obtained. The classification method enters the cross-domain meta-testing stage. The server samples small sample tasks in the meta-testing domain and inputs the data of the small sample tasks into the twin network in the graphics card to calculate the classification results. The updated gradient of the network parameters is obtained through the back propagation of the classification loss.

[0011] Step 4: After obtaining the updated gradient of the network parameters in the cross-domain meta-test phase, the classification algorithm enters the outer loop. The outer loop updates the meta-network parameters on the graphics card through the update direction obtained by the inner loop, so that the meta-network can obtain domain adaptation ability through the inner loop;

[0012] Step 5: After each update of the meta-network parameters, the algorithm updates the twin network with the parameters of the meta-network on the graphics card by exponential moving average, and enters the next inner loop meta-training phase of the outer loop nesting.

[0013] Preferably, the meta-network parameter θ in step 1 is m Siamese network parameters θ s ,Each step of the inner loop updates the Siamese network by learning the domain adaptation task, and the initial Siamese network is a copy of the meta-network.

[0014] Preferably, the twin network and the meta-network in step 1 have the same structure, both of which are composed of a feature extractor and a classification head; wherein the feature extractor is composed of five residual convolution blocks, wherein each residual convolution block is composed of three 3*3 convolution layers and a residual connection with a 1*1 convolution layer connected to the input, as well as an activation function and a maximum pooling layer; the classification head is composed of multiple fully connected layers, which reduces the feature dimension to the dimension of the number of categories, thereby obtaining the classification score of the image.

[0015] Preferably, in the meta-training phase described in step 2, each update of the twin network is performed with the help of a small sample learning task, and in the domain adaptation task DA i In the meta-training phase, Small sample classification task To update the twin network parameters A copy of in From the source domain Few-shot learning tasks with random sampling.

[0016] Preferably, the updating step in step 4 is expressed by the following formula:

[0017]

[0018] Where α represents the cross-domain learning rate, φ N represents the updated twin network parameters after learning the Nth small sample classification task, Indicates that the network parameter φ j-1 Learning small sample tasks based on The classification loss obtained, where α is a known quantity, φ N , φ j-1 As is an unknown quantity, we let the model learn the small sample classification tasks in different domains during the meta-training phase, so that it can quickly adapt to the small sample classification tasks in different domains.

[0019] Preferably, the task of sampling a small sample in the meta-test domain described in step 3 needs to test φ after the meta-training phase is completed. N Whether it has the expected domain adaptation ability, in the meta-testing stage, the data domain that has not been seen in the meta-training stage is selected as the query domain in the multi-source domain And sample small sample tasks from the query domain To test the twin network parameters φ obtained by meta-training N , through meta-test, the updated gradient g is obtained by back-propagating the calculated classification loss i , the calculation formula is as follows:

[0020]

[0021] in Indicates that the network parameter φ N Based on learning small sample tasks The classification loss obtained is N , is a known quantity, Is an unknown quantity.

[0022] Preferably, the network parameter update gradient obtained in the cross-domain meta-test phase in step 4 is used to update the meta-network parameters so that the meta-network parameters have domain adaptability. After the meta-network parameters are updated, the classification method enters the next outer loop, in which the twin network parameters are updated. The calculation formula is as follows:

[0023]

[0024] Where β represents the domain adaptation learning rate. When the meta-test phase is over, the learning of a domain adaptation task is completed. Where θ N , β, g i is a known quantity, is an unknown quantity;

[0025] The outer loop uses the domain adaptation task as the basic learning unit. Each step of the inner loop is to learn the domain adaptation task and transfer the domain generalization ability to the meta-network. After learning the domain adaptation task, it is necessary to transfer the domain generalization ability of the twin network to the meta-network and calculate the update direction d of the network parameters by updating the twin network parameters. i , the formula is as follows:

[0026]

[0027] The left side of the equation contains unknown quantities, and the right side of the equation contains unknown quantities;

[0028] Update the meta-network parameters according to the update direction The calculation formula is as follows:

[0029]

[0030] Where γ represents the meta-learning rate, which is a hyperparameter for updating the meta-network. The left side of the equation is an unknown quantity, and the right side of the equation is a known quantity.

[0031] Preferably, in step 5, the twin network parameters are updated by exponential moving average, and the twin network θ s and meta-network θ m Shared parameters, before entering the next outer loop, update the twin network parameters by exponential moving average. The formula is as follows:

[0032]

[0033] Where m represents the weighted weight of the moving average, the left side of the equation is the unknown quantity, and the right side of the equation is the known quantity. When the twin network parameters are updated, it enters the next domain adaptation inner loop stage.

[0034] Beneficial effects of the present invention: The present invention proposes to learn the real domain shift on multi-source domain data, and proposes a domain-independent meta-training classification method based on the multi-source domain, taking the domain adaptation task as the basic training unit, so that the model can learn its own ability to adapt to the unknown target domain. The domain-independent meta-training classification method is mainly divided into two layers of loops: In the inner loop, we focus on updating the twin network of the meta network. Each step of the inner loop is to learn small sample tasks collected from different source domains, so as to update the parameters of the twin network. In order to test the domain adaptability of the parameters obtained by the inner loop, we test pseudo-target domain data on the twin network; the entire inner loop is defined by us as a domain adaptation task, so each step of the outer loop is to update the parameters of the meta network through the update direction obtained by the inner loop. In this process, although the model parameters are not the optimal solution for the source domain, they have the ability to quickly adapt to unseen target domains. The present invention uses a cross-domain meta-training classification method based on an inner and outer two-layer loop to update network parameters. By learning the domain adaptation task through the inner loop, the network has domain generalization ability, avoiding overfitting of parameters to source domain data and falling into the trap of local optimal solution. Moreover, through cross-domain meta-training under multiple source domains, the model can learn the real domain offset, thereby alleviating the adverse effects of source domain offset on model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the overall training flow chart of the network;

[0036] Figure 2 A schematic diagram of the backbone network structure;

[0037] Figure 3 Schematic diagram of the residual block structure of the backbone network. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0039] The domain-independent meta-training small sample target classification method of this embodiment is: Figure 1 This is the overall training flow chart of the network. The first step is to set the initial parameters of the meta-network when entering the outer loop for the first time. Passed to the twin network θ s , the structure of the feature extractor is as follows Figure 2 As shown in Figure 1, the network is divided into two parts: feature extractor and classifier. The feature extractor consists of five residual convolution blocks with increasing channels in sequence. The classifier consists of an activation function, an average pooling layer, and a 512*k-shot fully connected layer. The specific structure of the residual connection block is as follows: Figure 3As shown in the figure, the input features are convolved through multiple 3*3 convolutional layers, batch normalization layers and ReLU activation functions, and then the input feature map is concatenated with the output feature map. Then, the feature map is output through the activation function and the maximum pooling layer. In the second step, the twin network will learn the domain adaptation task TD, as shown in Figure 1 As shown, it enters the inner loop update, takes AID, PatternNet, and NWPU-RESISC45 remote sensing data sets as source domains, and samples multiple small sample learning tasks to update the twin network parameters. Specifically, the training method of updating the twin network through small sample tasks adopts the small sample learning method proposed by Finn et al. in the literature "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks". The third step is to evaluate the domain generalization ability of the twin network parameters through the small sample classification tasks sampled from the pseudo test domain UCMerced Land Use Dataset after learning the small sample classification tasks from different source domains. Through the test domain small sample classification tasks, the update direction of the network can be obtained. The fourth step is to update the meta-network parameters through the update direction obtained by the inner loop, in order to expect the meta-network parameters to have the ability to adapt to the domain by themselves, and pass the updated meta-network parameters to the twin network parameters through the exponential moving average method, and go to the first step to prepare for the next outer loop. In the second step, the network updates the network parameters by the following formula (1). In the third step, the update direction of the meta-network is calculated by the following formula (2). The loss used in the formula is the cross entropy loss used in the classification task. After multiple outer loop updates, the meta-network will learn a non-optimal solution parameter. This parameter has the ability to adapt to the domain through the learning of domain adaptation tasks in multiple source domains, and the domain generalization ability of the network is tested on the RSI-CB256 / 128 dataset.

[0040] The domain-independent meta-training small sample target classification method of this embodiment uses the domain adaptation task as the basic learning unit, and enables the model to acquire the ability to adapt to unseen target domains through internal and external loop training. The specific steps are as follows:

[0041] The entire training process is mainly divided into two stages: domain adaptation inner loop and cross-domain outer loop;

[0042] (1) Domain Adaptation Internal Circulation Stage

[0043] Each step of the inner loop updates the twin network by learning the domain adaptation task, and the initial twin network is a copy of the meta network. Specifically, the twin network and the meta network have the same structure, both consisting of a feature extractor and a classification head. Among them, the feature extractor consists of five residual convolution blocks, each of which consists of three 3*3 convolutional layers and a residual connection with a 1*1 convolutional layer connected to the input, as well as an activation function and a maximum pooling layer. The classification head consists of multiple fully connected layers, which reduces the feature dimension to the number of categories, thereby obtaining the classification score of the image. The domain adaptation task DA defined in our algorithm is divided into meta-training and meta-testing stages.

[0044] Specifically, in the meta-training phase, each update of the twin network is performed with the help of a small sample learning task. Therefore, as a plug-and-play meta-training classification method, we can use the meta-learning task proposed by Finn et al. in the paper "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" as the small sample learning task in the training phase. i In the meta-training phase, we extract Small sample classification task To update the twin network parameters A copy of in From the source domain The update step of meta-training can be expressed as follows:

[0045]

[0046] Where α represents the cross-domain learning rate, φ N represents the updated twin network parameters after learning the Nth small sample classification task, Indicates that the network parameter φ j-1 Learning small sample tasks based on The classification loss obtained. In the meta-training phase, we let the model learn small sample classification tasks in different domains, hoping that it can quickly adapt to small sample classification tasks in different domains. Therefore, after the meta-training phase, it is necessary to test φ N Whether it has the domain adaptation ability we expect. Therefore, in the meta-testing phase, we select the data domain that has not been seen in the meta-training phase as the query domain in the multi-source domain. And sample small sample tasks from the query domain To test the twin network parameters φ obtained by meta-training N Through meta-testing, the updated gradient g can be obtained by back-propagating the calculated classification loss i , the calculation formula is as follows:

[0047]

[0048] in Indicates that the network parameter φ N Based on learning small sample tasks The classification loss is obtained. Then we update the twin network parameters The calculation formula is as follows:

[0049]

[0050] Where β represents the domain adaptation learning rate. When the meta-test phase ends, the learning of a domain adaptation task ends.

[0051] (2) Domain Adaptation Outer Loop

[0052] The outer loop uses the domain adaptation task as the basic learning unit, and each step of the inner loop is to learn the domain adaptation task and transfer the domain generalization ability to the meta-network. Specifically, after learning the domain adaptation task, it is necessary to transfer the domain generalization ability of the twin network to the meta-network. Therefore, by updating the parameters of the twin network, we can calculate the update direction d of the network parameters. i , the formula is as follows:

[0053]

[0054] Then update the meta-network parameters according to the update direction The calculation formula is as follows:

[0055]

[0056] Where γ represents the meta-learning rate, which is a hyperparameter for updating the meta-network. Each update of the meta-network parameters represents the end of an outer loop. In addition, because the twin network θ s and meta-network θ m Shared parameters, so before entering the next outer loop, the twin network parameters are updated by exponential moving average. The formula is as follows:

[0057]

[0058] Where m represents the weighted weight of the moving average. Once the twin network parameters are updated, the next domain adaptation inner loop phase can be entered.

[0059] The domain-independent meta-training small sample target classification method of this embodiment adapts to small sample learning tasks in different source domains through an inner loop, tests the domain adaptability of the parameters, and updates the meta-network parameters in the outer loop through the update direction obtained by the inner loop domain adaptation task, and transfers the domain adaptability to the meta-network. Through the training of the above classification method, the model parameters will not obtain the optimal solution, and the model will not overfit the source domain data, but the model will acquire domain generalization ability by learning the domain adaptation task each time.

Claims

1. A domain-independent meta-training method for small sample target classification, characterized by The following steps are involved: Step 1: Load the meta-network and twin network structures and initial network parameters into the graphics card, where the meta-network and twin network have the same network structure and initial parameters. In subsequent steps, the meta-network and twin network share network parameters, where the meta-network parameters are θ m , the twin network parameter is θ s ,The network structure is composed of residual convolution blocks and fully connected layers. After the graphics card loads the initial values ​​of the network parameters, it enters the inner loop step of meta-training; Step 2: In the meta-training phase of the inner loop, small sample classification tasks are sampled from multiple source domains as the basic learning unit to train the model. The support set of each small sample classification task contains N categories, and each category contains K images. The Ubuntu-based server loads the images into the classification model on the graphics card for calculation, and updates the twin network parameters through back propagation of the loss. Step 3: Through learning of small sample tasks in multiple different source domains, the updated twin network parameters are obtained. The classification method enters the cross-domain meta-testing stage. The server samples small sample tasks in the meta-testing domain and inputs the data of the small sample tasks into the twin network in the graphics card to calculate the classification results. The updated gradient of the network parameters is obtained through the back propagation of the classification loss. Step 4: After obtaining the updated gradient of the network parameters in the cross-domain meta-test phase, the classification algorithm enters the outer loop. The outer loop updates the meta-network parameters on the graphics card through the update direction obtained by the inner loop, so that the meta-network can obtain domain adaptation ability through the inner loop; Step 5: After each update of the meta-network parameters, the algorithm updates the twin network with the parameters of the meta-network on the graphics card by exponential moving average, and enters the next inner loop meta-training phase of the outer loop nesting.

2. The domain-independent meta-training small sample target classification method according to claim 1, characterized in that The meta-network parameters θ described in step 1 m Siamese network parameters θ s ,Each step of the inner loop updates the Siamese network by learning the domain adaptation task, and the initial Siamese network is a copy of the meta-network.

3. The domain-independent meta-training small sample target classification method according to claim 2, characterized in that The twin network and meta-network described in step 1 have the same structure, both of which are composed of a feature extractor and a classification head; the feature extractor is composed of five residual convolution blocks, each of which is composed of three 3*3 convolution layers and a residual connection with a 1*1 convolution layer connected to the input, as well as an activation function and a maximum pooling layer; the classification head is composed of multiple fully connected layers, which reduces the feature dimension to the dimension of the number of categories, thereby obtaining the classification score of the image.

4. The domain-independent meta-training small sample target classification method according to claim 1, characterized in that In the meta-training phase described in step 2, each update of the twin network is performed with the help of a small sample learning task. i In the meta-training phase, Small sample classification task To update the twin network parameters A copy of in From the source domain Few-shot learning tasks with random sampling.

5. The domain-independent meta-training small sample target classification method according to claim 1, characterized in that The update step described in step 4 is expressed by the following formula: Where α represents the cross-domain learning rate, φ N represents the updated twin network parameters after learning the Nth small sample classification task, Indicates that the network parameter φ j-1 Learning small sample tasks based on The classification loss obtained, where α is a known quantity, φ N , φ j-1 As is an unknown quantity, we let the model learn the small sample classification tasks in different domains during the meta-training phase, so that it can quickly adapt to the small sample classification tasks in different domains.

6. The domain-independent meta-training small sample target classification method according to claim 1, characterized in that The task of sampling a small sample in the meta-test domain described in step 3 requires testing φ after the meta-training phase. N Whether it has the expected domain adaptation ability, in the meta-testing stage, the data domain that has not been seen in the meta-training stage is selected as the query domain in the multi-source domain And sample small sample tasks from the query domain To test the twin network parameters φ obtained by meta-training N , through meta-test, the updated gradient g is obtained by back-propagating the calculated classification loss i , the calculation formula is as follows: in Indicates that the network parameter φ N Based on learning small sample tasks The classification loss obtained is N , is a known quantity, Is an unknown quantity.

7. The domain-independent meta-training small sample target classification method according to claim 1, characterized in that The network parameter update gradient obtained in the cross-domain meta-test phase in step 4 is used to update the meta-network parameters so that the meta-network parameters have domain adaptability. After updating the meta-network parameters, the classification method enters the next outer loop, in which the twin network parameters are updated. The calculation formula is as follows: Where β represents the domain adaptation learning rate. When the meta-test phase is over, the learning of a domain adaptation task is completed. Where θ N , β, g i is a known quantity, is an unknown quantity; The outer loop uses the domain adaptation task as the basic learning unit. Each step of the inner loop is to learn the domain adaptation task and transfer the domain generalization ability to the meta-network. After learning the domain adaptation task, it is necessary to transfer the domain generalization ability of the twin network to the meta-network and calculate the update direction d of the network parameters by updating the twin network parameters. i , the formula is as follows: The left side of the equation contains unknown quantities, and the right side of the equation contains unknown quantities; Update the meta-network parameters according to the update direction The calculation formula is as follows: Where γ represents the meta-learning rate, which is a hyperparameter for updating the meta-network. The left side of the equation is an unknown quantity, and the right side of the equation is a known quantity.

8. The domain-independent meta-training small sample target classification method according to claim 1, characterized in that In step 5, the twin network parameters are updated by exponential moving average, and the twin network θ s and meta-network θ m Shared parameters, before entering the next outer loop, update the twin network parameters by exponential moving average. The formula is as follows: Where m represents the weighted weight of the moving average, the left side of the equation is the unknown quantity, and the right side of the equation is the known quantity. When the twin network parameters are updated, it enters the next domain adaptation inner loop stage.

Images