Domain-independent meta-training for few-shot object classification

Abstract

The present invention relates to a small-sample target classification method using domain-independent meta-training, and belongs to the field of image information processing technology. The present invention adapts to small-sample learning tasks in different source domains through an inner loop, tests the domain adaptability of the parameters, and updates the meta-network parameters in the outer loop using the update direction obtained from the inner loop domain adaptation task, thereby migrating the domain adaptability to the meta-network. Through training using the above classification method, the model parameters will not obtain the optimal solution, and the model will not overfit the source domain data. Instead, the model will acquire domain generalization capabilities through each learning domain adaptation task.

Description

Technical Field

[0001] The present invention relates to a method for classifying cross-domain small-sample remote sensing image targets, and belongs to the technical field of image information processing. Background Art

[0002] Currently, the technologies related to the present invention include two aspects: the first is a small sample target classification method; the second is a cross-domain small sample target classification method.

[0003] Deep learning-based small-shot object classification algorithms can be broadly divided into three categories. The first category involves metric learning-based small-shot object classification algorithms. These algorithms learn a specific feature space, project the small-shot objects onto that feature space, and classify them based on feature similarity. In "Prototypical Networks for Few-shot Learning," Snell et al. proposed learning a prototype vector for a category and using Euclidean distance to measure the similarity between the sample and the prototype, thereby achieving small-shot classification. In "Matching Networks for One Shot Learning," Vinyals et al. proposed learning a metric network and using it to calculate the similarity between the query sample and the support set samples, thereby achieving small-shot classification. The second category of meta-learning-based small-shot object classification algorithms improves the model's learning ability in small-shot scenarios by learning how to quickly learn new tasks from multiple related tasks. In "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks," Finn et al. used a series of related "meta-tasks" as training units. Through training, the model learned a parameter initialization that can quickly adapt to new tasks, enabling the model to learn new tasks with very few samples. In the paper "Edge-based defocus blur estimation with adaptive scale selection", Karaali et al. introduced a local adaptive scheme and a new filter to obtain better defocus maps for training the deblurring network. The third type of small-sample target classification algorithm based on transfer learning mainly uses fine-tuning or knowledge distillation to apply features learned from the base class to new categories, and adjusts the pre-trained model used for the source task to the target task. In the paper "Charting the Right Manifold: Manifold Mixup for Few-shot Learning", Mangla et al. introduced the Mixup image mixing technology to help fine-tune the model, making the model robust when there are slight changes in the data distribution, thereby improving the model's performance on small-sample tasks.

[0004] Cross-domain few-shot object classification methods have also received widespread attention this year. Cross-domain few-shot object classification algorithms can be roughly divided into two categories: enhancement-based methods and transfer learning-based methods. Enhancement-based methods enhance the model's robustness to domain shift by increasing the domain diversity of data at the task, feature, and pixel levels. In "StyleAdv: Meta-Style Adversarial Training for Cross-Domain Few-Shot Learning," Fu et al. used adversarial attacks to alter the mean and variance of features, maximizing the distribution of differences and making the model robust to domain shift. Transfer learning-based methods, on the other hand, use fine-tuning or learning specific subtasks to make the model more attentive to domain-independent information in the data, improving its generalization to unseen domains. In "Revisiting Prototypical Network for Cross-Domain Few-Shot Learning," Zhou et al. cropped image patches and used patch classification tasks to assist model learning, enabling the model to focus more on local features related to categories and improving its domain generalization.

[0005] However, most cross-domain small-sample algorithms based on reinforcement learning improve the domain generalization ability of the model by enhancing the domain diversity of single-source domain data. However, the fitting ability of a single source domain to the actual domain offset is often relatively limited. In addition, during the training and domain adaptation stages, the amount of source domain data and the target domain data in algorithms based on reinforcement learning or transfer learning have a large difference, which makes the model prone to overfitting the source domain. Summary of the Invention

[0006] To address the problem of limited domain offset and easy overfitting of the source domain in cross-domain small-sample target classification of remote sensing images, this paper proposes a domain-independent meta-training small-sample target classification method. This method uses an inner loop to adapt to small-sample learning tasks in different source domains and test the domain adaptability of the parameters. In the outer loop, the meta-network parameters are updated using the update direction obtained from the inner loop domain adaptation task, thereby migrating the domain adaptability to the meta-network. Through training with this classification method, the model parameters will not reach the optimal solution, and the model will not overfit the source domain data. Instead, the model will acquire domain generalization ability through each learning domain adaptation task.

[0007] The domain-independent meta-training small sample target classification method of the present invention is special in that it includes the following steps:

[0008] Step 1: Load the meta-network and twin network structures and initial network parameters into the graphics card, where the meta-network and twin network have the same network structure and initial parameters. In subsequent steps, the meta-network and twin network share network parameters, where the meta-network parameters are θ m , the twin network parameters are θ s ,The network structure consists of residual convolution blocks and fully connected layers.,After the graphics card loads the initial values ​​of the network parameters,,it enters the meta-training inner loop step;

[0009] Step 2: In the meta-training phase of the inner loop, small-sample classification tasks are sampled from multiple source domains as the basic learning unit to train the model. The support set of each small-sample classification task contains N categories, and each category contains K images. The Ubuntu-based server loads the images into the classification model on the graphics card for calculation and updates the Siamese network parameters through backpropagation of the loss.

[0010] Step 3: Through learning of small sample tasks in multiple different source domains, the updated twin network parameters are obtained. The classification method enters the cross-domain meta-testing phase. The server samples small sample tasks in the meta-testing domain and inputs the small sample task data into the twin network in the graphics card to calculate the classification results. The updated gradient of the network parameters is obtained through the back propagation of the classification loss.

[0011] Step 4: After obtaining the updated gradients of the network parameters in the cross-domain meta-test phase, the classification algorithm enters the outer loop. The outer loop updates the meta-network parameters on the graphics card using the update direction obtained in the inner loop, enabling the meta-network to gain domain adaptability through the inner loop.

[0012] Step 5: After each update of the meta-network parameters, the algorithm updates the twin network by using the exponential moving average method for the parameters of the meta-network on the graphics card, and enters the next inner loop meta-training phase of the outer loop nesting.

[0013] Preferably, the meta-network parameter θ in step 1 is m Siamese network parameter θ s ,Each step of the inner loop is to update the Siamese network by learning the domain adaptation task, and the initial Siamese network is a copy of the meta-network.

[0014] Preferably, the twin network and meta-network in step 1 have the same structure, and are both composed of a feature extractor and a classification head; wherein the feature extractor is composed of five residual convolution blocks, wherein each residual convolution block is composed of three 3*3 convolution layers and a residual connection with a 1*1 convolution layer connected to the input, as well as an activation function and a maximum pooling layer; the classification head is composed of multiple fully connected layers, which reduce the feature dimension to the category number dimension, thereby obtaining the classification score of the image.

[0015] Preferably, in the meta-training phase described in step 2, each update of the twin network is performed with the help of a small sample learning task, and in the domain adaptation task DA i In the meta-training phase, Small sample classification task To update the twin network parameters A copy of in From the source domain Few-shot learning tasks with mid-sampling.

[0016] Preferably, the updating step in step 4 is expressed by the following formula:

[0017]

[0018] Where α represents the cross-domain learning rate, φ N represents the updated twin network parameters after learning the Nth small sample classification task, Indicates the network parameter φ j-1 Learning small sample tasks based on The classification loss obtained, where α is a known quantity, φ N 、 φ j-1 As an unknown quantity, we let the model learn small-sample classification tasks in different domains during the meta-training phase, so that it can quickly adapt to small-sample classification tasks in different domains.

[0019] Preferably, the small sample task in the meta-test domain described in step 3 needs to test φ after the meta-training phase is completed. N Whether it has the desired domain adaptation ability, in the meta-testing phase, the data domain that has not been seen in the meta-training phase is selected as the query domain in the multi-source domain. And sample a small sample of tasks from the query domain To test the twin network parameters φ obtained by meta-training N , through meta-test, the updated gradient g is obtained by back-propagating the calculated classification loss i , the calculation formula is as follows:

[0020]

[0021] in Indicates the network parameter φ N By learning small sample tasks based on The classification loss obtained, where φ N 、 is a known quantity, is an unknown quantity.

[0022] Preferably, the network parameter update gradient obtained in the cross-domain meta-test phase in step 4 is used to update the meta-network parameters so that the meta-network parameters have domain adaptability. After updating the meta-network parameters, the classification method enters the next outer loop, in which the twin network parameters are updated. The calculation formula is as follows:

[0023]

[0024] Where β represents the domain adaptation learning rate. When the meta-test phase is over, the learning of a domain adaptation task is completed. N ,β,g i is a known quantity, is an unknown quantity;

[0025] The outer loop takes the domain adaptation task as the basic learning unit. Each step of the inner loop is to learn the domain adaptation task and transfer the domain generalization ability to the meta-network. After learning the domain adaptation task, it is necessary to transfer the domain generalization ability of the twin network to the meta-network and calculate the update direction d of the network parameters by updating the twin network parameters. i , the formula is as follows:

[0026]

[0027] The left side of the equation contains unknown quantities, and the right side of the equation contains unknown quantities;

[0028] Update the meta-network parameters according to the update direction The calculation formula is as follows:

[0029]

[0030] Where γ represents the meta-learning rate, which is a hyperparameter for updating the meta-network. The left side of the equation is the unknown quantity, and the right side of the equation is the known quantity.

[0031] Preferably, in step 5, the twin network parameters are updated by exponential moving average, and the twin network θ s and meta-network θ m Shared parameters, before entering the next outer loop, update the twin network parameters by exponential moving average. The formula is as follows:

[0032]

[0033] Where m represents the weighted weight of the moving average, the left side of the equation is the unknown quantity, and the right side of the equation is the known quantity. When the twin network parameters are updated, it enters the next domain adaptation inner loop phase.

[0034] Beneficial effects of the present invention: The present invention proposes to learn the real domain offset on multi-source domain data, and proposes a domain-independent meta-training classification method based on the multi-source domain, taking the domain adaptation task as the basic training unit, so that the model can learn the ability to adapt to the unknown target domain. The domain-independent meta-training classification method is mainly divided into two layers of loops: In the inner loop, we focus on updating the twin network of the meta-network. Each step of the inner loop is to learn small sample tasks collected from different source domains, thereby updating the parameters of the twin network. In order to test the domain adaptability of the parameters obtained by the inner loop, we test pseudo-target domain data on the twin network; the entire inner loop is defined by us as a domain adaptation task, so each step of the outer loop is to update the parameters of the meta-network through the update direction obtained by the inner loop. In this process, although the model parameters are not the optimal solution for the source domain, they have the ability to quickly adapt to unseen target domains. The present invention uses a cross-domain meta-training classification method based on an inner and outer two-layer loop to update network parameters. By learning domain adaptation tasks through the inner loop, the network has domain generalization capabilities, avoiding overfitting of parameters to source domain data and falling into the trap of local optimal solutions. Moreover, through cross-domain meta-training under multiple source domains, the model can learn the real domain offset, thereby alleviating the adverse effects of source domain offset on model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the overall training flow chart of the network;

[0036] Figure 2 This is a schematic diagram of the backbone network structure;

[0037] Figure 3 Schematic diagram of the residual block structure of the backbone network. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0039] The domain-independent meta-training small sample target classification method of this embodiment is Figure 1 The first step is to set the initial parameters of the meta-network when entering the outer loop for the first time. Passed to the Siamese network θ s , the structure of the feature extractor is as follows Figure 2 As shown in the figure, the network is divided into two parts: feature extractor and classifier. The feature extractor consists of five residual convolution blocks with increasing channels, and the classifier consists of an activation function, an average pooling layer, and a 512*k-shot fully connected layer. The specific structure of the residual connection block is as follows: Figure 3As shown in the figure, the input features are convolved through multiple 3*3 convolutional layers, batch normalization layers and ReLU activation functions, and the input feature map is concatenated with the output feature map. The feature map is then output through the activation function and the maximum pooling layer. In the second step, the twin network will learn the domain adaptation task TD, as shown in the figure. Figure 1 As shown, the inner update loop begins. Using the AID, PatternNet, and NWPU-RESISC45 remote sensing datasets as source domains, the network samples multiple small-sample learning tasks to update the twin network parameters. Specifically, the training method for updating the twin network using small-sample tasks follows the small-sample learning method proposed by Finn et al. in "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks." In the third step, after learning small-sample classification tasks from different source domains, the domain generalization ability of the twin network parameters is evaluated using small-sample classification tasks sampled from the pseudo-test domain, the UCMerced Land Use Dataset. Using the test domain small-sample classification tasks, the network update direction is determined. In the fourth step, the meta-network parameters are updated using the update direction obtained from the inner loop, hoping that the meta-network parameters will be able to adapt to the domain. The updated meta-network parameters are then transferred to the twin network parameters using an exponential moving average, and the network returns to the first step to prepare for the next outer loop. In the second step, the network parameters are updated using the following formula (1). In the third step, the update direction of the meta-network is calculated using the following formula (2). The loss used in the formula is the cross-entropy loss used in the classification task. After multiple outer loop updates, the meta-network will learn a non-optimal solution parameter. This parameter has the ability to adapt to the domain through learning domain adaptation tasks in multiple source domains. The network's domain generalization ability is tested on the RSI-CB256 / 128 dataset.

[0040] The domain-independent meta-training small-sample target classification method of this embodiment uses the domain adaptation task as the basic learning unit. Through internal and external loop training, the model acquires the ability to adapt to unseen target domains. The specific steps are as follows:

[0041] The entire training process is mainly divided into two stages: domain adaptation inner loop and cross-domain outer loop;

[0042] (1) Domain Adaptation Internal Circulation Stage

[0043] Each step of the inner loop updates the Siamese network by learning the domain adaptation task. The initial Siamese network is a copy of the meta-network. Specifically, the Siamese network and the meta-network have the same structure, both consisting of a feature extractor and a classification head. The feature extractor consists of five residual convolution blocks, each of which consists of three 3*3 convolution layers and a residual connection with a 1*1 convolution layer connected to the input, as well as an activation function and a maximum pooling layer. The classification head consists of multiple fully connected layers, which reduce the feature dimension to the number of categories, thereby obtaining the classification score of the image. The domain adaptation task DA defined by our algorithm is divided into meta-training and meta-testing stages.

[0044] Specifically, in the meta-training phase, each update of the Siamese network is performed with the help of a small sample learning task. Therefore, as a plug-and-play meta-training classification method, we can use the meta-learning task proposed by Finn et al. in the paper "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" as the small sample learning task in the training phase. i In the meta-training phase, we learn from multiple source domains Small sample classification task To update the twin network parameters A copy of in From the source domain The update step of meta-training can be expressed as follows:

[0045]

[0046] Where α represents the cross-domain learning rate, φ N represents the updated twin network parameters after learning the Nth small sample classification task, Indicates the network parameter φ j-1 Learning small sample tasks based on The classification loss obtained. In the meta-training phase, we let the model learn small sample classification tasks in different domains, hoping that it can quickly adapt to small sample classification tasks in different domains. Therefore, after the meta-training phase, it is necessary to test φ N Whether it has the domain adaptation ability we expect. Therefore, in the meta-testing phase, we select the data domain that has not been seen in the meta-training phase as the query domain in the multi-source domain. And sample a small sample of tasks from the query domain To test the twin network parameters φ obtained by meta-training N Through meta-testing, the updated gradient g can be obtained by back-propagating the calculated classification loss i , the calculation formula is as follows:

[0047]

[0048] in Indicates the network parameter φ N By learning small sample tasks based on The classification loss is obtained. Then we update the twin network parameters The calculation formula is as follows:

[0049]

[0050] Where β represents the domain adaptation learning rate. When the meta-test phase ends, the learning of a domain adaptation task ends.

[0051] (2) Domain Adaptation External Loop

[0052] The outer loop uses the domain adaptation task as the basic learning unit, and each step of the inner loop is to learn the domain adaptation task and transfer the domain generalization ability to the meta-network. Specifically, after learning the domain adaptation task, it is necessary to transfer the domain generalization ability of the twin network to the meta-network. Therefore, by updating the twin network parameters, we can calculate the update direction d of the network parameters. i , the formula is as follows:

[0053]

[0054] Then update the meta-network parameters according to the update direction The calculation formula is as follows:

[0055]

[0056] Where γ represents the meta-learning rate, which is a hyperparameter for updating the meta-network. Each update of the meta-network parameters represents the end of an outer loop. In addition, because the twin network θ s and meta-network θ m Shared parameters, so before entering the next outer loop, the twin network parameters are updated by exponential moving average. The formula is as follows:

[0057]

[0058] Where m represents the weight of the moving average. Once the Siamese network parameters are updated, it can enter the next domain adaptation inner loop phase.

[0059] The domain-independent meta-training small-sample target classification method of this embodiment adapts to small-sample learning tasks in different source domains through an inner loop, tests the domain adaptability of the parameters, and updates the meta-network parameters in the outer loop using the update direction obtained from the inner loop domain adaptation task, thereby transferring the domain adaptability to the meta-network. Through training with this classification method, the model parameters will not reach the optimal solution, and the model will not overfit the source domain data. Instead, the model will acquire domain generalization capabilities through each learning domain adaptation task.

Claims

1. Domain-independent meta-training for small-sample target classification, characterized by The following steps are involved: Step 1: Load the meta-network and twin network structures and initial network parameters into the graphics card, where the meta-network and twin network have the same network structure and initial parameters. In subsequent steps, the meta-network and twin network share network parameters, where the meta-network parameters are θ m , the twin network parameters are θ s ,The network structure consists of residual convolution blocks and fully connected layers.,After the graphics card loads the initial values ​​of the network parameters,,it enters the meta-training inner loop step; Step 2: In the meta-training phase of the inner loop, small-sample classification tasks are sampled from multiple source domains as the basic learning unit to train the model. The support set of each small-sample classification task contains N categories, and each category contains K images. The Ubuntu-based server loads the images into the classification model on the graphics card for calculation and updates the Siamese network parameters through backpropagation of the loss. Step 3: Through learning of small sample tasks in multiple different source domains, the updated twin network parameters are obtained. The classification method enters the cross-domain meta-testing phase. The server samples small sample tasks in the meta-testing domain and inputs the small sample task data into the twin network in the graphics card to calculate the classification results. The updated gradient of the network parameters is obtained through the back propagation of the classification loss. Step 4: After obtaining the updated gradients of the network parameters in the cross-domain meta-test phase, the classification algorithm enters the outer loop. The outer loop updates the meta-network parameters on the graphics card using the update direction obtained in the inner loop, enabling the meta-network to gain domain adaptability through the inner loop. Step 5: After each update of the meta-network parameters, the algorithm updates the twin network by using the exponential moving average method for the parameters of the meta-network on the graphics card, and enters the next inner loop meta-training phase of the outer loop nesting.

2. The domain-independent meta-training small sample target classification method according to claim 1 is characterized in that The meta-network parameters θ described in step 1 m Siamese network parameter θ s ,Each step of the inner loop is to update the Siamese network by learning the domain adaptation task, and the initial Siamese network is a copy of the meta-network.

3. The domain-independent meta-training small sample target classification method according to claim 2 is characterized in that The twin network and meta-network described in step 1 have the same structure, both consisting of a feature extractor and a classification head; the feature extractor consists of five residual convolution blocks, each of which consists of three 3*3 convolution layers and a residual connection with a 1*1 convolution layer connected to the input, as well as an activation function and a maximum pooling layer; the classification head consists of multiple fully connected layers, which reduce the feature dimension to the number of categories, thereby obtaining the classification score of the image.

4. The domain-independent meta-training small sample target classification method according to claim 1 is characterized in that In the meta-training phase described in step 2, each update of the twin network is performed with the help of a small sample learning task. i In the meta-training phase, Small sample classification task To update the twin network parameters A copy of in From the source domain Few-shot learning tasks with mid-sampling.

5. The domain-independent meta-training small sample target classification method according to claim 1 is characterized in that The update step described in step 4 is expressed by the following formula: Where α represents the cross-domain learning rate, φ N represents the updated twin network parameters after learning the Nth small sample classification task, Indicates the network parameter φ j-1 Learning small sample tasks based on The classification loss obtained, where α is a known quantity, φ N 、 φ j-1 As an unknown quantity, we let the model learn small-sample classification tasks in different domains during the meta-training phase, so that it can quickly adapt to small-sample classification tasks in different domains.

6. The domain-independent meta-training small sample target classification method according to claim 1 is characterized in that The small sample task in the meta-test domain described in step 3 requires testing φ after the meta-training phase. N Whether it has the desired domain adaptation ability, in the meta-testing phase, the data domain that has not been seen in the meta-training phase is selected as the query domain in the multi-source domain. And sample a small sample of tasks from the query domain To test the twin network parameters φ obtained by meta-training N , through meta-test, the updated gradient g is obtained by back-propagating the calculated classification loss i , the calculation formula is as follows: in Indicates the network parameter φ N By learning small sample tasks based on The classification loss obtained, where φ N 、 is a known quantity, is an unknown quantity.

7. The domain-independent meta-training small sample target classification method according to claim 1 is characterized in that The network parameter update gradient obtained in the cross-domain meta-test phase in step 4 is used to update the meta-network parameters so that the meta-network parameters have domain adaptability. After updating the meta-network parameters, the classification method enters the next outer loop, in which the twin network parameters are updated. The calculation formula is as follows: Where β represents the domain adaptation learning rate. When the meta-test phase is over, the learning of a domain adaptation task is completed. N ,β,g i is a known quantity, is an unknown quantity; The outer loop takes the domain adaptation task as the basic learning unit. Each step of the inner loop is to learn the domain adaptation task and transfer the domain generalization ability to the meta-network. After learning the domain adaptation task, it is necessary to transfer the domain generalization ability of the twin network to the meta-network and calculate the update direction d of the network parameters by updating the twin network parameters. i , the formula is as follows: The left side of the equation contains unknown quantities, and the right side of the equation contains unknown quantities; Update the meta-network parameters according to the update direction The calculation formula is as follows: Where γ represents the meta-learning rate, which is a hyperparameter for updating the meta-network. The left side of the equation is the unknown quantity, and the right side of the equation is the known quantity.

8. The domain-independent meta-training small sample target classification method according to claim 1 is characterized in that As described in step 5, the twin network parameters are updated by exponential moving average, and the twin network θ s and meta-network θ m Shared parameters, before entering the next outer loop, update the twin network parameters by exponential moving average. The formula is as follows: Where m represents the weighted weight of the moving average, the left side of the equation is the unknown quantity, and the right side of the equation is the known quantity. When the twin network parameters are updated, it enters the next domain adaptation inner loop phase.

Images