A multi-scale metric few-shot learning method based on class features

Abstract

The present invention relates to a multi-scale metric few-sample learning method based on class features, comprising: S1, data preprocessing step; S2, feature embedding step; S3, class feature extraction step: fusing multiple sample features of the same type in a support set through a dynamic routing mechanism, and updating the weight vector of the input vector in an iterative manner to obtain the class overall feature; S4, multi-scale measurement step: measuring the similarity between the support set class features and the query set samples by fusing three measurement criteria. The present invention adopts a dynamic routing mechanism to generate class overall features. Compared with the algorithm of direct weighted average, the class overall features obtained by the algorithm are more representative. In the measurement module, an attention mechanism is introduced into the measurement method of the parameterized network, and the advantages and disadvantages of multiple measurement methods are combined to jointly determine the similarity between sample features, thereby obtaining a CFMMN network model with better expressiveness.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-scale metric few-sample learning method based on class features. Background Art

[0002] For learning tasks in different fields, a large number of labeled data samples need to be obtained according to the specific requirements of the tasks, but this is very difficult and requires huge manpower and financial resources. Therefore, the lack of sample set data resources is a difficult problem that needs to be solved in deep learning. Humans have the ability to learn effectively from very small samples. Inspired by this, the concept of few-sample learning came into being. Few-sample learning attempts to quickly summarize limited supervised experience by transforming prior knowledge, and imitates the ability of humans to acquire knowledge from a few examples by analogy.

[0003] Among the existing few-shot learning algorithms, few-shot learning models are divided into three major categories: methods based on data enhancement, methods based on model optimization, and methods based on algorithm optimization. Few-shot learning algorithms based on data enhancement are a type of solution that is directly derived from the analysis of practical problems. They attempt to expand the dataset of currently missing data resources, thereby providing rich supervised information. The main idea of ​​the few-shot learning algorithm model based on model optimization is to further narrow the optimization space of parameters after fully mining the high-level semantic information of limited supervised sample resources, thereby reducing the difficulty of optimizing model parameters. Metric-based few-shot methods not only focus on how to get more raw data, but also start from the perspective of how to make better use of limited sample data, cleverly transforming the few-shot problem into finding a more accurate embedding expression and a better distance measurement strategy. Few-shot learning methods based on algorithm optimization attempt to explore more appropriate search strategies in the hypothesis space. Many models in the algorithm optimization-based methods choose to explore a better initialization parameter model, thereby obtaining better model results with fewer iterations.

[0004] In the existing model, for N-way K-shot (K>1) few-sample tasks, the overall class features of the category in high-dimensional space are obtained by simply weighting or averaging the feature maps of samples of the same category. The sample features are obtained through the previous convolutional neural network training, but due to the spatial limitations of the convolutional neural network, the weight information of a certain position on the feature map is only obtained from the relevant adjacent area mapped to the original image position, and cannot well integrate all the information of the entire image. Therefore, when there is a large difference in the position information of different samples of the same class, a large amount of noise information will be introduced to eliminate the original effective information, resulting in inaccurate final image classification results. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-scale metric few-sample learning method based on class features, which solves the problem of inaccurate class feature extraction in the existing few-sample learning.

[0006] The object of the present invention is achieved by the following technical solution: a multi-scale metric few-sample learning method based on class features, the few-sample learning method comprising:

[0007] S1, data preprocessing step: enhance the data by randomly fixing the angle to expand the data volume and add image samples of different angles for the same class, and obtain the support set and query set by the N-way K-shot method;

[0008] S2. Feature embedding step: through feature embedding network For samples x in the support set and query set i After embedding, the features are obtained

[0009] S3, class feature extraction step: fuse multiple sample features of the same type in the support set through a dynamic routing mechanism, and update the weight vector of the input vector in an iterative manner to obtain the overall class feature;

[0010] S4, multi-scale measurement step: The similarity between the support set class features and the query set samples is measured by integrating the three measurement criteria of parameterized network measurement, cosine distance measurement and Euclidean distance measurement.

[0011] The method of obtaining the support set and the query set by the N-way K-shot method includes:

[0012] Randomly extract N classes from the data set, extract k samples from each class as the support set, and the samples in the support set are used to generate prototypes of N classes;

[0013] Then, k samples are extracted from the remaining samples of each N class as the query set, and the query set is used to calculate the accuracy of the network and the performance of the model.

[0014] The specific contents of the class feature extraction step include:

[0015] The support set sample feature vector e obtained in the feature embedding step ij Transform to get Among them, W s 、b s To transform the matrix and bias term, the Squash function is a nonlinear function that compresses the vector so that its length is between 0 and 1 to normalize the length of the vector;

[0016] By iteratively The weight vector is updated to obtain the overall characteristics of the class.

[0017] The specific iteration process includes:

[0018] d ij =softmax(b i )

[0019]

[0020]

[0021]

[0022] Among them, d ij Represents the input vector and the output class feature c i The relationship between ij The initial value of is 0, and after the Softmax function, it becomes uniformly distributed. i is the class feature of the output i-th support set sample.

[0023] The multi-scale measurement step specifically includes:

[0024] According to the feature embedding step, the query set sample feature e is obtained. q And the class feature extraction step obtains the class feature c of the i-th class support set i , the matching score between the i-th support set sample and the q-th query sample is obtained through the Euclidean distance for

[0025] And get the cosine similarity method as the matching score of the metric

[0026] When the metric is a parametric network with an attention mechanism, the specific parameters in the network are obtained through optimization learning, and then the matching score is obtained. Where C(.,.) is the concatenation function, M Attention (.) represents the metric with attention mechanism, f φ represents a fully connected network with an activation function;

[0027] Select the category i with the largest matching score summed up by the three metrics as the query sample x q Category label

[0028] The few-sample learning method further includes a step of setting a loss function; the loss function is a loss function with a spacing, and its calculation formula is: Among them, m + represents the interval, α represents the weight coefficient, 1iq represents the indicator function, r iq represents the matching score between the query sample and the i-th support set sample;

[0029] The loss function calculation formula represents the result of the mutual constraints between the query sample and all other class-level features. The inward pull between samples of the same type and the outward push between samples of different types are It represents the pulling force between the query sample q and the i-th supporting feature. The optimization goal is to reduce the distance between samples of the same type. The minimum distance between non-similar samples cannot be less than the threshold m. + .

[0030] The present invention has the following advantages: a multi-scale metric few-sample learning method based on class features, inspired by the previous metric-based few-sample learning, focuses on the N-way K-shot (K>1) few-sample classification task, and uses a dynamic routing mechanism to generate class overall features. Compared with the direct weighted average algorithm, the class overall features obtained by this algorithm are more representative. In the measurement module, an attention mechanism is introduced into the measurement method of the parameterized network, and the advantages and disadvantages of multiple measurement methods are combined to jointly determine the similarity between sample features, thereby obtaining a CFMMN network model with better expressiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the flow chart of the multi-scale metric few-shot learning algorithm based on class features;

[0032] Figure 2 CFMMN network model for multi-scale metric few-shot learning based on class features;

[0033] Figure 3 It is the network structure of feature embedding module;

[0034] Figure 4 It is a parameterized metric network structure;

[0035] Figure 5 This is an example of a few-shot 5-way 1-shot task.

[0036] Figure 6 This is a comparison chart of the experimental results of the Omniglot dataset;

[0037] Figure 7 This is a comparison chart of the experimental results of the mini-ImageNet dataset. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided below in conjunction with the drawings is not intended to limit the scope of protection of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application. The present invention is further described below in conjunction with the drawings.

[0039] The work of this invention mainly focuses on the N-way K-shot (K>1) few-shot classification task in the few-shot learning algorithm, as well as the core problem of class feature selection and extraction. In addition, the advantages and disadvantages of multiple measurement methods are combined to generate a multi-scale measurement method. Finally, two widely used datasets in the field of few-shot learning are selected: the Omniglot dataset and the mini-ImageNet dataset, and the algorithm in this invention is used to perform a few-shot image classification test. Figure 1 The flowchart of the multi-scale metric few-shot learning algorithm based on class features is shown in Figure 2 The network model of the algorithm is shown in Figure 2. The main steps of this method are as follows:

[0040] Step 1: Data preprocessing. Due to the particularity of the lack of data resources for few-shot learning, the simplest random rotation of 90°, 180° and 270° is first used to enhance the data. On the one hand, it can expand the amount of data, and on the other hand, adding image samples of different angles for the same class is conducive to testing the effectiveness of model class feature extraction. In addition, the training iteration of the few-shot image classification model is actually completed through several "tasks", so it is necessary to first generate the image data task and then input it into the model. This training method is also called N-way, K-shot method. First, N classes are randomly selected from the data set, and k samples are selected from each class as the support set. The samples in the support set are used to generate prototypes of N classes, and then k samples are selected from each class of the remaining samples of the N classes as the query set. The query set is used to calculate the accuracy of the network to verify the performance of the model. Each task contains a small number of classes, and each class contains a small number of samples. Such a task setting simulates the scenario of few-shot image classification.

[0041] Step 2: Feature embedding. For a given support set with N classes and each class having k samples and queryset Sample x iAfter feature embedding network Then we get the feature h i :

[0042]

[0043] Among them, the embedded module specific network Structure Figure 3 As shown in the figure, the embedding module of the relational network consists of four convolutional blocks, each of which contains a convolutional layer with a convolution kernel of 64*3*3, a batch normalization layer, and a ReLU layer. The first and second convolutional blocks are followed by a 2*2 maximum pooling layer to adjust the network specifications, while the next two convolutional blocks do not have a pooling layer.

[0044] Step 3: Class feature extraction. After completing feature extraction of the support set S and query set Q, a dynamic routing mechanism is used to fuse multiple sample features of the same type in the support set. In order to make the model adaptable to task inputs of more forms of support set samples, the support set sample feature vector e obtained after step 2 is ij First make the following transformation:

[0045]

[0046] Among them, W s 、b s To transform the matrix and bias term, the Squash function behaves similarly to sigmoid, which is a nonlinear function that compresses the vector so that its length is between 0 and 1 to normalize the length of the vector. For any vector s i , the calculation of the Squash function is as follows:

[0047]

[0048] By iteratively The weight vector is updated to obtain the overall characteristics of the class. The specific iterative process is:

[0049] d ij =softmax(b i )

[0050]

[0051]

[0052]

[0053] Among them, d ij Represents the input vector and the output class feature c i The relationship between ijThe initial value of is 0, and after the Softmax function, it becomes uniformly distributed. i is the class feature of the output support set sample of the i-th category. If the current sample feature belongs to a certain category, the similarity will be higher and the weight will increase in the next iteration. If not, the weight vector should be smaller. In general, after multiple iterations, the contribution of each sample under the same class will become different after learning. After the iteration round is over, the class-level features will be obtained, which can usually be completed in 3 rounds.

[0054] Step 4: Multi-scale measurement. After obtaining the class features of N categories in the support set, it is necessary to take appropriate measures to measure the similarity between the class features of the support set and the query sample. Generally speaking, there are two methods for measuring feature similarity: cosine distance and Euclidean distance. Figure 2 As shown in the network model, this algorithm will integrate three measurement criteria: parametric network measurement, cosine distance measurement and Euclidean distance measurement, specifically:

[0055] After the pre-feature embedding network, the query sample feature is obtained as e q And the class feature c of the i-th class support set is obtained through the dynamic routing module i , then the matching score between the i-th support set sample and the q-th query sample is obtained through the Euclidean distance for:

[0056]

[0057] If the cosine similarity method is used as the metric, the matching score for:

[0058]

[0059] like Figure 4 As shown in the figure, when the metric is a parametric network with an attention mechanism, it is necessary to obtain the specific parameters in the network through optimization learning, and the final matching score is:

[0060]

[0061] Where C(.,.) is the concatenation function, M Attention (.) represents the metric with attention mechanism, f φ Represents a fully connected network with an activation function. Specifically, the attention mechanism is to generate three new feature maps A, B, and C after the concatenated feature matrix P passes through three 1x1 convolution kernels. The calculation method after the attention layer is as follows:

[0062]

[0063] H(A, B) = soft max(ATB)

[0064] The above formula obtains the feature map P with attention weights through the residual neutralization idea AttentionOut ,Introducing attention in the network can not only comprehensively examine the ,features of each class in the support set, but also find the more ,targeted parts between the class features and the query features for ,metric learning.

[0065] The matching score is obtained by combining the above three measurement methods The final result of the few-sample task classification is jointly determined, and the category i with the largest matching score summed by these three metrics is selected as the query sample x q Category labels:

[0066]

[0067] Step 5: Loss function design. In the present invention, the loss function is used as a learning criterion and is associated with the optimization problem, that is, the model is evaluated by minimizing the loss function. The present invention specifically designs a loss function calculation method with intervals for the few-sample learning scenario of CFMMN:

[0068]

[0069] Among them, m + represents the interval, α represents the weight coefficient, 1 iq represents the indicator function, r iq Represents the matching score between the query sample and the i-th class support set sample. The above formula represents the result of the mutual constraints between the query sample and all other class-level features, which generates an inward pull between samples of the same class and an outward push between samples of different classes. The first term represents the pull between the query sample q and the i-th class support class feature. The optimization goal is to try to reduce the distance between samples of the same class; the second term in the formula constrains the minimum distance between samples of different classes to be no less than the threshold m + .

[0070] The multi-scale metric few-shot learning network model based on class features in the present invention is trained and tested in task mode, so it is necessary to sample the original data set D to construct tasks. First, the original data set D is divided into a training data set and a test data set, which correspond to the training and testing stages of few-shot learning, respectively. Multiple tasks are generated by randomly sampling the training data set and the test data set, where a single task contains two parts: a support sample set and a query sample set. The query sample label of the task must be included in the support sample label. That is to say, the purpose of the model is to determine which category label in the support set the query set sample belongs to in the test task after learning a large number of training tasks. If the support sample set of a task has N classes, and each class has K samples, such a task is called an N-way K-shot task, as follows Figure 5 Shown is a typical 5-way 1-shot mission.

[0071] In order to evaluate the performance of the multi-scale metric few-shot learning CFMMN network model based on class features in the present invention, 5-way 1-shot, 5-way 5-shot, 20-way 1-shot and 20-way 5-shot experiments were conducted on the Omniglot and mini-ImageNet datasets, and compared and analyzed with other algorithms. The present invention adopts a training set: test set = 8:2 mode, and the model evaluation standard is the accuracy Acc of querying sample labels on the test set. The baselines of the MN, PN and RN network models listed below are the same as those in this article.

[0072] The experimental results of the CFMMN network model on the Omniglot dataset are as follows: Figure 6 As shown: the accuracy of the 5-way 1-shot task reached 99.34%±0.27%, and the accuracy of the 5-way 5-shot task reached 99.55%±0.19%, which were 1.74% and 1.25% higher than those of MN, 2.04% and 0.65% higher than those of PN, and 0.44% and 0.51% higher than those of RN; the accuracy of the 20-way 1-shot task was improved by 1.82% based on RN, and the accuracy of the 20-way 5-shot task was improved by 0.51% based on RN.

[0073] The experimental results of the CFMMN network model on the mini-ImageNet dataset are as follows Figure 7 As shown in the figure, the classification accuracy of the 5-way 1-shot task and the 5-way 5-shot task is improved by 5.35% and 6.74% compared with RN.

[0074] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-scale metric few-shot learning method based on class features, characterized by: The few-sample learning method comprises: S1, data preprocessing step: enhance the data by randomly fixing the angle to expand the data volume and add image samples of different angles for the same class, and obtain the support set and query set by the N-way K-shot method; S2. Feature embedding step: through feature embedding network For samples x in the support set and query set i After embedding, the features are obtained S3, class feature extraction step: fuse multiple sample features of the same type in the support set through a dynamic routing mechanism, and update the weight vector of the input vector in an iterative manner to obtain the overall class feature; S4, multi-scale measurement step: the similarity between the support set features and the query set samples is measured by integrating the three measurement criteria of parameterized network measurement, cosine distance measurement and Euclidean distance measurement; The support set and query set obtained by the N-way K-shot method include: Randomly extract N classes from the data set, extract k samples from each class as the support set, and the samples in the support set are used to generate prototypes of N classes; Then, k samples are extracted from the remaining samples of each class as the query set. The query set is used to calculate the accuracy of the network to verify the model performance. The specific contents of the class feature extraction step include: The support set sample feature vector e obtained in the feature embedding step ij Transform to get Among them, W s 、b s To transform the matrix and bias term, the Squash function is a nonlinear function that compresses the vector so that its length is between 0 and 1 to normalize the length of the vector; By iteratively The weight vector is updated to obtain the overall characteristics of the class; The specific iteration process includes: Among them, d ij Represents the input vector and the output class feature c i The relationship between ij The initial value of is 0, and after the Softmax function, it becomes uniformly distributed. i is the class feature of the output i-th class support set sample; The multi-scale measurement step specifically includes: According to the feature embedding step, the query set sample feature e is obtained. q And the class feature extraction step obtains the class feature c of the i-th class support set i , the matching score between the i-th support set sample and the q-th query sample is obtained through the Euclidean distance for And get the cosine similarity method as the matching score of the metric When the metric is a parametric network with an attention mechanism, the specific parameters in the network are obtained through optimization learning, and then the matching score is obtained. Where C(.,.) is the concatenation function, M Attention (.) represents the metric with attention mechanism, f φ represents a fully connected network with an activation function; Select the category i with the largest matching score summed up by the three metrics as the query sample x q Category label .

2. The multi-scale metric few-sample learning method based on class features according to claim 1, characterized in that: The few-sample learning method further includes a step of setting a loss function; the loss function is a loss function with a spacing, and its calculation formula is: Among them, m + represents the interval, α represents the weight coefficient, 1 iq represents the indicator function, r iq represents the matching score between the query sample and the i-th support set sample; The loss function calculation formula represents the result of the mutual constraints between the query sample and all other class-level features. The inward pull between samples of the same type and the outward push between samples of different types are It represents the pulling force between the query sample q and the first supporting feature. The optimization goal is to reduce the distance between samples of the same type. The minimum distance between non-similar samples cannot be less than the threshold m. + .

Images