A small sample learning method and device based on cross-sample attention aggregation

Abstract

The present invention discloses a small sample learning method based on cross-sample attention aggregation, comprising: inputting a support set into a convolutional neural network and then compressing it into a matrix form to obtain a multi-category feature vector, inputting the multi-category feature vector into an attention aggregation module to obtain a plurality of category aggregation vectors of the support set; inputting a query set into a convolutional neural network and then compressing it into a matrix form to obtain a query feature vector, inputting the query feature vector into an attention aggregation module to obtain a query aggregation vector; obtaining a category prediction probability value of a query image through a distance-aware probability activation method based on the category aggregation vector, the query aggregation vector and the true category aggregation vector; constructing a cross-entropy loss function based on the category prediction probability value, and optimizing the parameters in the training model through the cross-entropy loss function to obtain a category prediction model; the method can clearly distinguish between objects and background areas, and can accurately extract category features.

Description

Technical Field

[0001] The present invention belongs to the field of small sample learning, and specifically relates to a small sample learning method and device based on cross-sample attention aggregation. Background Art

[0002] Humans can learn new concepts from very few examples and have strong generalization capabilities, which deep learning algorithms do not currently have. That is, humans can learn a new concept from one or a few examples, but standard deep learning algorithms require more examples to barely achieve the same ability. Deep learning has made great progress in image recognition, image segmentation, natural language processing and other fields. In practical applications, the large amount of training samples is often expensive, and its disadvantage of large training sample size has been criticized.

[0003] Researchers have designed various algorithms to try to achieve the same effect with a small number of training samples. In addition to the common data augmentation and transfer learning, the research on small sample algorithms has also received widespread attention. Existing research programs can be roughly divided into the following three categories based on their starting points and motivations: (1) small sample learning based on data augmentation; (2) small sample learning based on meta-learning; (3) small sample learning based on metrics. However, so far, the effects of conventional small sample learning algorithms represented by prototype networks are still limited.

[0004] Most conventional small-sample learning algorithms consider local and category instances of images to be equally important. When the sample size is large enough, the impact of feature vector outliers is not very large. However, in the case of small samples, this approach will obviously cause model training to fluctuate violently, and outliers will cause great interference to the prediction results.

[0005] Literature Snell, J., Swersky, K. & Zemel, R. Prototypical networks for few-shot learning. in Advances in Neural Information Processing Systems. 4077-4087 (2017) .The public prototype network is a small sample learning algorithm. After extracting features from an image to obtain a feature map, it will perform global average pooling or splicing on the feature map to form a vector to represent the features of the image, and then use the average of all features of the same category to represent a category "prototype". The disadvantage of this algorithm is that in the case of small samples, the background changes a lot, and the images presented by objects vary greatly depending on the angle, light, etc. It is obvious that it is easy to overemphasize background features or features of objects at extreme angles by only using average pooling or splicing, so that the effect of representing a category "prototype" through such a vector is not ideal.

[0006] Therefore, it is urgent to design a small sample learning algorithm to avoid treating all parts and category instances equally, which can effectively solve the problem of over-emphasis on background and unimportant instances, thereby better extracting category features and improving the effect of small sample learning. Summary of the invention

[0007] The present invention provides a small sample learning method based on cross-sample attention aggregation, which can clearly distinguish objects and background areas and accurately extract category features.

[0008] A small sample learning method based on cross-sample attention aggregation, comprising:

[0009] (1) dividing the original image into a training sample set and a verification sample set, dividing the training sample set into a first support set and a first query set, the first support set includes a multi-category image set, each category image set includes a plurality of images of the same category, and the first query set includes a plurality of query images of the same category;

[0010] (2) constructing a training model, the training model includes a support sub-model and a query sub-model, wherein the support sub-model includes a first convolutional neural network and a first attention aggregation module, inputting the first support set into the first convolutional neural network and compressing it into a matrix form to obtain a multi-category feature vector, and inputting the multi-category feature vectors into the first attention aggregation module respectively to obtain a plurality of category aggregation vectors of the first support set;

[0011] The query sub-model includes a second convolutional neural network and a second attention aggregation module. The query set is input into the second convolutional neural network and then compressed into a matrix form to obtain a single category feature vector. The single category feature vector is input into the second attention aggregation module to obtain a query aggregation vector of the first query set.

[0012] Based on multiple category aggregation vectors, query aggregation vectors and true category aggregation vectors, a distance-aware probability activation method is used to obtain the category prediction probability value of the query image.

[0013] (3) constructing a loss function, constructing a cross entropy loss function based on the predicted probability value of the category of a single category query image, and optimizing the parameters in the training model through the cross entropy loss function to obtain a category prediction model;

[0014] (4) When applied, the multi-category image set and the query image set are input into the category prediction model to obtain the predicted category of each query image.

[0015] The first support set is input into the first convolutional neural network and then compressed into a matrix form to obtain multiple category feature vectors, including:

[0016] Multiple category image sets in the first support set are input into the first convolutional neural network to obtain multi-category feature atlases, and each category feature atlas is compressed into a matrix form to obtain each category feature vector.

[0017] Each category feature vector is input into the first attention aggregation module to obtain each support category aggregation vector, including:

[0018] The average of each category feature vector is obtained, the average category feature vector and each category feature vector are input into the measurement function, the measurement result is normalized to obtain the weight vector, and the transposed weight vector is matrix multiplied with each category feature vector to obtain each support category aggregation vector.

[0019] Normalizing the measurement results to obtain the weight vector W is:

[0020] W = g(M) = softmax(-k*M)

[0021] Among them, g(﹒) is the normalization function, M is the measurement result, k is the hyperparameter, and softmax(﹒) is the activation function.

[0022] Based on multiple category aggregation vectors, query aggregation vectors and true category aggregation vectors, the category prediction probability value of the query image is obtained through the distance-aware probability activation method. for:

[0023]

[0024] Among them, x q is the qth query image in the first query set, y q is the true category label corresponding to the qth query image in the first query set, b(x q ) is the query aggregation vector corresponding to the qth query image in the first query set, a(y q) is the category aggregation vector in the first support set corresponding to the true category label of the qth query image in the first query set, a(k) is the category feature vector corresponding to the kth category image in the first support set, K is the number of categories, and d(﹒) is the metric function.

[0025] The category feature vector a(k) corresponding to the k-th category image in the first support set is:

[0026]

[0027] Among them, aggregation(﹒) is the aggregation function, reshape(﹒) is the matrix function, is the first convolutional neural network, is the trainable parameter in the first convolutional neural network, x i is the i-th image in the first support set, y i is the category label corresponding to the i-th image in the first support set.

[0028] The query aggregation vector b(x) corresponding to the qth query image in the first query set q )for:

[0029]

[0030] in, is the second convolutional neural network, is the trainable parameter in the second convolutional neural network.

[0031] The loss value Loss is obtained through the cross entropy loss function:

[0032]

[0033] The trainable parameters in the first and second convolutional neural networks are optimized through the back propagation algorithm so that the loss value reaches the loss threshold to complete the training of the training model and obtain the category prediction model. The parameters in the training model are the trainable parameters in the first and second convolutional neural networks.

[0034] It also includes the verification of the category prediction model, the specific steps are:

[0035] (1) dividing the verification sample set into a second support set and a second query set, inputting the second support set and the second query set into the category prediction model respectively to obtain multiple category prediction probability values ​​based on the second query image, and taking the category corresponding to the maximum category prediction probability value as the predicted category of the second query image;

[0036] (2) After repeating step (1) for a predetermined number of verifications, the predicted category of the second query image obtained in each verification is compared with the corresponding true category. If the comparison result reaches the accuracy threshold, the verification is completed. If the accuracy threshold is not reached, the parameters in the training model are continuously optimized.

[0037] A small sample learning device based on cross-sample attention aggregation, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the above-mentioned category prediction model is used in the computer memory;

[0038] When the computer processor executes the computer program, the following steps are implemented:

[0039] The multi-category image set and the query image set are input into the category prediction model to obtain the predicted category of each query image.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention uses an attention aggregation module to average and measure the input feature vectors, and performs matrix multiplication on the probabilistic weight vector of the measurement result and the feature vector to obtain an aggregation vector, thereby avoiding the influence of outliers and accurately distinguishing the object area and background area of ​​the image.

[0042] (2) The present invention divides the original image set into a support set and a query set, divides the support set into different categories, obtains different category aggregation vectors through the attention aggregation module, and trains the different category aggregation vectors and the query aggregation vector through the loss function to obtain the category that can accurately predict the query image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A block diagram of a small sample learning method based on cross-sample attention aggregation provided for a specific implementation method;

[0044] Figure 2 A schematic diagram of an attention aggregation module provided for a specific implementation;

[0045] Figure 3 A schematic diagram of aggregation using an attention aggregation module provided for a specific implementation;

[0046] Figure 4 A picture effect picture using an attention aggregation module is provided for a specific implementation. DETAILED DESCRIPTION

[0047] In order to make the category vector extracted by the model take into account the characteristics of different image regions and instances of different categories, we designed a small sample learning method with cross-sample attention aggregation. Compared with the traditional small sample learning algorithm, this algorithm can comprehensively consider the characteristics of different samples and different regions and aggregate more suitable category vectors.

[0048] The present invention provides a small sample learning method based on cross-sample attention aggregation, such as Figure 1 As shown, the specific steps are:

[0049] (1) The support set takes 2-way 3-shot (i.e., 2 categories, 3 images in each category) as an example, and the query set takes one image as an example.

[0050] (2) Constructing a training model, the training model includes a support sub-model and a query sub-model, wherein the support sub-model includes a first convolutional neural network and a first attention aggregation module, and the query sub-model includes a second convolutional neural network and a second attention aggregation module.

[0051] The support set is input into the first convolutional neural network to obtain two feature groups, each feature group corresponds to a category, each feature group includes three feature maps, and the dimension of each feature map is 5*5*64 (that is, the length and width of the feature map are 5); the query set is input into the second convolutional neural network to obtain a feature map with a dimension of 5*5*64, wherein the first convolutional neural network and the second convolutional neural network share parameters, that is, the parameter values ​​are the same after training, so the first convolutional neural network and the second convolutional neural network are both represented as CNN.

[0052] The feature maps of the support set and the query set are reshaped, that is, compressed into matrix form. The support set needs to be operated separately according to categories. Therefore, the feature maps of the support set are reshaped to obtain two groups of 75*64 matrices, that is, multi-category feature vectors, in which each row is a feature vector of dimension 64, with a total of 75 different vectors, which are used to represent the feature vectors of all different regions of pictures of different categories. Similarly, the pictures of the query set form a single category feature vector of dimension 64 for 25 different regions. Subsequently, the multi-category feature vector of the support set and the single category feature vector of the query set are respectively formed into two 2*64 category aggregation vectors and one 1*64 query aggregation vector through the first and second attention aggregation modules, for a total of three aggregation vectors.

[0053] Among them, the category feature vector a(k) is:

[0054]

[0055] Among them, aggregation(﹒) is the aggregation function, reshape(﹒) is the matrix function, is the first convolutional neural network, is the trainable parameter in the first convolutional neural network, x i is the i-th image in the first support set, y i is the category label corresponding to the i-th image in the first support set.

[0056] Query the aggregate vector b(x q )for:

[0057]

[0058] in, is the second convolutional neural network, is the trainable parameter in the second convolutional neural network.

[0059] Based on multiple category aggregation vectors, query aggregation vectors and true category aggregation vectors, the category prediction probability value of the query image is obtained through the distance-aware probability activation method. for:

[0060]

[0061] Among them, x q is the qth query image in the first query set, y q is the true category label corresponding to the qth query image in the first query set, b(x q ) is the query aggregation vector corresponding to the qth query image in the first query set, a(y q ) is the aggregation vector of the category in the first support set corresponding to the true category label of the qth query image in the first query set, a(k) is the category feature vector corresponding to the kth category image in the first support set, K is the number of categories, d(﹒) is the metric function, and the Euclidean distance metric function is selected in this algorithm.

[0062] (3) The cross entropy loss function is used to optimize the parameters in the training model to obtain the category prediction model; then the following loss function Loss can be used:

[0063]

[0064] The trainable parameters in the first and second convolutional neural networks are optimized by the back propagation algorithm so that the loss value reaches the loss threshold to complete the training of the training model and obtain the category prediction model. The parameters in the training model are are the trainable parameters in the first and second convolutional neural networks.

[0065] Among them, the multi-category feature vector of the support set and the single category feature vector of the query set are respectively formed into two category aggregation vectors and one query aggregation vector through the first and second attention aggregation modules, a total of three aggregation vectors. The aggregation steps of the first and second attention aggregation modules are the same, such as Figure 2 The specific steps are as follows:

[0066] For convenience, x 64-dimensional feature vectors are defined as feature vector R, where x is 75 and x is a category feature vector, and x is a query feature vector, respectively. The feature vector R is input into the attention aggregation module, and the feature vector R is averaged to obtain a 64-dimensional average vector. Then, the feature vector R and the average vector are calculated using a metric function (the metric function used in the present invention is a Euclidean distance metric function) to obtain a metric result M. The metric result M is normalized to obtain a weight vector W. The weight vector W is transposed and then matrix product operation is performed with the feature vector R to obtain a 64-dimensional aggregation vector.

[0067] Normalizing the measurement results to obtain the weight vector W is:

[0068] W = g(M) = softmax(-k*M)

[0069] Among them, g(﹒) is the normalization function, M is the measurement result, k is the hyperparameter, and softmax(﹒) is the activation function.

[0070] like Figure 3 As shown in the figure, the blue dots represent the feature vectors of different samples, and the green triangle represents the average vector of all these blue dots. This is the way to obtain the class prototype used in the prototype network. It can be seen that the introduction of outliers causes it to deviate from the center of the sample points with a larger number in the lower left corner; and the red five-pointed star is the vector aggregated according to the attention mechanism. Obviously, it is closer to the center of the sample points with a larger number in the lower left corner, indicating that it is less affected by outliers.

[0071] The above algorithm is used to train on the small sample dataset miniImagenet provided in the literature Vinyals, O., Blundell, C., Lillicrap, T. & Wierstra, D. Matching networks for one shot learning. in Advances in neural information processing systems. 3630-3638 (2016)., and the attention effect is visualized. The results are presented in Figure 4 In. From Figure 4It is found that the attention aggregation module has a good focusing effect on the objects in the image (red part), while the background part shows a lower level of attention (blue part). Therefore, the cross-sample attention aggregation algorithm described in this article can achieve the expected effect of improving the degree of attention to objects.

[0072] The original image set is divided into a training sample set and a validation sample set:

[0073] To train the training model, the specific steps are:

[0074] Extract K categories of images from the training sample set, with N samples for each category, to form the first support set S1, with a total of K*N samples; then sample T samples from each category in the remaining samples of the K categories in the training set to form the first query set Q1, with a total of K*T samples; perform back propagation training on the training model through the first support set S1 and the first query set Q1; put all samples back into the training sample set, and repeat the above steps so that the loss value reaches the loss threshold or reaches the specified number of loop executions, so as to complete the training of the training model and obtain a category prediction model;

[0075] Verify the category prediction model. The specific steps are as follows:

[0076] Extract K categories of images from the validation sample set, with N samples for each category, to form the second support set S2, with a total of K*N samples; then sample T samples from each category from the remaining samples of the K categories in the validation sample set to form the second query set Q2, with a total of K*T samples; input the second support set S2 and the second query set Q2 into the category prediction model to obtain multiple category prediction probability values ​​based on the second query image, and take the category corresponding to the maximum category prediction probability value, or directly select the category with the closest distance from the query sample aggregation vector to each category aggregation vector as the predicted category of the query sample. After multiple verifications, compare the predicted category of the second query image obtained in each verification with the corresponding true category. If the comparison result reaches the accuracy threshold, the verification is completed. If the accuracy threshold is not reached, continue to optimize the parameters in the training model.

Claims

1. A small sample learning method based on cross-sample attention aggregation, characterized in that: include: (1) dividing the original image set into a training sample set and a verification sample set, dividing the training sample set into a first support set and a first query set, the first support set includes a multi-category image set, each category image set includes a plurality of images of the same category, and the first query set includes a plurality of query images of the same category; (2) constructing a training model, the training model includes a support sub-model and a query sub-model, wherein the support sub-model includes a first convolutional neural network and a first attention aggregation module, inputting the first support set into the first convolutional neural network and compressing it into a matrix form to obtain a multi-category feature vector, and inputting the multi-category feature vectors into the first attention aggregation module respectively to obtain a plurality of category aggregation vectors of the first support set; The query sub-model includes a second convolutional neural network and a second attention aggregation module. The query set is input into the second convolutional neural network and then compressed into a matrix form to obtain a single category feature vector. The single category feature vector is input into the second attention aggregation module to obtain a query aggregation vector of the first query set. Based on multiple category aggregation vectors, query aggregation vectors and true category aggregation vectors, a distance-aware probability activation method is used to obtain the category prediction probability value of the query image. (3) constructing a loss function, constructing a cross entropy loss function based on the predicted probability value of the category of a single category query image, and optimizing the parameters in the training model through the cross entropy loss function to obtain a category prediction model; (4) When applied, the multi-category image set and the query image set are input into the category prediction model to obtain the predicted category of each query image; Each category feature vector is input into the first attention aggregation module to obtain each support category aggregation vector, including: The average of each category feature vector is obtained, the average category feature vector and each category feature vector are input into the measurement function, the measurement result is normalized to obtain the weight vector, and the transposed weight vector is matrix multiplied with each category feature vector to obtain each support category aggregation vector.

2. The small sample learning method based on cross-sample attention aggregation according to claim 1 is characterized in that: The first support set is input into the first convolutional neural network and then compressed into a matrix form to obtain multiple category feature vectors, including: Multiple category image sets in the first support set are input into the first convolutional neural network to obtain multi-category feature atlases, and each category feature atlas is compressed into a matrix form to obtain each category feature vector.

3. The small sample learning method based on cross-sample attention aggregation according to claim 1 is characterized in that: Normalizing the measurement results to obtain the weight vector W is: W = g(M) = softmax(-k*M) Among them, g(﹒) is the normalization function, M is the measurement result, k is the hyperparameter, and softmax(﹒) is the activation function.

4. The small sample learning method based on cross-sample attention aggregation according to claim 1 is characterized in that: Based on multiple category aggregation vectors, query aggregation vectors and true category aggregation vectors, the category prediction probability value of the query image is obtained through the distance-aware probability activation method. for: Among them, x q is the qth query image in the first query set, y q is the true category label corresponding to the qth query image in the first query set, b(x q ) is the query aggregation vector corresponding to the qth query image in the first query set, a(y q ) is the category aggregation vector in the first support set corresponding to the true category label of the qth query image in the first query set, a(k) is the category feature vector corresponding to the kth category image in the first support set, K is the number of categories, d(﹒) is the metric function, are the parameters in the training model.

5. The small sample learning method based on cross-sample attention aggregation according to claim 4 is characterized in that: The category feature vector a(k) is: Among them, aggregation(﹒) is the aggregation function, reshape(﹒) is the matrix function, is the first convolutional neural network, is the trainable parameter in the first convolutional neural network, x i is the i-th image in the first support set, y i is the category label corresponding to the i-th image in the first support set.

6. The small sample learning method based on cross-sample attention aggregation according to claim 5 is characterized in that: Query the aggregate vector b(x q )for: in, is the second convolutional neural network, is the trainable parameter in the second convolutional neural network.

7. The small sample learning method based on cross-sample attention aggregation according to claim 1 is characterized in that: The loss value Loss is obtained through the cross entropy loss function: The trainable parameters in the first and second convolutional neural networks are optimized by the back propagation algorithm so that the loss value reaches the loss threshold to complete the training of the training model and obtain the category prediction model. The parameters in the training model are are the trainable parameters in the first and second convolutional neural networks.

8. The small sample learning method based on cross-sample attention aggregation according to claim 1 is characterized in that: It also includes the verification of the category prediction model, the specific steps are: (1) dividing the verification sample set into a second support set and a second query set, inputting the second support set and the second query set into the category prediction model respectively to obtain multiple category prediction probability values ​​based on the second query image, and taking the category corresponding to the maximum category prediction probability value as the predicted category of the second query image; (2) After repeating step (1) for a predetermined number of verifications, the predicted category of the second query image obtained in each verification is compared with the corresponding true category. If the comparison result reaches the accuracy threshold, the verification is completed. If the accuracy threshold is not reached, the parameters in the training model are continuously optimized.

9. A small sample learning device based on cross-sample attention aggregation, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: The computer memory uses the category prediction model described in any one of claims 1 to 8; When the computer processor executes the computer program, the following steps are implemented: The multi-category image set and the query image set are input into the category prediction model to obtain the predicted category of each query image.

Images