A two-stream convolutional neural network for few-shot learning

Abstract

The present invention discloses a two-stream convolutional neural network for small sample learning, including feature vector embedding and similarity scoring based on the main network and auxiliary stream network, and further score fusion. The support set and query set are constructed using the training set of small sample image classification, and the auxiliary stream embedding network is introduced into the convolutional neural network to extract the significant features of the main target and merge them with the overall embedding features of the main network. After applying the two-stream convolutional neural network of the present invention, facing the small sample learning scenario, the rationality, stability and overall recognition of the convolutional neural network used for feature embedding are improved, thereby improving the accuracy of the training results in small sample learning.

Description

Technical Field

[0001] The present invention relates to an improved convolutional neural network, and in particular to a two-stream convolutional neural network for small sample learning. Background Art

[0002] Deep learning has achieved great success in the field of computer vision thanks to the excellent performance of convolutional neural networks. However, training convolutional neural networks requires a large amount of labeled data, and the process of obtaining labeled data is often difficult and expensive. At the same time, for deep learning, limited labeled data can easily lead to overfitting. Studies have found that the human visual system can use previous knowledge to learn a new concept through a small number of samples after a single exposure to the data. Given a limited number of labeled samples, the problem of learning unlabeled samples belonging to categories that have never been seen is called small sample learning. For example, children only need a small number of pictures in books to recognize the appearance of zebras, rhinos, etc. with high accuracy. Therefore, small sample learning hopes that after learning a large amount of data in a certain category, the model only needs a small number of samples to quickly learn for new categories.

[0003] In order to learn knowledge from limited samples of supervised information, researchers have proposed a large number of methods to improve the performance of small sample learning, as revealed in references [1, 2, 3, 4, 5, 6, 7]. The most representative small sample learning method based on metric learning represents the artifacts in a common feature space through an embedding function, and then further identifies the specific category to which the embedded vector belongs through a predefined distance metric method.

[0004] However, in the process of studying and using small sample learning techniques, it is found that the existing small sample methods are prone to produce poor embedding feature representations due to the presence of a large amount of interference information, especially cluttered backgrounds. Although this problem can be alleviated through a large number of training iterations and other regularization techniques, the failure to highlight the main target in the embedding vector will inevitably have a negative impact on the results. Figure 1As shown in the figure, the original image in the first column on the left side of the support set is three puppies on green grass, while the query image is a kitten on green grass. The classification result may be misclassified as the same category; from the foreground image in the second column on the left side, the difference between the puppies in the support set and the kitten in the query image is obvious, so the accurate classification result of "different categories" can be obtained; on the other hand, the original image in the second column on the right side of the support set is three birds under the blue sky and white clouds, while the query image is a bird landing on a dead tree, but the background is mainly green mountains and green trees, so the classification result will also be misclassified as different categories; from the foreground image in the first column on the right side, both the support set and the query image are birds, so the classification result of "same category" is obtained. It can be seen that background interference may cause samples from different (or the same) categories to be incorrectly classified as the same (or different) categories.

[0005] The bibliographic information of the references is listed below:

[0006] [1] Chelsea Finn, Pieter Abbeel, and Sergey Levine, “Model-agnosticmeta-learning for fast adaptation of deep networks,” in ICML, 2017.

[0007] [2] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and StefanoSoatto, “Meta-learning with differentiable convex optimization,” in CVPR, 2019.

[0008] [3] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in

[0009] CVPR, 2018.

[0010] [4] Oriol Vinyals, Charles Blundell, Tim Lillicrap, KorayKavukcuoglu, and Daan Wierstra, “Matching networks for one shot learning,” inNeurIPS, 2016.

[0011] [5] Jake Snell, Kevin Swersky, and Richard S. Zemel, “Prototypicalnetworks for few-shot learning,” in NeurIPS, 2017.

[0012] [6] Christian Simon, Piotr Koniusz, Richard Nock, and MehrtashHarandi, “Adaptive subspaces for few-shot learning,” in CVPR, 2020.

[0013] [7] Wenbin li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and JieboLuo, “Revisiting local descriptor based image-to-class measure for few-shotlearning,” in CVPR, 2019.

[0014] [8] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun, “Saliencyoptimization from robust background detection,” in CVPR, 2014.

[0015] [9] Jianming Zhang, Stan Sclaroff, Zhe L. Lin, Xiaohui Shen, Brian L.Price, and Radom´ır Mech, “Minimum barrier salient object detection at 80FPS,” in ICCV,2015.

[0016]

[10] Xuebin Qin, Zichen Vincent Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin J¨agersand, “Basnet: Boundary-aware salient object detection,” in CVPR, 2019. Summary of the invention

[0017] The purpose of the present invention is to propose a two-stream convolutional neural network for small sample learning to highlight effective main features and reduce or even eliminate some interference information.

[0018] The technical solution of the present invention to achieve the above-mentioned purpose is a two-stream convolutional neural network for small sample learning, which is characterized by: using the training set of small sample image classification to construct a support set and a query set, introducing an auxiliary stream embedding network in the convolutional neural network, extracting the significant features of the main target, and integrating them with the overall embedding features of the mainstream network.

[0019] The above two-stream convolutional neural network for small sample learning further includes feature vector embedding based on the mainstream network, and a distance function is given to obtain the similarity score from the mainstream network based on the distance measurement between the support samples of N categories and the query sample It includes feature vector embedding and similarity scoring based on auxiliary flow network, obtaining foreground images from the support set and generating embedding vectors, constructing auxiliary flow loss, and obtaining similarity scores from auxiliary flow network based on distance metric. It also includes score fusion of similarity scores. First, the sum of each similarity score of all categories is adjusted, then the final score is calculated, and finally, the category with the largest score is used as the final prediction of the classification result through the argmax function.

[0020] The technical solution of the dual-stream convolutional neural network applied in the present invention has significant advancement compared with other existing methods: in the convolutional neural network of small sample learning, the significant features of the main target are extracted through the auxiliary stream network and integrated with the overall embedded features of the mainstream network, thereby enhancing the overall recognition and stability of the network part, and improving the accuracy of the training results and other learning performance in small sample learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Illustration of the interference between foreground objects and background in the few-shot learning task.

[0022] Figure 2It is a schematic diagram of the process of the two-stream convolutional neural network of the present invention.

[0023] Figure 3 The figure shows the comparison of the application effects of various salient target detection methods. DETAILED DESCRIPTION

[0024] The specific implementation modes of the present invention will be further described below in conjunction with the accompanying drawings of the embodiments, so that the technical solution of the present invention is easier to understand and grasp, thereby making a clearer definition of the protection scope of the present invention.

[0025] In the field of computer image classification, images often contain various interference information, especially interference from cluttered backgrounds, which leads to the lack of discernment of learned features. Through technical analysis and research, it is known that the rational use of key features of the main target (foreground) can correct misclassification. Therefore, the present invention considers designing a more ideal small sample learning method, so that when learning with small samples, effective main features can be highlighted and some interference information can be reduced or even eliminated.

[0026] The two-stream convolutional neural network for small sample learning of the present invention is summarized as follows: using the training set of small sample image classification to construct the support set and query set, introducing the auxiliary stream embedding network in the convolutional neural network, thereby extracting the significant features of the main target and integrating them with the overall embedding features of the mainstream network, thereby suppressing interference information. In detail, the standard small sample learning method is usually in " type Sample" classification task, where N and K represent the number of categories and the number of labeled samples in each category, respectively. Meta-learning ideas are widely used in small sample learning, that is, in the first In the training iterations, we first sample categories, and then randomly sample from each category A support set is constructed from labeled data, which can be expressed mathematically as: . At the same time, from the same The remaining samples after classification and sampling into the support set are used as the query set: the mathematical expression is: ,in, is a sample with the dimension The characteristic vector of is the corresponding label.

[0027] In order to train the classification model by simulating the test phase setting. Therefore, the training process of few-shot image classification is achieved by minimizing the error of predicting the labels in the query set based on the given support set. Specifically, for the first The model is trained for epochs by minimizing the loss of the negative log-likelihood of the true class of each query sample. accomplish:

[0028] ,in are model parameters, and Represents the rth sample and label.

[0029] The dual-stream convolutional neural network structure and data processing flow of the present invention are as follows: Figure 2 As shown, the main body can be divided into three main parts: mainstream network, auxiliary stream network and score fusion, which are orthogonal to the metric-based small sample learning method, and are refined as follows.

[0030] From the first component and data processing, it includes feature vector embedding and similarity scoring based on mainstream networks. Indicates that support is concentrated in the There are labeled samples of categories, and the representation vector of dimension Y is embedded by the function Get, where is the trainable parameter of the model, training rounds, by calculating the The mean vector of the embedded samples of the category gets each feature prototype , ,in and are the tth sample and its label respectively.

[0031] Then, given a distance function , a sample in the query set is classified according to its distance from the feature prototype in the embedding space Generate distributions corresponding to each category: . Thus, the true category is minimized by a random gradient descent optimizer The negative log probability of is used to train the model.

[0032] Then, based on the distance measurement between the support samples of N categories and the query sample, the similarity score from the mainstream network is obtained .

[0033] From the second component and data processing, it includes feature vector embedding and similarity scoring based on the auxiliary flow network. The feature extraction operation of the auxiliary flow network is redirected to the main target features that humans pay more attention to. That is, the foreground image is obtained from the support set and the embedding vector is generated, the auxiliary flow loss is constructed, and the similarity score from the auxiliary flow network is obtained based on the distance metric. The above-mentioned acquisition of the foreground image is to predict the salient map using the salient target detection method, and then extract the foreground image. For an image , define the resulting salient image , foreground image , then let , which is to expand the vector along a single dimension to a larger dimension by replication, and the mathematical expression for extracting the foreground image is ,in represents the Hadamard product.

[0034] The above-mentioned salient object detection methods have multiple options, such as unsupervised methods RBD, MB (see references 8 and 9 for details) and deep supervised methods BASNet (see reference 10 for details). Figure 3 As shown in the application effect diagram, the supervised detection method performs better than the unsupervised detection method, especially for more complex application scenarios. Therefore, the present invention preferably uses BASNet pre-trained on a data set that is not related to small sample image classification.

[0035] Then the foreground image is sent to the auxiliary flow network, and the auxiliary flow loss is constructed using the same loss construction method as the mainstream network, and the similarity score from the auxiliary flow network is obtained based on the distance metric. .

[0036] From the third data processing process, it also includes the score fusion of similarity scores. First, the sum of each similarity score of all categories is adjusted. The mathematical expression of the adjustment is: ,in represents the scaled auxiliary stream similarity score.

[0037] Then calculate the final score , the mathematical expression of the calculation is: ,in is the fusion coefficient, and finally through the argmax function, The category with the largest score is used as the final prediction of the classification result. According to experimental determination, the fusion coefficient The value is 0.9.

[0038] In the specific model training, the optimizer uses stochastic gradient descent, the momentum is 0.9, and the weight decay is 5e -4 The model was trained for 80 stages or until convergence, with each stage containing 1000 rounds. The initial learning rate was set to 0.1 and decayed to the original 0.1 every 20 stages.

[0039] From the above introduction to the dual-stream convolutional neural network combined with the illustrated scheme and detailed description of the embodiments, it can be seen that compared with other existing methods, it has significant advancement: in the convolutional neural network of small sample learning, the auxiliary stream network extracts the significant features of the main target and integrates them with the overall embedded features of the mainstream network, thereby enhancing the overall recognition and stability of the network part, and improving the accuracy of the training results and other learning performance in small sample learning.

[0040] In addition, the two-stream convolutional neural network and its training rules proposed in the present invention are also applicable to other small sample learning methods based on metric learning, and the orthogonality with other small sample learning methods is due to the fact that this method acts on the network structure part.

[0041] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the scope of protection required by the present invention.

Claims

1. A two-stream convolutional neural network for small sample learning, characterized by: The support set and query set are constructed using the training set of small sample image classification. The auxiliary stream network is introduced into the convolutional neural network to extract the salient features of the main target and merge them with the overall embedding features of the mainstream network. Contains feature vector embedding and similarity scoring based on mainstream networks. The representation vector of dimension Y is embedded through the embedding function Get, where is the trainable parameter of the model, training rounds, by calculating the The mean vector of the embedded samples of the category gets each feature prototype , ,in and are the tth sample and its label respectively; given a distance function , a sample in the query set is classified according to its distance from the feature prototype in the embedding space Generate distributions corresponding to each category: , based on the distance measurement between the support samples of N categories and the query sample, the similarity score from the mainstream network is obtained ; Contains feature vector embedding and similarity scoring based on auxiliary flow network, obtains foreground image from support set and generates embedding vector, constructs auxiliary flow loss, and obtains similarity score from auxiliary flow network based on distance metric , where obtaining the foreground image is to predict the salient map using the salient target detection method, and then extract the foreground image, where the salient target detection method is selected as the pre-trained, deep supervised BASNet. For an image , define the resulting salient image , foreground image , then let , the vector is expanded along a single dimension to a larger dimension by replication, and the mathematical expression for extracting the foreground image is ,in represents the Hadamard product; It also includes the score fusion of similarity scores. First, the sum of each similarity score of all categories is adjusted. The mathematical expression of the adjustment is ,in represents the scaled auxiliary stream similarity score; Then calculate the final score , the mathematical expression of the calculation is: ,in is the fusion coefficient, and finally through the argmax function, The category with the largest score is taken as the final prediction of the classification result.

2. The two-stream convolutional neural network for small sample learning according to claim 1, characterized in that: The construction process of the support set is to first sample N categories from the training set, and then randomly sample K labeled data from each category. The mathematical expression of the support set is: ; and the query set is obtained from a batch of samples remaining after sampling from the same N categories and classifying them into the support set. The mathematical expression of the query set is: ,in, is a sample with the dimension The characteristic vector of is the corresponding label.

3. The two-stream convolutional neural network for small sample learning according to claim 1, characterized in that: The fusion coefficient The value is 0.9.

Images