An Identical Difference Model for Few-Shot Learning to Solve Identical Difference Problems on a Comprehensive Visual Reasoning Test
By proposing a small amount of learning alike difference model in the comprehensive visual inference test, using twin networks and pattern recognition modules, the problem of deep learning models identifying the 'same' and 'difference' visual categories under a small amount of data is solved, and high-precision Visual reasoning goes beyond human performance.
Patent Information
- Application Number
- CN202010149739.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-03-06
AI Technical Summary
Existing deep learning models require a large amount of label data and are prone to overfitting in solving highly abstract visual learning tasks, especially when the same visual inference test task, making it difficult to identify 'same' and 'difference' visual categories through a small amount of learning. .
A small number of learning to solve the same-difference model for comprehensive visual reasoning tests is proposed, including input module, interest region selection module, twin network and pattern recognition module. Meta-training is performed through fraternal twin networks, identical twin networks learn relationship scores, and using The pattern recognition module is classified and sets the relationship score threshold to alleviate overfitting.
The highly abstract ‘same’ and ‘different’ visual concepts are realized under a small amount of data, significantly improving the classification accuracy of comprehensive visual reasoning tests, exceeding human performance, and performing excellently on multiple tasks.
Smart Images

Figure CN111368925B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of machine learning, and in particular relates to an isometry model for solving isometry problems of comprehensive visual reasoning tests through small amount of learning. Background Art
[0002] In recent years, computer vision has witnessed many major breakthroughs in standard recognition tasks such as image classification, image segmentation (the process of dividing a digital image into multiple segments) or object detection, and most of these gains are the result of applying deep convolutional neural networks (CNNs). However, visual learning tasks that require attention to highly abstract concepts such as "same" and "difference" have proven to be particularly difficult for standard deep convolutional neural networks.
[0003] Research on different problems has many practical significance. One of the popular applications is face similarity recognition, which can help us verify identity or do some face analysis. In modern mobile devices, many functions are developed based on face recognition, such as being able to open a mobile phone or open a banking application through facial recognition. Face recognition can also help us lock on to many people's targets. Research on the same problem can significantly improve the performance of face recognition. In addition, humans can learn these highly abstract visual concepts such as "same" and "difference" without supervision. When a person has seen someone once, they can usually remember who the person is when they meet on the street next time. In the comprehensive visual reasoning test task, it was also demonstrated that humans can learn the concepts of "same" and "difference" with only a few examples; however, for most deep learning models, they need a lot of labeled training data to train a large number of parameters, considering that there are not many labeled datasets in real life, and new classes require a lot of annotations, which is very time-consuming and laborious. In some cases, for new classes, there may not be a large number of annotated images at all. Emerging categories such as rare animals will recognize visual categories based on very few labeled examples.
[0004] Learn More from Less: The same task aims to identify “same” and “different” visual categories based on very few labeled examples. Solving the same problem is extremely challenging for the following reasons: (1) prior work has shown that standard deep learning methods cannot solve the same problem even when provided with millions of training examples; and (2) providing only one or a few examples can challenge the standard “fine-tuning” practice in deep learning and more easily lead to overfitting.
[0005] To this end, a small-scale learning approach is required to handle the same different categories.
[0006] In particular, we learn to solve the same problem on a synthetic visual reasoning test task several times, and then apply the model to solve the same more complex problem in real life. Summary of the invention
[0007] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to solve the above-mentioned problems existing in machine learning in the prior art.
[0008] In order to achieve the above object, the present invention adopts the following technical solutions:
[0009] A same-difference model for solving the same-difference problem of a comprehensive visual reasoning test by a small amount of learning, the same-difference model comprising an input module, an interest region selection module, a twin network and a pattern recognition module;
[0010] The input module is used to input test samples;
[0011] The region of interest selection module is used to obtain a plurality of regions of interest in the test sample, and randomly divide the plurality of regions of interest into a plurality of pairs, and then feed them into the twin network;
[0012] The twin network includes a fraternal twin network and an identical twin network with the same structure, the fraternal twin network is used for meta-training, and the identical twin network is used for learning;
[0013] The identical twin network includes an embedding module and a relationship module, wherein the embedding module generates a feature map and an operator according to each pair of regions of interest fed in, feeds the feature map into the relationship module, generates a scalar from 0 to 1 for indicating similarity, calculates the relationship score between each pair of regions of interest, and sets a relationship score threshold;
[0014] The pattern recognition module includes a filter module P and a K nearest classifier. The filter module P is used to set the relationship score below the relationship score threshold to 0, and add the relationship scores of each pair of regions of interest to obtain a similarity value. The K nearest neighbor classifier is used to classify the same questions of the comprehensive visual reasoning test based on the predicted relationship scores.
[0015] For the improvement of the present invention, the relationship score threshold is set by obtaining the relationship score threshold through a training set and a validation set.
[0016] In the improvement of the present invention, the obtaining of the plurality of regions of interest in the test sample is performed by selective search.
[0017] For the improvement of the present invention, a measurement standard of the intersection between each pair of regions of interest is calculated, and the measurement standard of the intersection is used to measure the detection accuracy of the corresponding object.
[0018] For the improvement of the present invention, before calculating the detection accuracy of the intersection between each region of interest, it also includes setting an overlap threshold. When the detection accuracy of the intersection between a pair of regions of interest is greater than the overlap threshold, the region of interest with a smaller area is deleted.
[0019] For the improvement of the present invention, the embedding module is composed of several convolutional layers and maximum pooling layers, the size of the convolutional layer is 3, the fixed step size is 1, and the number of convolutional layers is specified as a multiple of 16.
[0020] For the improvement of the present invention, the relationship module is composed of a first convolution block, a second convolution block, a first connection layer and an output layer, the first convolution block and the second convolution block respectively include a convolution layer with 256 filters, the first connection layer and the output layer are fully connected, the first connection layer is 64-dimensional and uses a ReLU activation function, the output layer is 1-dimensional and uses a Sigmoid activation function.
[0021] For the improvement of the present invention, a dropout layer is provided between the first connection layer and the output layer to alleviate overfitting.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] The present invention provides a same-difference model for solving the same-difference problem of a comprehensive visual reasoning test with a small amount of learning. The same-difference model includes an input module, an interest region selection module, a twin network and a pattern recognition module; the input module is used to input a test sample; the interest region selection module is used to obtain a number of interest regions in the test sample, and the several interest regions are randomly divided into a number of pairs, and then fed into the twin network; a fraternal twin network is used for meta-training, and then an identical twin network is used for learning, and the relationship score of each pair of interest regions is predicted, and finally the pattern recognition module is used to classify the same problem of the comprehensive visual reasoning test based on the predicted relationship score. The present invention recognizes patterns by obtaining the similarity of the interest regions in the test sample, and can recognize highly abstract visual concepts of "sameness" and "difference". BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a structural schematic diagram of the isochronous model of the present invention.
[0025] Figure 2 The figure is a simplified schematic diagram of the process of solving the isochronous problem using the isochronous model of the present invention.
[0026] Figure 3 Some example images of identity-difference problems taken from the comprehensive visual reasoning test task.
[0027] Figure 4An example graph of the Omniglot dataset.
[0028] Figure 5 Some random data augmentation samples generated for three images in the Omniglot dataset.
[0029] Figure 6 Four examples for the same visual classification task.
[0030] Figure 7 Here are some examples of the Fashion-MNIST 10-shot task.
[0031] Figure 8 Four examples of the Fashion-MNIST isometry visual reasoning task.
[0032] Fig. 9 A sample of the dataset generated for the AT&T face database.
[0033] Fig.10 An example of a one-shot task involving face recognition. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0035] To solve the same problem by learning with a few shots, three datasets are needed: training set, support set and test set. The support set and test set share the same label space. They come from the few-shot problem of the target. In the case of the support set alone, in principle, a classifier can be trained to predict the classification label y for each sample x in the test set; however, due to the lack of labeled samples in the support set, this will lead to overfitting, and the performance of such a classifier is usually unsatisfactory; therefore, it is necessary to perform a meta-training task in advance with the training set. The training set has its own label space, which is different from the support and test sets, and is mainly used to perform meta-training so that transferable knowledge can be learned to help perform better one-shot learning on the support set and classify the test set more effectively. When the support set contains C different classes and each class has K examples, the target few-shot problem is called C to K shot. Consider the same difference problem of a comprehensive visual reasoning test task consisting of randomly generated objects in each category of each task (for example, each image contains two identical objects, but the same objects are different between images). When solving this problem, it is necessary to understand the similarity between the objects of each image; therefore, the problem under study can be defined as one-shot learning, which means that the machine needs to predict whether another object is the same as the object that the machine has only seen once.
[0036] In order to solve the SD problem in the comprehensive visual reasoning test task (SVRT), a SD model needs to be established:
[0037] See also Figure 1 and Figure 2 , the present invention provides a same-difference model for solving the same-difference problem of comprehensive visual reasoning test by a small amount of learning, the same-difference model 1 includes an input module 11, an interest region selection module 12, a twin network 13 and a pattern recognition module 14;
[0038] The input module 11 is used to input a test sample;
[0039] The region of interest selection module 12 is used to obtain a plurality of regions of interest in the test sample, and randomly divide the plurality of regions of interest into a plurality of pairs, and then feed them into the twin network;
[0040] First, the test sample x i Input into the region of interest selection module to obtain several regions of interest (r i , r j , ...), then randomly group these regions of interest into pairs, and feed each pair of regions of interest into the same Siamese network;
[0041] The twin network 13 includes a fraternal twin network and an identical twin network with the same structure, the fraternal twin network is used for meta-training, and the identical twin network is used for learning;
[0042] The Siamese network 13 is based on a convolutional Siamese neural network by replacing the fixed L1 distance metric on top of the Siamese network with a learnable convolutional neural network metric;
[0043] The identical twin network includes an embedding module f φ and relation module g φ , the embedded module f φ Generate a feature map (f φ (r i ) and f φ (r j )) and operator (C(f φ (r i ) and f φ (r j )) combination), the operator is a deep concatenation on the feature map, feeding the feature map into the relation module g φ In the above example, a scalar between 0 and 1 is generated to represent the similarity, and the relationship score between each pair of regions of interest is calculated. For example, the region of interest (r i ) and the region of interest (r j) is scored as follows:
[0044] g φ (C(f φ (r i ), f φ (r j )))
[0045] Set the relationship score threshold, and then use the pattern recognition module to determine x based on the relationship score. i Which categories the samples belong to.
[0046] The described identical twin network is based on a convolutional Siamese neural network. The metric learning-based approach aims to learn a set of projection functions so that when images are represented in an embedding space, they can be easily identified using simple nearest neighbor or linear classifiers. In the convolutional Siamese neural network, the L1 distance metric is used to calculate the distance between each pair of top-level feature representations. In this work, a learnable convolutional neural network metric called a relation module is used to replace the L1 distance metric. Compared with the L1 distance that focuses on learning the shallow (linear) Mahalanobis metric for fixed feature representations, the convolutional neural network metric can learn a deep nonlinear metric (similarity function) that can better identify matching / unmatching pairs; a new embedding module f is developed for this purpose. φ , which can accommodate 84x84 SVRT images.
[0047] The pattern recognition module 14 includes a filter module P and a K nearest classifier. The filter module P is used to set the relationship score below the relationship score threshold to 0, and add the relationship scores of each pair of regions of interest to obtain a similarity value. The K nearest neighbor classifier is used to classify the same questions of the comprehensive visual reasoning test based on the predicted relationship scores.
[0048] In summary, the same-difference model includes an input module, an interest region selection module, a twin network and a pattern recognition module; the input module is used to input a test sample; the interest region selection module is used to obtain a number of interest regions in the test sample, and the several interest regions are randomly divided into a number of pairs, and then fed into the twin network; the fraternal twin network is used for meta-training, and then the identical twin network is used for learning, and the relationship score of each pair of interest regions is predicted, and finally the pattern recognition module is used to classify the same questions of the comprehensive visual reasoning test based on the predicted relationship score. The present invention recognizes patterns by obtaining the similarity of the interest regions in the test sample, and can recognize highly abstract visual concepts of "sameness" and "difference".
[0049] In some embodiments, the relationship score threshold is set by obtaining the relationship score threshold through a training set and a validation set.
[0050] It should be noted that some visual reasoning test tasks (SVRT) involve more than two objects and require solving highly abstract "grouping" concepts, see Figure 3 ,like Figure 3 SVRT#5 and SVRT#7 in , for SVRT#7, the objects in the positive class can be divided into three groups, each group contains two identical objects, and the objects in the negative class can be divided into two groups, each group contains three identical objects; therefore, in addition to determining the similarities and differences of each pair, we also need to learn the pattern of each class. Through the same twin network, we can obtain the relationship score of each pair of regions of interest, and finally get all the relationship scores of all pairs. According to the relationship score, a simple way to identify the pattern of each category is to obtain the sum of these relationship scores; however, since the image in SVRT#7 is too simple and consists only of lines, many pairs belonging to different categories also have high correlation scores; therefore, it is necessary to use the relationship score threshold obtained by the training set and the validation set to distinguish the relationship scores. When the relationship score is lower than the relationship score threshold, the pair has a higher possibility of belonging to different categories, and the relationship score of this pair is set to 0. Thereafter, the sum of all relationship scores is taken as the similarity value; in this way, since the similarity is one-dimensional and can reflect the pattern of each category, a simple nearest neighbor or linear classifier can be used to identify the category. Finally, a k-nearest neighbor classifier is trained to predict the class label based on the similarity.
[0051] In some embodiments, the obtaining of the plurality of regions of interest in the test sample is performed by selective search.
[0052] It should be noted that the selective search combines the advantages of exhaustive search and segmentation, and can capture all possible regions of interest with a high recall rate. Selective search has been developed as an independent detector. The selective search method contains three hyperparameters: scale, sigma, and minSize. By configuring reasonable parameters for the hyperparameters sigma, scale, and minSize, the region of interest can be effectively located. Of course, to obtain the region of interest in the test sample, other methods are also possible, such as using Python-OpenCV.
[0053] In some embodiments, a measurement criterion of an intersection between each pair of regions of interest is calculated, and the measurement criterion of the intersection is used to measure the detection accuracy of the corresponding object.
[0054] Among all the detection results, some are inaccurate and highly repetitive. In order to remove the repeated areas and improve the detection accuracy, it is necessary to calculate the detection accuracy of the intersection of interest (IoU) between each pair of regions of interest. In the image field, IoU is a standard for measuring the detection accuracy of corresponding objects, as shown in the following formula:
[0055]
[0056] The above formula shows how to calculate the IoU in an image field, which is the area where two detection regions overlap divided by the total area.
[0057] In some embodiments, in order to eliminate erroneous detection results for the same difference problem in the comprehensive visual reasoning test task, before calculating the detection accuracy of the intersection between each region of interest, it also includes setting an overlap threshold. When the detection accuracy of the intersection between a pair of regions of interest is greater than the overlap threshold, the region of interest with a smaller area is deleted; each bounding box is defined by a four-tuple (r, c, h, w), which specifies its upper left corner (r, c) and its height h and width w.
[0058] In some embodiments, the embedded module f φ It is a VGGlike architecture. The embedding module consists of several convolutional layers and maximum pooling layers. The size of the convolutional layer is 3, the fixed step size is 1, and the number of convolutional layers is specified as a multiple of 16; the convolutional layer refers to a convolutional Siamese neural network. This setting can optimize the performance of extracting features. Embedding applies the rectified linear unit (ReLU) activation function to the output feature map. Each set of convolutional layers is followed by a 2x2 maximum pooling layer with a step size of 2.
[0059] Through the embedded module After obtaining a pair of feature maps of the region of interest, the feature maps are deeply concatenated and input into the relationship module g φ middle.
[0060] In some embodiments, the relationship module consists of a first convolution block, a second convolution block, a first connection layer and an output layer, each convolution block (the first convolution block and the second convolution block) contains a convolution layer with 256 filters, followed by batch normalization, a ReLU nonlinear activation function and a 2x2 maximum pooling layer, and the kernel size of each filter in the convolution layer is 3x3; the two convolution blocks are followed by two fully connected layers (the first connection layer and the output layer), the first connection layer is 64-dimensional and uses the ReLU activation function, and the output layer is 1-dimensional and uses the Sigmoid activation function (logic function) to generate a relationship score within a reasonable range for each twin pair.
[0061] It should be noted that the training includes:
[0062] Loss function, indexing the i-th mini-batch (the number of training data in one iteration), assuming the mini-batch size is m, let y(x i , x i ) represents a vector of length m. This vector refers to the true label of the mini-batch. When x1 and x2 belong to the same category, y(x i , x i )=1, otherwise y(x i , x i )=0, let the vector p(x i , x i ) represents the order result of the previous mini-batch. We use binary cross entropy as the loss function. Binary cross entropy can be calculated as:
[0063] L(x i , x i )=-y(x i , x i )logp(x i , x i )-(1-y(x i , x i ))log(1-(p(x i , x i )))
[0064] Optimization,Backpropagation is a supervised learning algorithm widely used for feedforward neural network training, which uses the standard backpropagation algorithm to optimize the objective function by calculating the gradient. We set the minimum batch size to 128 and the learning rate to α j , with momentum β j , adjust the weight to λ j , so the update rules in time period T are as follows:
[0065] j(x1, x2) = w k j +Δw k j (x1, x2)+2λ j |w k j |
[0066] Δw kilotons (x i , x i ) = -αjvw kilotons + β j Δw kT -1
[0067] where vw kj It is the partial derivative with respect to the weight between the jth neuron in a certain layer and the kth neuron in the consecutive layer;
[0068] Weight initialization, before training, initialize the weights and biases of all convolutional layers and two fully connected layers (the first connected layer and the output layer);
[0069] For all convolutional layers, the weights are taken from layers with zero mean and 10 -2 The standard deviation of a normal distribution,
[0070] From a mean of 0.5 and a standard deviation of 10 -2 The fully connected layers are initialized in the same way as the convolutional layers, but the weights have a mean of zero and a standard deviation of 2x10 -1 Initialize with a normal distribution.
[0071] Learning rate: Set the initial learning rate to α, and then gradually reduce the learning rate by dividing by 10. For example, if the initial learning rate is 0.06, then gradually reduce the learning rate by 0.006 and 0.0006.
[0072] The validation error was monitored while training the same twin network, and when the validation accuracy did not increase in 50 iterations, it was stopped based on the validation accuracy and the parameters of the model were used at the best iteration step. Then, the learning rate was updated to continue training the network.
[0073] Hyperparameter Optimization. For the learning rate and regularization hyperparameters, we set the initial learning rate αE[10 -2 , 10 -1 ], momentum βE[0, 1] and adjustment weight λE[0, 0.1], for the neural network hyperparameters, the number of convolutional filters in each layer is set from 16 to 256 using multiples of 16, for the fully connected layers (first connected layer and output layer) hyperparameters, we set the units from 16 to 256, also in multiples of 16, these hyperparameters are obtained by maximizing the accuracy of a single task in the validation set.
[0074] In some embodiments, a dropout layer is provided between the first connection layer and the output layer to alleviate overfitting.
[0075] In summary, we combine some experiments and compare them with strong baselines.
[0076] Using a meta-training (learning) strategy to solve the same problem with little one-time learning requires two datasets. The first dataset is used for meta-training to extract some transferable knowledge from some auxiliary tasks, which can help the same-difference model learn generalization ability; the second dataset is a dataset for meta-testing, which is the same-difference problem of the comprehensive visual reasoning task that currently needs to be solved.
[0077] We use the Omniglot dataset as a meta-training dataset to extract some transferable knowledge and to study one-shot learning and develop more human-like learning algorithms. The Omniglot dataset contains more than 1,600 handwritten characters from 50 different alphabets, ranging from familiar Latin to unfamiliar local dialects (see Figure 4 ).
[0078] The number of characters (classes) in each alphabet ranges from about 15 to a maximum of 40 characters. Each class contains 20 samples drawn by different people. Lake divides these 50 different alphabets into a 40-letter background set and a 10-letter evaluation set. The background set is used to develop the same fraternal twin network by learning hyperparameters and feature mappings, and the evaluation set is used to test the single-shot classification performance of the network. Each image in the Omniglot dataset is a 105x105 binary value image, which is manually drawn on an online canvas. In order to transfer the developed twins network to solve the problem of the comprehensive visual reasoning test task, the size of the Omniglot dataset is resized to 84x84 to be consistent with the comprehensive visual reasoning test task. Since the images in the comprehensive visual reasoning test task are in RGB image format, the binary value images need to be changed to RGB images. In addition, considering that each category contains only 20 samples, a ground set (training set) with data augmentation is also added. This strategy can significantly increase the diversity of data available for training models without actually collecting new data. Data augmentation techniques include scaling, rotation, horizontal flipping and translation. (See Figure 5 ) shows some random data augmentation samples generated for three images in the Omniglot dataset.
[0079] We solve the same-difference problem in the comprehensive visual reasoning test task in three steps. The first step is to search and select the regions of interest, and the second step is to meta-train the identical twin network. The last step is to use the trained identical twin network to learn once and predict the relationship score for each pair of regions of interest. After that, the pattern recognition module is used to classify the same questions in the comprehensive visual reasoning test task based on the predicted relationship scores.
[0080] To select the “region of interest”, we set the hyperparameters scale to 1000, sigma to 0.3, and minimum size to 10, which helps to effectively locate the regions of interest. For IoU, we set the overlap threshold to 0.9, which means that when the IoU between two regions of interest is greater than 0.9, the region with the smaller area will be removed.
[0081] For the identical twin network, the Omniglot dataset can be used for meta-training, which helps the identical twin network learn some transferable knowledge; first, the Omniglot background set is randomly divided into several pairs, and then these pairs are fed into the same fraternal twin network to train the network; the hyperparameters follow the following settings: batch size 128, Adam optimizer (initial learning rate 0.06, momentum 0.6, adjustment weight 0.02), Siamese, the validation task is set to 20 single-shot networks, and 1000 validation tasks are set for each evaluation to help us choose the best parameters, the network with the highest classification accuracy is used to predict the similarity of the region of interest on the same-difference task in the comprehensive visual reasoning test task; PyTorch implements the same twin network.
[0082] Finally, the pattern recognition module is used to predict the category of the same-difference task samples in the new comprehensive visual reasoning test task based on the similarity of the regions of interest. First, the pattern recognition module filters the relationship scores and sets the relationship scores below the threshold to 0. The threshold is learned by the same-difference task validation set in the comprehensive visual reasoning test task, and the value with the maximum classification accuracy on the validation set is selected. Thereafter, the similarity value of each image is obtained by adding the correlation scores of all pairs (each pair) of regions of interest. Then, a K-nearest classifier is trained to predict the class label of each new comprehensive visual reasoning test task image based on similarity.
[0083] The value of k is selected from {1, 2, 3, 4, 5}, and the optimal value is found to be k=2.
[0084] In addition to comparing with the few-shot learning results on the synthetic visual reasoning test task, the same task was obtained by Fleuret et al. We also compare our method with several other methods that perform well on few-shot classification tasks, including fine-tuning deep convolutional neural networks ModelAgnostic Meta-Learning (MAML, model-agnostic meta-learning), Prototype Nets, and Relation Nets. For all these models, we use synthetic visual reasoning test images of size 84x84, ten training samples, 1000 validation samples, and 10000 test samples. Performance is measured as classification accuracy. To produce all experimental results, we averaged ten trials, each with 1000 different unseen trial images. For hyperparameter tuning, we used 1000 validation images for each task. In addition, for the meta-learning methods, ModelAgnostic Meta-Learning (MAML), Prototype Nets, and Relation Nets, we also meta-trained using the Omniglot dataset to extract transferable knowledge based on our method, and used the same-difference task from the synthetic visual reasoning test in the meta-testing stage. Since each comprehensive visual reasoning test task is a binary classification problem and has 10 training samples (5 positive samples, 5 negative samples), we define the meta-learning task as 2 5-shots; 2 5-shot meta-training tasks are generated through the Omniglot dataset; 2 5-shot meta-test tasks are also generated by the same-difference problem in the comprehensive visual reasoning test (10 training samples, 10,000 test samples).
[0085] Fine-tuning a deep convolutional neural network is mainly about reusing a pre-trained model on a new problem. When solving a new problem, most of the layers in the front of the model are frozen, and we only need to fine-tune the last few layers. For this approach, we fine-tune the Vgg-16 architecture pre-trained on ImageNet, other architectures are also possible, such as ResNet, InceptionV3, Xcep-tion.
[0086] The fully connected layer in the Vgg-16 architecture was replaced by two fully connected layers of width 256 and 64, which were activated by a sigmoid function; the softmax classifier was trained using the Adam optimizer provided by Keras, and the loss function used the binary cross entropy loss; we trained the deep convolutional neural network for 200 epochs and selected the optimal number of epochs for each comprehensive visual reasoning test equivalent task based on the classification accuracy on the validation set; the optimal number of epochs for each task was selected from {8, 16, 32, 64}.
[0087] Model-agnostic meta-learning (MAML) aims to meta-learn an initial set of neural network weights (parameters of explicitly trained models), so that the neural network model can be effectively fine-tuned to produce good generalization performance on new few-shot learning tasks within a few gradient update steps; the implementation process is publicly available and used. 2 5-shot meta-training tasks are generated through the Omniglot dataset; we set 60,000 meta-train iterations and used 1,000 validation images for each task. Finally, we selected the model that maximized the prediction accuracy of the same-difference validation images in the synthetic visual reasoning test task for testing 10,000 unseen same-difference images in the synthetic visual reasoning test task for each task for ten trials (one trial is 1,000 test samples).
[0088] Prototypical networks are developed based on the idea that each class can be represented in a representation space learned on a neural network for a meta-training task by the average of its support examples; we used the same architecture for this approach, and the Omniglot dataset was used to perform a 2-way 5-step meta-learning task for the prototypical network. Ten synthetic visual reasoning test task training samples were used as support sets. Finally, we predicted the class of 10,000 homogeneous test samples in the synthetic visual reasoning test task for ten trials (1,000 trial samples for one trial); the optimal number of epochs for each task was selected from {8, 16, 32, 64}.
[0089] The relation network aims to learn a deep distance metric by simulating the few-shot setting in each episode. For this model, the Omniglot dataset is used to perform 2 five-shot learning tasks for the relation network; ten comprehensive visual reasoning test task training samples are used as support sets; finally, by using the trained model, the class of 10,000 test samples in the same-difference task in the comprehensive visual reasoning test task can be predicted for 10 trials (one trial is 1,000 test samples); the batch number for each class is set to 5. The number of training and testing times is 10,000 and 1,000 respectively.
[0090] Our experimental results include two parts: one for meta-training on the Omniglot dataset, and the other for meta-testing on the same problems in the synthetic visual reasoning test task.
[0091] In summary, we use a meta-training (learning) strategy to train the same twin network. As shown in Table 1 below, the meta-training experimental results of the fraternal twin network on the Omniglot dataset are shown;
[0092]
[0093] Table 1
[0094] Since our identical twin network is based on the Siamese network, we also compare the 20 one-way performances of the network with the Siamese network. As can be seen in Table 1, our identical twin network greatly improves the accuracy of one-shot classification by nearly 5% compared with the Siamese network.
[0095] As shown in Table 2 below, the performance difference between our method and all other models we compared with, it can be seen that with only ten training samples provided, our model significantly outperforms other models on the same problems in all comprehensive visual reasoning test tasks, with an average advantage of more than 35%.
[0096]
[0097] Table 2
[0098] As shown in Table 2, on some same-difference problems, our same-difference model even exceeds the performance reported by humans. To perform benchmarks on same-difference tasks such as Prototype Networks, MAML, Relation Network, and fine-tuned deep convolutional neural networks, they all account for about 50%, which is consistent with Stabinger et al. that it is difficult for convolutional neural networks to learn the required “same” and “difference” features despite a large number of training samples.
[0099] like Figure 3 SVRT#1 and SVRT#15 in the paper contain two and four random objects in each category, respectively. In these two tasks, the shapes of the objects are different in category 1 but the same in category 2. Our method performs very well on these two problems, with accuracies of 97.3% and 99.1% respectively. These results are explainable. When the number of objects in an image increases, there will be more identical or different features in this image. Our method also achieves a classification accuracy of 98.6% on SVRT#22. For SVRT#22, category 2 contains three identical aligned objects, while category 1 contains three different objects, but the differences between the objects are small. All of these prove that our same-difference model is very suitable for classifying visual tasks with abstract visual concepts of "same" and "difference".
[0100] SVRT#5 and SVRT#7 require attention to the highly abstract concept of “grouping”. Our same-difference model also performs well on the two tasks, with accuracies of 96.2% and 90.3% respectively. Another amazing result of SVRT#16 is that it shows the highly abstract concept of “reflection”. Humans have an average classification accuracy of 78% with only 12.45 images, but our same-difference model can achieve 92.1% accuracy with only ten training examples. All of these prove that our same-difference model is very good at classifying visual tasks with more abstract “same” and “difference” visual concepts of “grouping”.
[0101] SVRT#19, #20, and #21 require machines to be able to recognize shape similarity up to many invariants: scaling invariant, reflection invariant, and scaling and rotation invariant; for SVRT#19, each image contains two objects of different sizes; in class 2, the two shapes are equivalent before scaling and translation; for SVRT#20, each image contains two objects; in class 2, one object can be obtained from the other object by reflection and rotation; for SVRT#21, each image contains two objects, one object in class 2 can be obtained from the other object by scaling, translation, and rotation; our iso-difference model achieves 85.2%, 55.2%, and 62.3% performance in these three tasks, respectively; SVRT#20 and #21 are the most difficult iso-difference tasks for our iso-difference model; in the future, we need to improve our iso-difference model to address these invariants.
[0102] How to use the twin network to solve other similarity-difference problems through single-shot learning, and how to use the similarity-difference model in our comprehensive visual reasoning task to solve other similarity-difference visual reasoning tasks; in addition, we also tested whether the network trained on other tasks can be well generalized to new datasets.
[0103] The MNIST dataset has become a very popular database for researchers to conduct research on machine learning algorithms. In the MNIST dataset, there are ten digit categories, each containing thousands of examples. The MNIST dataset is completely different from the Omniglot dataset used to meta-train our Identical Twins network. The Omniglot dataset contains a small number of handwritten letters in each category, and the number of categories far exceeds the number of examples in each category. This Omniglot dataset can be seen as a "transpose" of MNIST, and we think it is interesting that the Identical Twins network that has been trained on the Omniglot dataset can generalize well to the MNIST dataset. Therefore, we created 10 unidirectional classification tasks and the same-difference visual reasoning task of MNIST to test our adaptation.
[0104] We generated 1000 10-way one-shot tasks from MNIST to test the adaptability of the same Siamese network and the generalization performance of Omniglot on MNIST; Table 3 shows examples of 10 one-shot tasks from the MNIST dataset; during testing, we excluded any fine-tuning on the MNIST training set; each image to be classified was paired with the other ten MNIST images; each pair was then input into a simplified version of the same fraternal Siamese network that we had pre-trained on the Omniglot dataset, downsampled to 28x28.
[0105] method Test accuracy 1 nearest neighbor 26.5 Convolutional Siamese Network 70.3 Identical Twins Network 68.2
[0106] Table 3
[0107] In addition, we also tested the performance of the difference model on the MNIS equivalent visual reasoning classification task generated using MNIST; Figure 6 Several examples of the same visual classification task are shown; the code for generating this dataset is publicly available; in RelationaIMNIST-S, class 2 contains an exact copy, while objects in class 1 are completely different; RelationaIMNIST-SI contains some invariants, meaning that class 2 contains a copy, but this copy is obtained from one object by rotation and scaling; RelationMNIST-C and RelationMNIST-CI contain some fuzzy duplicates, meaning that the "duplicates" only come from the same class in these two tasks; compared to RelationMNIST-C, the "duplicates" in RelationMNIST-CI have some added rotations and scalings.
[0108] We use the same model and the same experimental settings to solve the same visual reasoning task in the synthetic visual reasoning task to solve the same visual reasoning task of MNIST; the same twin network is meta-trained on the Omniglot dataset; the same twin network is not fine-tuned when predicting the class label of the new image. After obtaining the relationship score scores between the regions of interest through the same twin network, they are input into the pattern recognition module to predict the class label. For each MNIST same-difference visual reasoning classification task, we used 10 training sets, 1000 validation sets, and 10000 test sets; as with the same-difference task in the synthetic visual reasoning task, we also evaluated the performance of multiple strong baselines, and the performance is the average of the classification accuracy of ten trials (each trial has 1000 test sets); as shown in Table 4 below, the experimental results of the baseline and our model for the same visual reasoning task of MNIST are shown; it can be seen that our same-difference model significantly outperforms other strong baselines, which are almost random guessing with an accuracy of about 50%.
[0109] method S SI C word Fine-tuning deep CNN 50.6 50.1 49.3 48.1 Prototype Network 51.1 49.9 49.6 48.2 Network 52.2 49.5 49.3 48.5 Model-agnostic meta-learning 51.8 50.3 49.2 48.7 Our 94.5 77.8 58.5 52.2
[0110] Table 4
[0111] like Figure 7 As shown, in Fashion-MNIST, there are ten classes, the same as MNIST, namely T-shirts, pants, pullovers, dresses, coats, sandals, shirts, sneakers, bags, ankle boots; Fashion-MNIST is often used instead of MNIST to benchmark machine learning algorithms; each image in Fashion-MNIST is a 28x28 grayscale image.
[0112] Through MNIST experiments, it has been demonstrated that our same-difference model can be used to solve MNIST problems; it can also be observed that we can obtain reasonable generalizations from the features learned on Ominiglot without training MNIST; Fashion-MNIST is a more complex dataset compared to the MNIST dataset; we want to know whether our same-difference model can be used to solve the more complex Fashion-MNIST dataset, and we also want to know whether the model trained on the Omniglot dataset can be well generalized to more complex problems such as Fashion-MNIST; therefore, we created 10 single-shot classification tasks and Fashion-MNIST visual reasoning classification tasks to test the generalization performance and adaptability of this same-difference model.
[0113] By testing the generalization performance using Fashion-MNIST, we generated 1000 10-way single-shot tasks; Figure 7 Figure 2 shows some examples from the Fashion-MNIST 10-shot task; during testing, we exclude any fine-tuning on the Fashion-MNIST training set.
[0114] The image to be classified is paired with ten other images; each pair is then put into a simplified version of our Identical Twins network, which has been trained on the Omniglot dataset and downsampled to 28x28; finally, the class with the highest score is chosen as the predicted class.
[0115] We also evaluate the nearest neighbor baseline on this Fashion-MNIST 10-shot single-shot task; Table 5 shows the experimental results for the Fashion-MNIST 10-shot single-shot experiment; it can be seen from the experiments that even without Fashion-MNIST training, we are still able to gain generalizability from the features learned on Ominiglot; more complex convolutional neural networks (e.g., our identical twin network) can achieve better performance than simple nearest neighbor classifiers.
[0116] Fashion MNIST method Test accuracy 1 nearest neighbor 25.3 Identical Twins Network 47.2
[0117] Table 5
[0118] We also tested the performance of our same-difference model on the Fashion-MNIST same / different visual reasoning classification task using the Fashion-MNIST generated datasets; Figure 8 Figure 2 shows several examples of the same-difference visual reasoning task for Fashion-MNIST; this dataset was generated in the same way as the same-difference dataset for MNIST and is publicly available; in RelationaIFashionMNIST-S, class 2 contains an exact replica, while objects in class 1 are completely different;
[0119] RelationaIfashionMNIST-SI contains some invariants, which means that category 2 contains a duplicate, but this duplicate is obtained from one object through rotation and scaling. RelationFashionMNIST-C and RelationFashionMNIST-CI contain some fuzzy duplicates, which means that the "duplicates" only come from the same class in these two tasks. Compared with RelationFashionMNIST-C, the "duplicates" in RelationFashionMNIST-CI have some additional rotations and scaling.
[0120] We use the same model and the same experimental setup for solving the same-difference problem in the synthetic visual reasoning test task to solve the same-difference visual reasoning task in Fashion-MNIST. The same Siamese network is meta-trained on the Omniglot dataset without any fine-tuning for the same-difference visual reasoning task in FashionMNIST. As with the same-difference task in the synthetic visual reasoning task, we also evaluate the performance on several strong baselines. Table 6 contains the experimental results of the baseline and our model for different visual reasoning tasks on FashionMNIST; it can be observed that our same-difference model greatly improves the performance of some same-difference tasks compared to other baselines.
[0121] method S SI C word Fine-tuning deep CNN 50.3 49.1 48.5 48.3 Prototype Network 50.1 49.5 49.2 48.7 Network 49.5 49.3 49.0 48.5 Model-agnostic meta-learning 50.2 49.4 48.7 48.8 Our 78.1 60.1 56.5 51.2
[0122] Table 6
[0123] Facial recognition has become a hot research area in recent years and has become one of the common functions used in mobile applications. Convolutional Siamese networks are widely used for face recognition. Our Siamese network is developed based on the convolutional Siamese network. We are also interested in whether our Siamese network can be applied to facial recognition. We downloaded the AT&T face database from a public website (such as Fig.10 The dataset contains images of 40 objects from various angles. There are ten different images of each subject.
[0124] The same twin network requires the input value and the label together as a pair, so we have to generate data in this way; we take two random images from the same object and label them as a pair, and then take an image from two different objects and label them as another pair; Fig. 9 As shown in Figure 2, a sample of a dataset generated using the AT&T face database is shown in Figure 2. Fig. 9 As you can see, the same pair of images has the same task, but different people have different images of different people.
[0125] like Fig.10 An example of a one-shot task on face recognition is shown; only one example is given for each subject, and we need to classify new images; in the AT&T face database, there are 40 images of objects at different angles; we randomly select 35 subjects as training sets to meta-train our same twin network, and the remaining 5 subjects are used as test sets; we generate 1000 5-shot one-shot face recognition tasks to test the performance of the equivalent twin network on face recognition; we also evaluate the performance of the convolutional Siamese network in the AT&T face database; the code of the convolutional Siamese network is publicly available; the test results of this experiment are shown in Table 7 below; it can be seen that compared with the convolutional Siamese network, our twin network can be applied to face recognition and improve the performance.
[0126]
[0127] Table 7
[0128] The ability to recognize highly abstract visual concepts is a ubiquitous human skill that has not made significant progress; the different problems are visual reasoning tasks with highly abstract visual concepts; a large number of studies have shown that standard deep convolutional neural networks cannot solve such tasks, and even after training with a large number of training samples, the performance of convolutional neural networks is very poor; the same problems of the comprehensive visual reasoning test task, as a representative of the same-difference visual reasoning task, include a series of examples that require the machine to be able to recognize highly abstract "same" and "difference" visual concepts, up to many invariants: scale invariance, reflection invariance, and scale and rotation invariance.
[0129] To solve the different visual reasoning problems of one-shot learning, this paper proposes a new method to recognize patterns by checking the similarity of objects in one image; when checking the similarity, it is defined as generalized one-shot learning, and we adopt a meta-learning strategy to solve this problem so that transferable knowledge can be learned to help better one-shot learning on the computer; target task and more effectively classify unseen images; for our isomorphism model, first, we build an identical twin network to calculate the similarity score of any pair of images; second, the regions of interest are extracted from the image by the selective search method, and then these regions of interest are randomly grouped into pairs, and each pair is fed into the identical twin network. The identical twin network can predict the relationship score of each pair; after that, the pattern recognition module calculates the similarity of each image by setting the relationship score below the relationship score threshold to 0, and then adding all the relationship scores between all regions of interest; finally, a k-nearest neighbor classifier is trained to map the similarity to the class label.
[0130] From the experimental results, we can see that for the same difference problem of all comprehensive visual reasoning test tasks, our method greatly exceeds the published results, and also exceeds other strong baselines that have achieved state-of-the-art performance in other few shot image tasks. Most of these algorithms are random guessing with an accuracy of about 50%. Our method can achieve more than 95% accuracy on multiple same difference tasks with only ten training samples, and an average accuracy of more than 85%; on the same difference tasks in some comprehensive visual reasoning test tasks (such as SVRT#5, SVRT#15 and SVRT#16), our same difference model even exceeds the reported manual performance; if using a large number of training samples to train convolutional neural networks cannot solve the problem of same visual views in comprehensive visual reasoning test tasks, our same difference model shows strong single-shot learning ability for same visual views with highly abstract concepts.
[0131] In addition, we also applied the identical twin network and our same-difference model to the same-difference task in the comprehensive visual reasoning test task to solve more other same-difference (same-different) problems, such as MNIST, FashionMNIST, and face recognition, and achieved good performance on these tasks; it has been demonstrated that our same-difference model can be used to solve a wider range of same problems; in the process of solving these tasks, we also learned that some knowledge can be transferred to solve other tasks.
[0132] Finally, we identify how to enable machines to learn visual tasks with one-shot learning, and also explain how machines can generalize highly abstract concepts such as “sameness”, “difference”, “grouping”, and “invariance under rotation, reflection, and scaling” using only a small number of training samples; this opens the door for future research in this area; we believe that this work will inspire future developments that solve the same abstract visual reasoning tasks and one-shot learning problems.
[0133] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the protection scope of the claims attached to the present invention.
Claims
1. A small amount of learning to solve the same difference model of facial recognition, characterized by: The iso-difference model includes an input module, an area of interest selection module, a twin network and a pattern recognition module; The input module is used to input a test sample of a facial recognition image; The region of interest selection module is used to obtain a plurality of regions of interest in the test sample, and randomly divide the plurality of regions of interest into a plurality of pairs, and then feed them into the twin network; The twin network includes a fraternal twin network and an identical twin network with the same structure, the fraternal twin network is used for meta-training, and the identical twin network is used for learning; The identical twin network includes an embedding module and a relationship module, wherein the embedding module generates a facial recognition feature map and an operator according to each pair of regions of interest fed in, feeds the facial recognition feature map into the relationship module, generates a scalar from 0 to 1 for indicating similarity, calculates a relationship score between each pair of regions of interest, and sets a relationship score threshold; The pattern recognition module includes a filter module P and a K nearest neighbor classifier. The filter module P is used to set the relationship score below the relationship score threshold to 0, and add the relationship scores of each pair of regions of interest to obtain a similarity value. The K nearest neighbor classifier is used to classify the same problem of facial recognition based on the predicted relationship score.
2. The isochronous model according to claim 1, characterized in that: The relationship score threshold is set by using a training set and a validation set.
3. The isochronous model according to claim 1, characterized in that: The obtaining of the plurality of regions of interest in the test sample is performed by selective search.
4. The isochronous model according to claim 3, characterized in that: A measure of the intersection between each pair of regions of interest is calculated, and the measure of the intersection is used to measure the detection accuracy of the corresponding object.
5. The isochronous model according to claim 4, characterized in that: Before calculating the detection accuracy of the intersection between each pair of regions of interest, an overlap threshold is set. When the detection accuracy of the intersection between a pair of regions of interest is greater than the overlap threshold, the region of interest with a small area is deleted.
6. The isochronous model according to claim 1, characterized in that: The embedding module consists of several convolutional layers and maximum pooling layers, the size of the convolutional layer is 3, the fixed step size is 1, and the number of convolutional layers is specified as a multiple of 16.
7. The isochronous model according to claim 1, characterized in that: The relationship module consists of a first convolution block, a second convolution block, a first connection layer and an output layer, the first convolution block and the second convolution block respectively include a convolution layer with 256 filters, the first connection layer and the output layer are fully connected, the first connection layer is 64-dimensional and uses a ReLU activation function, and the output layer is 1-dimensional and uses a Sigmoid activation function.
8. The isochronous model according to claim 7, characterized in that: A dropout layer is provided between the first connection layer and the output layer to alleviate overfitting.
