Few-Shot Open-Set Recognition Method Based on Global-Local Energy Collaborative Representation

Abstract

The present invention discloses a few-shot open-set recognition method with global-local energy collaborative representation, including two classification modules: one for closed-set classification and one for open-set recognition. In addition to classifying closed-set samples using class features, a new energy-based open-set classifier is learned using both class features and pixel features. The global energy score is learned through class features, and the local energy score is learned through pixel features, forcing samples that deviate from known classes in class features or pixel features to be assigned a larger energy score, otherwise a smaller energy score. The present invention improves the few-shot open-set recognition ability of the model by fusing global and local information.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a small-sample open-set recognition method based on global-local energy collaborative representation. Background Art

[0002] In recent years, deep learning has developed vigorously in various fields, and the scale of training data in the closed-world learning setting has been increasing, that is, the training set and the test set share exactly the same categories. However, such a setting often does not hold in many practical applications. This is because it is difficult or expensive to obtain a large amount of labeled data and the models deployed in the open-world environment need to continuously process samples from unknown categories. For example, in the application of deep learning for diagnosing rare diseases, the number of samples is limited. In this case, the model is prone to overfitting, resulting in a significant decline in performance. In addition, due to our limited understanding of these diseases, there may be unknown variants of these diseases. Therefore, the model needs to accurately classify the categories shown by the limited samples while detecting samples from unknown categories. The latter ability is very important, especially for application scenarios such as medical or security. For example, in the disease diagnosis example, it reminds of unknown cases for human investigation, or requests human intervention to handle unknown objects in autonomous driving.

[0003] Few-shot learning and open-set recognition are techniques to solve these two problems respectively. Few-shot learning aims to obtain good generalization ability on new tasks with only a small number of samples for training. However, few-shot learning methods are carried out in a closed environment. It lacks the ability to distinguish categories not seen during training. On the other hand, the goal of open-set recognition is to maintain the classification ability of closed-set samples while identifying open-set samples. However, its classification ability often relies on the availability of a large number of training samples. Therefore, when the number of training samples is small, open-set recognition methods often cannot work effectively.

[0004] Few-shot open-set recognition is a research area that combines few-shot learning and open-set recognition problems. Few-shot open-set recognition requires the model to effectively obtain the ability of closed-set classification and open-set recognition simultaneously using only a small number of training samples. Existing few-shot open-set recognition methods are based on the prototypical networks proposed in the literature "Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning[J]. Advances in neural information processing systems, 2017, 30.", and classify by measuring the distance between the prototype of each class and the features of the query samples. They improve the original closed-set classifier to identify open-set samples by learning an additional open-set class using pseudo open-set samples. However, these methods only consider the channel dimension information of the samples and ignore the spatial dimension information of the samples. In addition, the optimization objectives of few-shot learning and open-set recognition are also different. Therefore, training only with an open-set classifier may limit the performance of these models. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the present invention provides a few-shot open-set recognition method based on global-local energy collaborative representation, including two classification modules: one for closed-set classification and one for open-set recognition. In addition to classifying closed-set samples using class features, a new energy-based open-set classifier is learned using both class features and pixel features. The global energy score is learned through class features, and the local energy score is learned through pixel features, forcing samples that deviate from known classes in class features or pixel features to be assigned a larger energy score, otherwise a smaller energy score. The present invention improves the few-shot open-set recognition ability of the model by fusing global and local information.

[0006] The technical solutions adopted by the present invention to solve its technical problems include the following steps:

[0007] Step 1: For a few-shot open-set recognition task It contains N classes, with K samples in each class as the support set

[0008] Query set Contains two parts Wherein Is the known query set, which contains Q samples in each of the N classes, and the classes Are the same as the support set; Is the unknown query set, which contains N classes from the unknown class set Of each class with Q samples, and

[0009] Each sample consists of a natural image And its corresponding category label y, where h and w represent the height and width of the image respectively;

[0010] First, input the image into the feature extractor Extract features from , where θ is the parameter of the feature extractor, and get the support set images respectively and query collection images Feature Embedding And feature map Among them, the channel dimension of the feature is dim;

[0011]

[0012]

[0013]

[0014]

[0015] Step 2: For the global branch, first calculate the mean value of each category sample of the N support set categories to obtain the category prototype p; specifically: for category n, its category prototype p n The embedding calculation is done by embedding all the supporting features of this class:

[0016]

[0017] Then a self-attention module is used to enhance the category prototypes. Enhanced category prototype matrix P * It is calculated by:

[0018] P q =W q P,P k =W k P,P v =W v P (6)

[0019]

[0020] in, is the coefficient matrix used to linearly map the category prototype matrix. YesP k The channel dimension of

[0021] Finally, the global similarity is obtained by calculating the distance between the feature embedding of the query set and the enhanced category prototype

[0022]

[0023] Among them, is the similarity between and the prototype of category n, and distance(·,·) is a distance function, measured using the Euclidean distance;

[0024] For closed-set classification, only the global similarity s c is used to classify the samples in the known query set by using the softmax function:

[0025]

[0026] The cross-entropy loss is used to optimize the closed-set classification:

[0027]

[0028] Among them, is an indicator function, which is 1 when the condition of condition is met and 0 otherwise; p(y=n|x i ) is the probability that the label of the sample image x m is category n;

[0029] Step 3: For the local branch, first calculate the category mean of the support set feature map; for category n, its category feature map f n is defined as:

[0030]

[0031] The scale calibration module is used to calibrate the feature map of the local branch. Specifically, first use pointwise convolution to halve the channel dimension of the feature, then perform batch normalization on it and activate it using the PReLU activation function;

[0032] f * = PReLU(BN(Conv(f))) (12)

[0033] Finally, for each pixel of the category feature map and the query set feature map, calculate the similarity of the pixel dimension using the cosine similarity; for each pixel of the query set feature map, select the sum of the similarities of the top k closest category feature maps to construct a robust fine-grained metric for open-set recognition;

[0034]

[0035] Among them and They are the feature map of class n and the calibrated feature map of the query set sample r, respectively; m is the spatial dimension of the feature map; ||.|| is applied to the fourth dimension of the feature map; For summing the remaining pixels after calculating topk; T is the temperature coefficient;

[0036] Step 4: For the query set images Its global similarity Is calculated by Step 2, and the local similarity Is calculated by Step 3; First, calculate the energies of the two similarities through the energy function respectively:

[0037]

[0038]

[0039] Add the two energies to obtain the energy of the image:

[0040] E = E c + E f (16)

[0041] Open-set recognition can be performed through the magnitude of the energy. Combining with the closed-set classification performed in Step 2 can complete small-sample open-set recognition;

[0042] For the training of the model, optimize the model using the boundary-based energy loss. Specifically, for a query sample image Its energy loss is:

[0043]

[0044]

[0045] Among them, M k And M u Are the boundaries of the closed-set and open-set samples respectively.

[0046] Preferably, the feature extractor adopts the residual network ResNet12.

[0047] The beneficial effects of the present invention are as follows:

[0048] By fusing global and local information, the present invention improves the small-sample open-set recognition ability of the model. By combining the similarities of the global and local branches, the model can more accurately identify the samples that deviate in class features or pixel features, and assign a larger energy score to them, otherwise assign a smaller energy score, improving the accuracy and robustness of the recognition. Description of the Drawings

[0049] Figure 1This is the structural diagram of the method of the present invention. Specific embodiments

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] As Figure 1 shown, a few-shot open-set recognition method based on global-local energy collaborative representation, characterized by including the following steps:

[0052] Step 1: For a few-shot open-set recognition task which contains N classes with K samples per class as the support set query set contains two parts where is the known query set, which contains Q samples per class of N classes, and the classes are the same as those in the support set. Different from few-shot learning, the query set of few-shot open-set recognition additionally contains as the unknown query set, which contains Q samples per class of N classes from the unknown class set and each sample includes a natural image and its corresponding class label y, where h and w respectively represent the height and width of the image. First, the image is input into the feature extractor to extract features, where θ is the parameter of the feature extractor, and the feature embeddings of the support set pictures and the feature maps of the query set pictures are obtained, where the channel dimension of the features is dim.

[0053]

[0054]

[0055]

[0056]

[0057] Step 2: For the global branch, first calculate the mean of the samples of each class in the N support set classes to obtain the class prototype p. Specifically, for class n, its class prototype p n is calculated through all the support feature embeddings of this class:

[0058]

[0059] Then, a self-attention module is used to enhance the class prototype. For the matrix composed of all class prototypes Enhanced class prototype matrix P * It is calculated as follows:

[0060] P q = W q P, P k = W k P, P v = W v P(6)

[0061]

[0062] Among them, is the coefficient matrix used for linear mapping of the class prototype matrix, is P k 's channel dimension.

[0063] Finally, the global similarity is obtained by calculating the distance between the feature embedding of the query set and the enhanced class prototype

[0064]

[0065] Among them, is the similarity between and the prototype of class n, and distance(·,·) is a distance function, measured using the Euclidean distance.

[0066] For closed-set classification, only the global similarity s c is used to classify the known query set samples by using the softmax function:

[0067]

[0068] The closed-set classification is optimized using the cross-entropy loss:

[0069]

[0070] Among them, is an indicator function, which is 1 when the condition of condition is met, otherwise 0. p(y = n|x m ) is the probability that the label of the sample image x m is class n.

[0071] Step 3: For the local branch, first calculate the class mean of the support set feature map using a method similar to that of the global branch. For class n, its class feature map f n is defined as:

[0072]

[0073] To facilitate the fusion of the distances between the global branch and local branch features on the same scale and reduce the computational complexity, a scale calibration module is first used to calibrate the feature maps of the local branch. Specifically, pointwise convolution is first used to halve the channel dimension of the features, followed by batch normalization and activation using the PReLU activation function.

[0074] f * = PReLU(BN(Conv(f))) (12)

[0075] Finally, for each pixel of the class feature map and the query set feature map, the cosine similarity is used to calculate the similarity of their pixel dimensions. For each pixel of the query set feature map, the similarities of the top k closest class feature maps are selected and summed to construct a robust fine-grained metric for open-set recognition.

[0076]

[0077] where and are the calibrated feature maps of the feature map of class n and the query set sample r, respectively; m is the spatial dimension of the feature map; ||·|| is applied to the fourth dimension of the feature map; is used to sum the remaining pixels after calculating topk; T is the temperature coefficient.

[0078] Step 4: For the query set image its global similarity is calculated in Step 2, and the local similarity is calculated in Step 3. First, the energies of the two similarities are calculated through an energy function.

[0079]

[0080]

[0081] Since the similarities have been calibrated using the scale calibration module, only the two energy scores need to be simply added to obtain the energy of the image:

[0082] E = E c + E f (16)

[0083] Open-set recognition can be performed based on the magnitude of the energy. Combining with the closed-set classification performed in Step 2, few-shot open-set recognition can be completed. For the training of the model, the model is optimized using a boundary-based energy loss. Specifically, for a query sample image its energy loss is:

[0084]

[0085]

[0086] Among them, M k and M u are the boundaries of the closed-set and open-set samples respectively. Specific embodiments:

[0088] 1. Data preprocessing

[0089] For a given dataset sampling is performed for the few-shot open-set recognition task therein. For a few-shot open-set recognition task it contains K samples per class of N classes as the support set query set contains two parts where is the known query set, which contains Q samples per class of N classes, and the classes are the same as those in the support set. Different from few-shot learning, the few-shot open-set recognition query set additionally contains as the unknown query set, which contains Q samples per class of N classes from the unknown class set , and each sample includes a natural image and its corresponding class label y, where h and w represent the height and width of the image respectively.

[0090] In addition, data augmentation methods such as random cropping, random horizontal flipping, and color jittering are adopted for the input images of training to enhance the generalization ability of the model.

[0091] 2. Model structure and initialization

[0092] The present invention adopts a metric-based few-shot learning architecture prototype network, and the feature extractor adopts the residual network ResNet12. To improve the closed-set classification ability of the present invention, a self-attention module is added on the basis of the original prototype network to enhance the class prototypes.

[0093] P q = W q P, P k = W k P, P v = W v P (19)

[0094]

[0095] Among them, is the coefficient matrix for linearly mapping the class prototype matrix, is the channel dimension of P k of P

[0096] In addition, to improve the open-set recognition ability, the present invention additionally adds a local branch, which includes a scale calibration module and a metric module. The scale metric module is used to make the distance between the global branch and the local branch features at the same scale for easy fusion and reduce the computational amount. Specifically, first, pointwise convolution is used to halve the channel dimension of the features, then batch normalization is performed on it, and the PReLU activation function is used for activation. The metric module uses cosine similarity to calculate the similarity of the feature image pixel dimension, and sums the similarities of the top k closest class feature maps to construct a robust fine-grained metric for open-set recognition.

[0097] 3. Model pre-training

[0098] Since there is only a small amount of data in the support set for each task, directly using the meta-learning method to train the network will cause the model to overfit. Therefore, first, the feature extractor of the model is pre-trained using the batch training method with the entire training set to better perform few-shot open-set detection. Specifically, a classifier is added after the feature extractor to supervise the training results of the network. That is, the pre-training of the model is to solve the following optimization problem:

[0099]

[0100]

[0101] where is the classification loss, is the number of training samples in each batch, and θ f are the parameters of the feature extractor.

[0102] After obtaining the parameters of the feature extractor, to better perform few-shot learning, a self-attention module is added, and the training set data is pre-trained using the meta-learning method. Sampling is performed in the way of few-shot learning to better perform closed-set classification. That is, the meta-learning pre-training of the model is to solve the following optimization problem:

[0103]

[0104]

[0105] where is the number of training samples in each few-shot meta-learning task, and θ s are the parameters of the self-attention module.

[0106] 4. Training process

[0107] The above-trained model only has the ability of few-shot learning and does not have the ability of open-set recognition. On the basis of the above model, the global and local energy modules proposed by the present invention are added, and other parameters of the model are fine-tuned under the condition that the learning rate of the feature extractor is kept low. The model fine-tuning is also trained using the meta-learning method, and sampling is performed in the way of few-shot open-set recognition during sampling to improve the few-shot open-set recognition performance of the model. That is, the model fine-tuning is to solve the following optimization problem:

[0108]

[0109]

[0110] where, is the number of training samples for each few-shot open-set recognition task, θ gel is the parameter of the global and local energy modules, is the boundary-based energy loss, and λ is a hyperparameter used to balance the two losses.

Claims

1. A few-shot open-set recognition method based on global-local energy collaborative representation, characterized in that It includes the following steps: Step 1: For a small-sample open-set recognition task It contains N classes, with K samples in each class as the support set Query set It contains two parts Among them is a known query set, which contains N types with Q samples in each type, and the categories are the same as those in the support set; is an unknown query set, which contains N categories from the unknown category set and has Q samples in each category, and Each sample includes a natural image and its corresponding class label y, where h and w represent the height and width of the image, respectively; First, input the image into the feature extractor to extract features, where θ are the parameters of the feature extractor, and obtain the feature embeddings of the support set images and the query set images respectively, as well as the feature maps Among them, the channel dimension of the features is dim;​ Step 2: For the global branch, first calculate the mean of each class of samples in the N support sets to obtain the class prototype p; specifically: for class n, its class prototype p n is calculated through all the support feature embeddings of this class: Then, a self-attention module is used to enhance the class prototypes. For the matrix composed of all class prototypes The enhanced class prototype matrix P * is calculated in the following way: P q = W q P, P k = W k P, P v = W v P (6) Among them, is the coefficient matrix for linearly mapping the class prototype matrix, is P k 's channel dimension; Finally, the global similarity is obtained by calculating the distance between the feature embedding of the query set and the enhanced class prototypes. Among them, is the similarity between and the prototype of category n, and distance(·,·) is a distance function measured using the Euclidean distance; For closed set classification, only the global similarity s c is used to classify the known query set samples by using the softmax function: Optimize the closed-set classification using cross-entropy loss: Among them, is an indicator function, which results in 1 when the condition of condition is met, and 0 otherwise; p(y = n|x i ) is the probability that the label of the sample picture x m is the category n; Step 3: For local branches, first calculate the class mean for the support set feature map; for class n, its class feature map f n is defined as: Use a scale calibration module to calibrate the feature maps of the local branch. Specifically, first use pointwise convolution to halve the channel dimension of the features, then perform batch normalization on it and activate it using the PReLU activation function; f * = PReLU(BN(Conv(f))) (12) Finally, for each pixel of the class feature map and the query set feature map, calculate the similarity of the pixel dimension using cosine similarity; for each pixel of the query set feature map, select the top k closest similarities of the class feature map and sum them to construct a robust fine-grained metric for open-set recognition; where and are the feature map of class n and the calibrated feature map of the query set sample r, respectively; m is the spatial dimension of the feature map; ||.|| is applied to the fourth dimension of the feature map; ∑ pixel · is used to sum the remaining pixels after calculating topk; T is the temperature coefficient; Step 4: For the query set of images Its global similarity is calculated in Step 2, and the local similarity is calculated in Step 3; First, calculate the energies of the two similarities respectively through the energy function: Add the two energies to obtain the energy of the image: E = E c + E f (16) Open-set recognition can be performed through the magnitude of the energy, and combined with the closed-set classification performed in step 2, small-sample open-set recognition can be completed; For the training of the model, the model is optimized using a boundary-based energy loss. Specifically, for a query sample image its energy loss is as follows: where M k and M u are the boundaries of the closed - set and open - set samples, respectively.

2. A few-shot open-set recognition method based on global-local energy collaborative representation according to claim 1, characterized in that The feature extractor uses the residual network ResNet12.

Images