A small sample face recognition method based on prototype calibration and adaptive fine tuning

Abstract

The application discloses a small sample face recognition method based on prototype calibration and adaptive fine-tuning, comprising the following steps: 1) pre-training the encoder on a large-scale face database, capturing rich semantic information and class distribution, constructing an evaluation set, and ensuring that the training identity information does not overlap with the constructed evaluation set; 2) initializing the classifier weight through the weight imprint method, ensuring smooth transition, and inducing the discriminative structure of the encoder; 3) calculating the cosine similarity between the fine-tuning class prototype and the pre-training prototype, fusing the pre-training class information through weighted average, and calibrating the biased fine-tuning class prototype; 4) introducing an adaptive margin loss function, adjusting the sample penalty strength, and selectively fine-tuning the BatchNorm layer and the classifier weight; and 5) using the field-aware similarity NAC to calibrate the unknown class rejection, enhancing the similarity relationship with the field sample, and obtaining the final matching score. The method can improve the recognition accuracy and rejection ability of the model in the small sample learning scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of face recognition, and particularly relates to a small sample face recognition method based on prototype calibration and adaptive fine-tuning. BACKGROUND

[0002] In recent years of deep learning research, face recognition technology has made significant progress, especially in large-scale face datasets pre-trained deep neural network encoders, which enable the model to extract highly discriminative face features. These pre-trained models demonstrate strong generalization ability in various scenarios by capturing rich semantic information and class distribution. However, although pre-trained models perform well on large-scale datasets, when faced with small sample or small amount of data of new classes, existing technologies still have some obvious shortcomings:

[0003] (1) Existing face recognition methods rely on a large amount of labeled data. Traditional deep learning methods require a large amount of labeled training data to achieve good recognition performance, but in practical applications, it is often difficult to collect enough labeled face data, especially when it comes to new identities or classes. In the small sample scenario, the model cannot effectively learn the distinguishing features of new classes, resulting in a significant decline in classification performance.

[0004] (2) Existing fine-tuning strategies often destroy the feature expression ability of pre-trained models. When fine-tuning on new tasks, directly fine-tuning all layers or parameters of pre-trained models can cause overfitting, especially in the case of extremely small training samples. This overfitting problem is particularly evident. In addition, adjusting all parameters during the fine-tuning process affects the original discriminative structure of the model, causing the rich semantic information and inter-class relationships of the pre-trained model to be destroyed, affecting the model's performance on new tasks.

[0005] (3) Existing loss functions have limited ability to distinguish small sample classes. Most standard loss functions (such as cross-entropy loss) cannot adequately adjust the inter-class and intra-class distance when faced with new classes, especially when faced with data-scarce classes, they cannot provide an effective class prototype calibration mechanism. This causes the feature representation of new classes to deviate, further reducing recognition accuracy.

[0006] (4) Existing technologies have deficiencies in handling unknown class rejection problems. In open set face recognition tasks, the model not only needs to recognize known classes, but also needs to reject misidentification when unknown classes appear. Existing methods usually rely on a fixed similarity threshold, however, this strategy is difficult to adjust flexibly according to the distribution of different scenarios or neighborhood samples, resulting in low accuracy of unknown class rejection.

[0007] Therefore, for small sample face recognition, the existing technology still has a large room for improvement in feature calibration of new categories, model fine-tuning, and unknown category rejection, and there is an urgent need for a small sample face recognition method that can smoothly transition the pre-trained model to new categories and adaptively adjust the classification decision to meet complex needs in small sample data environment. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art and provide a small sample face recognition method based on prototype calibration and adaptive fine-tuning, which has high recognition accuracy and good rejection recognition ability for open set unknown faces.

[0009] To achieve the above purpose, the present application adopts the following technical solutions.

[0010] A small sample face recognition method based on prototype calibration and adaptive fine-tuning, comprising the following steps:

[0011] Step S1, pre-training the encoder on a large-scale face database to capture rich semantic information and class distribution, constructing an evaluation set to ensure that the training identity information does not overlap with the constructed evaluation set;

[0012] Defining an encoder The encoder is pre-trained on a large-scale face database, so that its identity information does not overlap with the subsequent constructed evaluation set; the evaluation set is composed of a gallery and a probe set Q, wherein the probe set Q is further divided into a known probe set and an unknown probe set The gallery G and the known probe set K have no overlapping images x, but they share the same identity label y∈{1,...,C}, while the unknown probe set U has a label that does not overlap with the known identity, i.e. Fixing the number of images corresponding to each identity information as 3;

[0013] Step S2, initializing the classifier weight by the weight imprinting method to ensure smooth transition and induce the discriminative structure of the encoder;

[0014] Using the weight imprinting method to initialize the weight of the classifier as the weight of the pre-trained class prototype corresponding to the fine-tuned class prototype, and the specific formula expression is:

[0015]

[0016] In formula (1), w c is the weight vector of the c-th class; represents the prototype of the c-th class obtained by calculation; φ(x) is the embedding function of the encoder, which is used to convert the input sample x into a feature representation; Labels representing samples in the image library; Let m represent the input feature of the i-th sample in the image library; m is the number of samples belonging to the c-th class; ||·||2 is the l2 norm, which is unit normalized for the embedded feature φ(x), so ||φ(x)||2=1;

[0017] Step S3: Calculate the cosine similarity between the fine-tuned prototype and the pre-trained prototype, and fuse the pre-trained class information by weighted averaging to calibrate the biased fine-tuned prototype.

[0018] Step S4: Introduce an adaptive interval loss function, adjust the sample penalty intensity, and selectively fine-tune the weights of the BatchNorm layer and the classifier.

[0019] Step S5: In the evaluation phase, Domain Aware Similarity (NAC) is used to calibrate the rejection of unknown categories, enhance the similarity relationship with domain samples, and obtain the final matching score.

[0020] Specifically, the prototype mentioned in step S3 is defined as follows: Suppose there is class B in the pre-training dataset and class C in the fine-tuning stage. The prototype for calibrating the deviation in the fine-tuning dataset is given as a new prototype after calibration, represented as:

[0021]

[0022] In formula (2), w n (B≤n≤B+C-1) is the prototype of the fine-tuning class; For the new class prototype after calibration; Δw n It is a component of the pre-trained class; α is the hyperparameter that controls the calibration strength of the bias prototype. A larger α means that the calibrated prototype retains more of the original bias prototype, while a smaller α means that the calibrated prototype incorporates more of the pre-trained class prototype.

[0023] Furthermore, the formula for calculating the cosine similarity between the fine-tuned prototype and the pre-trained prototype in step S3 is as follows:

[0024]

[0025] In formula (3), τ (τ>0) is the scaling hyperparameter that controls the sharpness of the weight distribution; S b，n The similarity between the prototypes of the pre-trained class and the fine-tuned class; w b This is the prototype of the pre-trained class; w n This is the prototype for the fine-tuning class;

[0026] The process involves fusing pre-trained class information through weighted averaging to calibrate the fine-tuning class prototype, and for each fine-tuning class, only S is selected. b，nThe top_k pre-trained classes are used for calibration. The weights of the fine-tuned class prototype relative to the pre-trained class prototype are given by the softmax results on the top_k pre-trained classes, as expressed in the following formula:

[0027]

[0028] The deviation calibration formula for the fine-tuning prototype is expressed as follows:

[0029]

[0030] Specifically, the formula for the adaptive interval loss function AML in step S4 is as follows:

[0031]

[0032] In formula (6), Cosine similarity of positive samples; cosθ j is the cosine similarity of the negative samples; cos z is the maximum cosine similarity among all negative samples; m is the margin of the positive samples; m′ is the adaptive margin of the negative samples; β is the adaptive weight of the positive samples; s is the scaling factor; t is the threshold used to control the margin of the negative samples; a and mul are hyperparameters used to adjust the margin of the negative samples and the weight of the positive samples.

[0033] Furthermore, in step S4, the selective fine-tuning of the BatchNorm layer and classifier weights is first defined as θ, and all parameters of the model are divided into two parts, θ BN For the BatchNorm layer parameters, θ rest For other network parameters, during fine-tuning, θ rest Fixed, it does not participate in gradient update, only updates θ. BN We use AML as the loss function to measure the difference between the model's predicted output and the true label, and optimize the parameters of the BatchNorm layer and the classifier weights.

[0034] Specifically, step S5 involves using Domain Aware Similarity (NAC) to calibrate the rejection of unknown categories and enhance the similarity relationship with domain samples. The formula for Domain Aware Similarity (NAC) is as follows:

[0035]

[0036] In formula (7), N k It is the set of indices of the k nearest gallery prototypes to the probe feature p; 1 [i∈N] k] is the indicator function; the evaluation uses the detection and recognition rate DIR@FAR under a certain false alarm rate; when DIR@FAR=1, it means that no probe samples are rejected and only the DIR of the rank-1 recognition rate is considered; that is, DIR@FAR=1 represents the accuracy of rank-1 closed set recognition.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The method of this invention captures rich semantic information and class distribution by pre-training on a large-scale face database, and initializes the classifier weights through the weight imprinting method. This mechanism effectively avoids the problem of model overfitting in small sample situations, thereby improving the accuracy of small sample recognition.

[0039] 2. The method of this invention effectively integrates weighted imprinting technology with pre-trained class information to induce the discriminative structure of the encoder, ensuring a smooth transition from a large-scale pre-trained model to a small-sample task model. This not only preserves the rich semantic information and inter-class relationships of the pre-trained model but also prevents the feature destruction problem common in traditional fine-tuning strategies, making the feature structure more robust.

[0040] 3. The method of this invention calculates the cosine similarity between the fine-tuned class prototype and the pre-trained class prototype, and uses a weighted average to fuse the pre-trained class information to calibrate the class prototype bias in the fine-tuning process. This process can effectively reduce the class representation bias caused by small samples, ensure that the feature representation of the new class is more robust and accurate, and thus enhance the model's ability to distinguish new categories.

[0041] 4. The method of this invention introduces an adaptive interval loss function to adjust the sample penalty intensity, thereby achieving selective fine-tuning of the weights of the BatchNorm layer and the classifier. This adaptive fine-tuning mechanism allows the model to flexibly adjust the weights according to the sample distribution, avoiding the overfitting problem that may be caused by global fine-tuning, and further improving the generalization performance in small sample tasks.

[0042] 5. In the evaluation phase, the method of this invention introduces neighborhood-aware similarity (NAC) to calibrate the rejection problem of unknown categories. By enhancing the similarity relationship between the model and neighborhood samples, the model can more accurately distinguish between known and unknown categories, effectively reducing the false recognition rate and improving the rejection capability in open set tasks. This mechanism is particularly prominent in open set face recognition tasks. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the small-sample face recognition method based on prototype calibration and adaptive fine-tuning of the present invention;

[0045] Figure 2 This is a model architecture diagram of the small-sample face recognition method based on prototype calibration and adaptive fine-tuning of the present invention. Detailed Implementation

[0046] To facilitate understanding and implementation of the present invention by those skilled in the art, the various steps of the method proposed in this invention are described in detail below. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various modifications or alterations to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0047] Example

[0048] like Figure 1 and Figure 2 As shown, this invention discloses a few-sample face recognition method based on prototype calibration and adaptive fine-tuning, comprising the following steps:

[0049] Step S1: Pre-train the encoder on a large-scale face database to capture rich semantic information and class distribution, construct an evaluation set, and ensure that the training identity information does not overlap with the constructed evaluation set.

[0050] Define an encoder The encoder is pre-trained on a large-scale face database to ensure that its identity information does not overlap with the subsequently constructed evaluation set; the evaluation set consists of a database of images. It consists of a probe set Q, which is further divided into known probe sets. and unknown probe set Image library G and known probe set K have no overlapping images x, but they share the same identity labels y∈{1,...,C}, while unknown probe set U has labels that do not overlap with known identities, i.e. In this implementation, the number of images corresponding to each identity information is fixed at 3, which satisfies the constraint of few-shot learning;

[0051] Step S2: Initialize the classifier weights using the weight imprinting method to ensure a smooth transition and induce the discriminative structure of the encoder;

[0052] The weights of the classifier are initialized using the weight imprinting method to the weights of the pre-trained class prototype corresponding to the fine-tuned class prototype. The specific formula is as follows:

[0053]

[0054] In formula (1), w c Let be the weight vector for class c; φ(x) represents the prototype of the c-th class obtained after calculation; φ(x) is the embedding function of the encoder, which is used to convert the input sample x into a feature representation; Labels representing samples in the image library; Let m represent the input feature of the i-th sample in the image library; m is the number of samples belonging to the c-th class; ||·||2 is the l2 norm, which is unit normalized for the embedded feature φ(x), so ||φ(x)||2=1;

[0055] Step S3: Calculate the cosine similarity between the fine-tuned prototype and the pre-trained prototype, and fuse the pre-trained class information by weighted averaging to calibrate the biased fine-tuned prototype.

[0056] Step S4: Introduce an adaptive interval loss function, adjust the sample penalty intensity, and selectively fine-tune the weights of the BatchNorm layer and the classifier.

[0057] Step S5: In the evaluation phase, Domain Aware Similarity (NAC) is used to calibrate the rejection of unknown categories, enhance the similarity relationship with domain samples, and obtain the final matching score.

[0058] Specifically, the prototype mentioned in step S3 is defined as follows: Assume the pre-training dataset has class B, and the fine-tuning stage has class C. In this embodiment, the processed pre-training dataset Vggface2 is used as an example, where B = 8749, and the fine-tuning dataset CASIA has class C = 5287. The empirical prototype of the i-th class can be represented as w i Therefore, the prototype of the pre-trained class is w. b (0≤b≤B-1), the prototype of the fine-tuning class is w. n (B≤n≤B+C-1); Since the pre-training dataset contains sufficient classes and samples, the model trained on this dataset can capture the distribution of the pre-training classes well and obtain well-calibrated pre-training class prototypes. Based on the analysis of the prototypes, prototypes with these deviations are calibrated only in the fine-tuning dataset, thus giving a new calibrated class prototype, which is represented as:

[0059]

[0060] In formula (2), w n (B≤n≤B+C-1) is the prototype of the fine-tuning class; For the new class prototype after calibration; Δw n The calibration term is a component of the pre-trained class; α is the hyperparameter that controls the calibration intensity of the bias prototype. A larger α indicates that the calibrated prototype retains more of the original bias prototype, while a smaller α means that the calibrated prototype incorporates more of the pre-trained class prototype.

[0061] Furthermore, the formula for calculating the cosine similarity between the fine-tuned prototype and the pre-trained prototype in step S3 is as follows:

[0062]

[0063] In formula (3), τ (τ>0) is the scaling hyperparameter that controls the sharpness of the weight distribution; S b，n The similarity between the prototypes of the pre-trained class and the fine-tuned class; w b This is the prototype of the pre-trained class; w n This is the prototype for the fine-tuning class;

[0064] The process involves fusing pre-trained class information through weighted averaging to calibrate the fine-tuning class prototype, and for each fine-tuning class, only S is selected. b，n The top_k pre-trained classes are used for calibration. The weights of the fine-tuned class prototype relative to the pre-trained class prototype are given by the softmax results on the top_k pre-trained classes, as expressed in the following formula:

[0065]

[0066] The deviation calibration formula for the fine-tuning prototype is expressed as follows:

[0067]

[0068] Specifically, the formula for the adaptive interval loss function AML in step S4 is as follows:

[0069]

[0070] In formula (6), Cosine similarity of positive samples; cosθ j is the cosine similarity of the negative samples; cos z is the maximum cosine similarity among all negative samples; m is the margin of the positive samples; m′ is the adaptive margin of the negative samples; β is the adaptive weight of the positive samples; s is the scaling factor; t is the threshold used to control the margin of the negative samples; a and mul are hyperparameters used to adjust the margin of the negative samples and the weight of the positive samples.

[0071] Furthermore, in step S4, the selective fine-tuning of the BatchNorm layer and classifier weights is first defined as θ, and all parameters of the model are divided into two parts, θ BN For the BatchNorm layer parameters, θ rest For other network parameters, during fine-tuning, θ rest Fixed, it does not participate in gradient update, only updates θ. BN We use AML as the loss function to measure the difference between the model's predicted output and the true label, and optimize the parameters of the BatchNorm layer and the classifier weights.

[0072] Specifically, step S5 involves using Domain Aware Similarity (NAC) to calibrate the rejection of unknown categories and enhance the similarity relationship with domain samples. The formula for Domain Aware Similarity (NAC) is as follows:

[0073]

[0074] In formula (7), N k It is the set of indices of the k nearest gallery prototypes to the probe feature p; 1 [i∈N] k `DIR@FAR` is an indicator function; it evaluates the detection and recognition rates (DIR@FAR) under a certain false alarm rate. When `DIR@FAR = 1`, it means that no probe samples are rejected, and only the rank-1 recognition rate is considered; that is, `DIR@FAR = 1` represents the accuracy of rank-1 closed-set recognition. For example, `DIR@FAR = 0.01` represents the recognition rate with a false alarm rate of 0.01 (i.e., a 1% false alarm rate). This means that out of 100 probe samples, only 1 sample may be incorrectly accepted as a known identity, and the model's recognition accuracy under this strict condition is high. `DIR@FAR = 10.0` represents the recognition rate with a false alarm rate of 10.0 (i.e., a 10% false alarm rate). This means that out of 100 probe samples, at most 10 samples may be incorrectly accepted as a known identity, and the model's recognition accuracy under this condition is high. `DIR@FAR = 100.0` represents the recognition rate with a false alarm rate of 100.0 (i.e., a 100% false alarm rate). In this scenario, all probe samples are accepted as known identities, and no samples are rejected.

[0075] The superiority of the method of the present invention will be further illustrated by a specific experimental comparison below.

[0076] Experimental setup: This experiment uses VGG19 and ResNet-50 as encoders, with a feature dimension of 512, and uses CosFac... e Loss function in WebFac ePre-training was performed on a large dataset, with a fixed training duration of 20 epochs, a batch size of 128, and an Adam optimizer with cosine annealing. The initial learning rate was set to 1e-3. Data augmentation included random horizontal flipping and random pruning, with pruning ratios ranging from 0.7 to 1.0. In prototype calibration and loss function configuration, the hyperparameters (mul, a, t) of the AML and the prototype calibration parameters were selected with optimal values ​​based on different encoder-dataset combinations, and the optimal configuration of these parameters was determined during fine-tuning.

[0077] Experimental results: Table 1 below shows the comparison results of the recognition accuracy and rejection capability of the method of the present invention and the traditional method in the few-shot learning scenario. The results show that the method of the present invention is more effective in improving the recognition accuracy and rejection capability of the model in the few-shot learning scenario.

[0078] Table 1. Comparison of recognition accuracy and rejection rate in few-shot learning scenarios

[0079]

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A few-sample face recognition method based on prototype calibration and adaptive fine-tuning, characterized in that, Includes the following steps: Step S1: Pre-train the encoder on a large-scale face database to capture rich semantic information and class distribution, construct an evaluation set, and ensure that the training identity information does not overlap with the constructed evaluation set. Step S2: Initialize the classifier weights using the weight imprinting method to ensure a smooth transition and induce the discriminative structure of the encoder; Step S3: Calculate the cosine similarity between the fine-tuned prototype and the pre-trained prototype, and fuse the pre-trained class information by weighted averaging to calibrate the biased fine-tuned prototype. The prototype is defined as follows: Suppose the pre-training dataset contains... Class, fine-tuning stage has The class, which is the prototype for calibrating bias in the fine-tuning dataset, is represented as follows, given a new calibrated class prototype: （2）； In formula (2), For fine-tuning the prototype of the class, ; This is the prototype of the new class after calibration; It is a component of pre-trained classes; To control the calibration intensity of the hyperparameter-controlled deviation prototype, a larger This indicates that the calibrated prototype retains more of the original deviation prototype, with smaller deviations. This means that the calibrated prototype incorporates more pre-trained prototypes; The formula for calculating the cosine similarity between the fine-tuned prototype and the pre-trained prototype is as follows: （3）； In formula (3), The scaling hyperparameter controls the sharpness of the weight distribution. ; The similarity between the prototypes of the pre-trained class and the prototypes of the fine-tuned class; This serves as the prototype for pre-trained classes; This is the prototype for the fine-tuning class; The process involves fusing pre-trained class information through weighted averaging to calibrate the fine-tuning class prototype, selecting only the appropriate class prototype for each fine-tuning class. The top_k pre-trained classes are used for calibration. The weights of the fine-tuned class prototype relative to the pre-trained class prototype are given by the softmax results on the top_k pre-trained classes, as expressed in the following formula: （4）； The deviation calibration formula for the fine-tuning prototype is expressed as follows: （5）； Step S4: Introduce an adaptive interval loss function, adjust the sample penalty intensity, and selectively fine-tune the weights of the BatchNorm layer and the classifier. The adaptive interval loss function AML is expressed as follows: （6）； In formula (6), Cosine similarity of positive samples; Cosine similarity for negative samples; The maximum value among all negative samples' cosine similarity; The margin of the positive sample; Adaptive margins for negative samples; Adaptive weights for positive samples; This is the scaling factor; The threshold is used to control the margin of negative samples; and mul are hyperparameters used to adjust the negative sample margins and positive sample weights; Step S5: In the evaluation phase, Domain Aware Similarity (NAC) is used to calibrate the rejection of unknown categories, enhance the similarity relationship with domain samples, and obtain the final matching score.

2. The small-sample face recognition method based on prototype calibration and adaptive fine-tuning according to claim 1, characterized in that, The specific details of step S1 are as follows: Define an encoder φ, which is pre-trained on a large-scale face database to ensure that its identity information does not overlap with the subsequently constructed evaluation set; the evaluation set consists of a graph database. and detection set Composition, including the detector set Further divided into known detection sets and unknown probe set Image library and known probe set Non-overlapping images But they share the same identity tags And unknown probe set Having labels that do not overlap with known identities, i.e. The number of images corresponding to each identity information is fixed at 3.

3. The small-sample face recognition method based on prototype calibration and adaptive fine-tuning according to claim 1, characterized in that, The specific details of step S2 are as follows: The weights of the classifier are initialized using the weight imprinting method to the weights of the pre-trained class prototype corresponding to the fine-tuned class prototype. The specific formula is as follows: （1）； In formula (1), Let be the weight vector for class c; This represents the calculated first... The prototype of the class; is the embedding function of the encoder, used to convert the input sample x into a feature representation; Labels representing samples in the image library; This represents the input feature of the i-th sample in the image library; It is the number of samples belonging to class c; for Norm, for embedded features It is a unit normalization, with .

4. The small-sample face recognition method based on prototype calibration and adaptive fine-tuning according to claim 1, characterized in that, Step S4 describes the selective fine-tuning of the BatchNorm layer and classifier weights. First, the network parameters are defined as θ, and all model parameters are divided into two parts. For BatchNorm layer parameters, For other network parameters, during fine-tuning, Fixed, does not participate in gradient update, only updates We use AML as the loss function to measure the difference between the model's predicted output and the true label, and optimize the parameters of the BatchNorm layer and the classifier weights.

Images