Face recognition method, device and computer-readable storage medium

Abstract

The present application discloses a face recognition method, device, and computer-readable storage medium, which relate to the field of image processing technology. The method comprises: inputting a face image into a face feature extraction network to extract features and obtain intermediate layer features and face features; discretizing the intermediate layer features to obtain a discretization result; masking the discretization result to obtain a masked result, and inputting the masked result into a preset prediction network for prediction to obtain a prediction result; calculating the confidence level corresponding to the face image based on the discretization result and the prediction result, wherein the smaller the difference between the discretization result and the prediction result, the greater the confidence level; and performing face recognition based on the confidence level corresponding to the face image and the face features. The present application provides a face recognition solution to avoid the situation where the face recognition algorithm produces erroneous recognition results and improve the overall effect of the face recognition system.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a face recognition method, device, and computer-readable storage medium. Background Art

[0002] Facial recognition technology extracts features from the facial region and verifies identity through comparison. Facial recognition algorithms and products are relatively mature. However, due to the limited robustness of deep learning algorithms, as well as the black-box problem and unexplainability, the performance of facial recognition algorithms in practice is affected by various factors, such as environmental factors (light intensity, over / underexposure, rain or snow), imaging device factors (lens distortion, image clarity), and individual factors (posture, gender, occlusion, etc.). Under the influence of these factors, facial recognition algorithms may produce erroneous recognition results, thus affecting the overall effectiveness of the facial recognition system. Summary of the Invention

[0003] The main purpose of this application is to provide a face recognition method, device and computer-readable storage medium, aiming to propose a face recognition solution to avoid the situation where the face recognition algorithm produces erroneous recognition results and improve the overall effect of the face recognition system.

[0004] To achieve the above objectives, the present application provides a face recognition method, which includes the following steps:

[0005] Inputting the first facial image into a facial feature extraction network for feature extraction, obtaining first intermediate layer features output by at least one network layer in a backbone network and first facial features output by a feature extraction head, wherein the facial feature extraction network includes the backbone network and the feature extraction head;

[0006] Discretizing the first intermediate layer features to obtain a first discretization result;

[0007] performing masking on the first discretization result to obtain a first mask result, and inputting the first mask result into a preset prediction network for prediction to obtain a first prediction result, wherein the preset prediction network is trained using the same training data set as the facial feature extraction network and is used to predict values ​​of mask positions to restore the discretization result after the masking process;

[0008] Calculating a first confidence level corresponding to the first facial image based on the first discretization result and the first prediction result, wherein the smaller the difference between the first discretization result and the first prediction result, the greater the first confidence level;

[0009] Perform face recognition based on the first confidence level and the first facial features corresponding to the first facial image.

[0010] Optionally, the step of discretizing the first intermediate layer features to obtain a first discretization result includes:

[0011] The first intermediate layer features are vector quantized using a preset codebook, and a first discretization result is obtained according to the result of the vector quantization, wherein the preset codebook is trained using the same training data set as the facial feature extraction network.

[0012] Optionally, the preset codebook includes a plurality of prototype vectors, and the step of performing vector quantization on the first intermediate layer feature using the preset codebook and obtaining a first discretization result according to a result of the vector quantization includes:

[0013] Dividing the first intermediate layer features according to the channel dimension to obtain multiple first feature vectors;

[0014] For any target first eigenvector among the multiple first eigenvectors, respectively calculating the distance between the target first eigenvector and the multiple prototype vectors;

[0015] Obtaining a vector quantized result corresponding to the target first eigenvector according to a prototype vector closest to the target first eigenvector among the multiple prototype vectors;

[0016] The first discretization result is obtained according to the vector quantization results respectively corresponding to the multiple first eigenvectors.

[0017] Optionally, the step of obtaining a vector quantization result corresponding to the target first eigenvector according to a prototype vector having the shortest distance to the target first eigenvector among the multiple prototype vectors includes:

[0018] The prototype vector closest to the target first eigenvector among the multiple prototype vectors is used as the result of vector quantization corresponding to the target first eigenvector;

[0019] Or, the prototype vector among the multiple prototype vectors that is closest to the target first eigenvector is used as the target prototype vector, and the preset index number corresponding to the target prototype vector is used as the result of vector quantization corresponding to the target first eigenvector, wherein the multiple prototype vectors respectively correspond to different preset index numbers.

[0020] Optionally, before the step of performing vector quantization on the first intermediate layer features using a preset codebook and obtaining a first discretization result according to the vector quantization result, the step further includes:

[0021] Obtaining a plurality of second feature vectors obtained by dividing second intermediate layer features corresponding to the second facial image according to a channel dimension, wherein the second intermediate layer features are obtained by inputting the second facial image in a training data set into the trained facial feature extraction network for feature extraction, or by inputting the second facial image into the facial feature extraction network to be trained for feature extraction;

[0022] For any target second eigenvector among the plurality of second eigenvectors, respectively calculating distances between the target second eigenvector and a plurality of to-be-trained vectors in the to-be-trained codebook;

[0023] The to-be-trained vector having the shortest distance to the target second eigenvector among the plurality of to-be-trained vectors is used as the target to-be-trained vector corresponding to the target second eigenvector;

[0024] The codebook to be trained is updated according to the target vectors to be trained corresponding to the multiple second eigenvectors respectively, and based on the updated codebook to be trained, the step of obtaining the multiple second eigenvectors obtained by dividing the second intermediate layer features corresponding to the second facial image according to the channel dimension is returned to be executed until a preset training stop condition is met, and the preset codebook is obtained based on the updated codebook to be trained.

[0025] Optionally, the step of updating the codebook to be trained according to the target vectors to be trained corresponding to the multiple second eigenvectors respectively includes:

[0026] Calculating a quantization loss corresponding to the second facial image based on errors between the plurality of second feature vectors and the corresponding target training vectors;

[0027] Calculating a total quantization loss corresponding to the to-be-trained codebook according to the quantization loss corresponding to each second face image in the training data set;

[0028] The to-be-trained codebook is updated according to the total quantization loss.

[0029] Optionally, before returning to execute the step of obtaining a plurality of second feature vectors obtained by dividing the second intermediate layer features corresponding to the second facial image according to the channel dimension based on the updated codebook to be trained, the method further includes:

[0030] The face feature extraction network to be trained is updated according to the total quantization loss and the face recognition loss corresponding to the face feature extraction network.

[0031] Optionally, the step of updating the codebook to be trained according to the target vectors to be trained corresponding to the multiple second eigenvectors respectively includes:

[0032] For any target vector among the plurality of vectors to be trained, counting the total number of second eigenvectors of the target vector to be trained as of the current round of update process, and counting the cumulative vector of the target vector to be trained as the second eigenvector of the target vector to be trained;

[0033] The accumulated vector is divided by the total number to obtain an updated vector of the target vector in the current round of updating process, so as to update the codebook to be trained.

[0034] Optionally, the step of calculating a first confidence level corresponding to the first facial image based on the first discretization result and the first prediction result includes:

[0035] After performing multiple masking processes on the first discretization result to obtain multiple first prediction results corresponding to the first discretization result, respectively calculating similarities between the first discretization result and the multiple first prediction results;

[0036] The similarities are averaged to obtain a first confidence level corresponding to the first facial image.

[0037] Optionally, the step of performing face recognition based on the first confidence level and the first facial feature corresponding to the first facial image includes:

[0038] If the first confidence level is less than a preset threshold, a face recognition result indicating that the first face image fails to be recognized is obtained, or the first face image is deleted from a preset face database.

[0039] Optionally, the step of performing face recognition based on the first confidence level and the first facial feature corresponding to the first facial image includes:

[0040] Comparing the first facial feature corresponding to the first facial image with a third facial feature corresponding to a third facial image in a preset face database to obtain a first comparison result;

[0041] performing weighted processing on the first comparison result using the first confidence level and a third confidence level corresponding to the third facial image in the face database to obtain a second comparison result;

[0042] A face recognition result corresponding to the first face image is determined according to second comparison results respectively corresponding to the first face image and each of the third face images in the face database.

[0043] To achieve the above objectives, the present application further provides a face recognition device, comprising:

[0044] a feature extraction module, configured to input the first facial image into a facial feature extraction network for feature extraction, and obtain first intermediate layer features output by at least one network layer in a backbone network and first facial features output by a feature extraction head, wherein the facial feature extraction network includes the backbone network and the feature extraction head;

[0045] a discretization processing module, configured to discretize the first intermediate layer features to obtain a first discretization result;

[0046] a mask prediction module, configured to perform masking on the first discretization result to obtain a first mask result, and input the first mask result into a preset prediction network for prediction to obtain a first prediction result, wherein the preset prediction network is trained using the same training data set as the facial feature extraction network and is configured to predict values ​​of mask positions to restore the discretization result after masking;

[0047] a confidence calculation module, configured to calculate a first confidence level corresponding to the first facial image based on the first discretization result and the first prediction result, wherein the smaller the difference between the first discretization result and the first prediction result, the greater the first confidence level;

[0048] A face recognition module is used to perform face recognition based on the first confidence level and the first face feature corresponding to the first face image.

[0049] To achieve the above-mentioned purpose, the present application also provides a face recognition device, which includes: a memory, a processor, and a face recognition program stored in the memory and runnable on the processor. When the face recognition program is executed by the processor, the steps of the face recognition method described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also proposes a computer-readable storage medium, on which a face recognition program is stored. When the face recognition program is executed by a processor, the steps of the face recognition method described above are implemented.

[0051] In an embodiment of the present application, a first facial image is input into a facial feature extraction network for feature extraction, and a first intermediate layer feature output by at least one network layer in the backbone network and a first facial feature output by a feature extraction head are obtained, wherein the facial feature extraction network includes a backbone network and a feature extraction head; a first discretization result is obtained by discretizing the first intermediate layer feature, so as to first discretize the continuous space and represent the original continuous space with a sequence; then, the first discrete result is predicted based on a pre-trained preset prediction network to obtain a first prediction result, so as to fit the distribution of the discrete space in a sequence-based learning manner; and since the same data as that used in the training of the facial feature extraction network is used when training the preset prediction network, the prediction result of the preset prediction network always tends to fall within the distribution, so the degree to which the first facial feature falls within the distribution can be evaluated by the difference between the first prediction result and the first discrete result, that is, the first confidence level corresponding to the first facial feature can be obtained.

[0052] Because the first confidence level reflects the degree of deviation of the first facial feature from the spatial distribution of facial features of each facial image in the training dataset, the first confidence level can represent the quality of the first facial feature. The higher the first confidence level, the higher the quality of the first facial feature, that is, the more accurate the first facial feature. The lower the first confidence level, the lower the quality of the first facial feature, that is, the less accurate the first facial feature. Using the first confidence level and the first facial feature, which can represent the quality of the first facial feature, for face recognition can avoid false positives in the face recognition system by filtering out low-quality facial features, thereby improving the overall effectiveness of the face recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of an embodiment of the face recognition method of the present application;

[0054] Figure 2 A schematic diagram of a face recognition process involved in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of a face recognition process involved in an embodiment of the present application;

[0056] Figure 4 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of this application.

[0057] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0059] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the face recognition method of the present applicant.

[0060] The embodiments of the present application provide embodiments of the face recognition method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than here. In this embodiment, the execution subject of the face recognition method can be a personal computer, a smart phone, a server and other devices. This is not limited in this embodiment, and the execution subject is omitted below to explain each embodiment. In this embodiment, the face recognition method includes the following steps:

[0061] Step S10: Input the first facial image into a facial feature extraction network for feature extraction to obtain a first intermediate layer feature output by at least one network layer in a backbone network and a first facial feature output by a feature extraction head, wherein the facial feature extraction network includes the backbone network and the feature extraction head.

[0062] In face recognition application scenarios, it is necessary to perform feature extraction on the face image to be identified to obtain the facial features corresponding to the face image, and then compare the facial features of the face image with the facial features of each face image in the face database. The identity corresponding to the face image in the face database that has a high similarity with the face image is used as the identity obtained by the face image recognition.

[0063] In this embodiment, the first facial image may be a facial image that needs to be recognized, or may be a facial image in a facial database, which is not limited in this embodiment.

[0064] The facial feature extraction network can be a pre-trained network for extracting facial features. The network structure includes a backbone network and a feature extraction head. The backbone network is at the front end of the feature extraction head. That is, after a facial image is input into the facial feature extraction network, it first enters the backbone network for feature extraction. The output of the backbone network then enters the feature extraction head for feature extraction, and the feature extraction head outputs facial features. The backbone network includes at least one network layer. The features output by each network layer are called intermediate layer features, and the features output by the feature extraction head are called facial features. Facial features are used for final comparison. In one feasible embodiment, the intermediate layer features output by the network layer in the backbone network can be in tensor form, and the facial features output by the feature extraction head can be in one-dimensional vector form.

[0065] In this embodiment, the first face image is input into the face feature extraction network for feature extraction, and the face features output by the feature extraction head (hereinafter referred to as the first face features) are obtained. In addition, the intermediate layer features output by at least one network layer in the backbone network are also obtained (hereinafter referred to as the first intermediate layer features for distinction). In a specific embodiment, when there are multiple network layers in the backbone network, the first intermediate layer features output by at least one of the network layers can be obtained. In a feasible embodiment, when there are multiple network layers in the backbone network, the first intermediate layer features output by the last network layer can be obtained. The intermediate layer features output by the last network layer are closest to the face features output by the feature extraction head and can represent the distribution of face features in the face feature extraction network. Therefore, by using the intermediate layer features output by the last network layer as the first intermediate layer features, the overall effect of the face recognition system can be further improved.

[0066] Step S20: discretize the first intermediate layer features to obtain a first discretization result.

[0067] In this embodiment, there is no limitation on the discretization method. For example, in one feasible implementation, the first intermediate layer features can be discretized using vector quantization. The result obtained by discretizing the first intermediate layer features is referred to as the first discretization result for distinction.

[0068] Step S30: mask the first discretization result to obtain a first mask result, and input the first mask result into a preset prediction network for prediction to obtain a first prediction result, wherein the preset prediction network is trained using the same training data set as the facial feature extraction network, and is used to predict the value of the mask position to restore the discretization result after masking.

[0069] After obtaining the first discretization result, a masking process may be performed on the first discretization result. The result obtained by the masking process is referred to as the first masked result for distinction. The purpose of the masking process is to replace some values ​​in the first discretization result with specific values ​​to mask the values.

[0070] The preset prediction network can be pre-trained using the same training dataset as the facial feature extraction network. That is, the facial feature extraction network's training dataset includes multiple facial images, and the preset prediction network is also trained using these multiple facial images. The purpose of training the preset prediction network is to enable it to predict the values ​​of mask positions to restore the discretized results after masking. In this embodiment, the specific training process of the preset prediction network is not limited. For example, in one feasible implementation, the preset prediction network in the training stage is referred to as the prediction network to be trained. The prediction network to be trained can be implemented using networks such as PixelCNN (an autoregressive model), Transformer, Diffusion Model (diffusion model), and there is no limitation in this implementation; multiple face images in the training data set can be respectively subjected to feature extraction using a face feature extraction network to obtain intermediate layer features corresponding to each face image output by at least one network layer in the backbone network, and each intermediate layer feature is discretized to obtain a discretization result, and each discretization result is respectively masked to obtain a masked result, and each masked result is respectively predicted using the prediction network to be trained to obtain a prediction result, and the difference between each discretization result and the corresponding prediction result is calculated, and the prediction loss corresponding to the prediction network to be trained is calculated based on the difference, and the prediction network to be trained is updated based on the prediction loss; after the prediction network to be trained is iteratively updated for multiple rounds in the above manner, the trained prediction network to be trained is used as the preset prediction network.

[0071] The first mask result is input into the preset prediction network for prediction, and the predicted result is referred to as the first prediction result for distinction. It is understood that when the first prediction result output by the preset prediction network is compared with the first mask result, the value of the mask position is predicted.

[0072] Step S40: Calculate a first confidence level corresponding to the first facial image based on the first discretization result and the first prediction result, wherein the smaller the difference between the first discretization result and the first prediction result, the greater the first confidence level.

[0073] After obtaining the first prediction result, a confidence level corresponding to the first facial image can be calculated based on the first discretization result and the first prediction result (hereinafter referred to as the first confidence level for distinction). In this embodiment, the specific calculation method for calculating the first confidence level based on the first discretization result and the first prediction result is not limited and can be set as needed. The calculation method can ensure that the calculated first confidence level and the difference between the first discretization result and the first prediction result satisfy the following relationship: the smaller the difference between the first discretization result and the first prediction result, the greater the first confidence level; conversely, the greater the difference between the first discretization result and the first prediction result, the smaller the first confidence level.

[0074] For example, in one feasible embodiment, the difference between the first discretization result and the first prediction result can be calculated, and the first confidence level can be calculated based on the difference. There are many ways to calculate the difference, which are not limited in this embodiment. For another example, in one feasible embodiment, the similarity between the first discretization result and the first prediction result can be calculated, and the first confidence level can be calculated based on the similarity. There are many ways to calculate the similarity, which are not limited in this embodiment. In a specific embodiment, calculating the difference between the first discretization result and the first prediction result can specifically be: calculating the difference between the first discretization result and the first prediction result as a whole, or calculating the difference between the first discretization result and the numerical value of the mask position in the first prediction result. Accordingly, in a specific embodiment, calculating the similarity between the first discretization result and the first prediction result can specifically be: calculating the similarity between the first discretization result and the first prediction result as a whole, or calculating the similarity between the first discretization result and the numerical value of the mask position in the first prediction result.

[0075] Step S50: performing face recognition based on the first confidence level and the first face feature corresponding to the first face image.

[0076] After obtaining the first confidence level, facial recognition can be performed based on the first confidence level and the first facial features. The specific implementation of facial recognition based on the first confidence level and the first facial features may vary depending on the source of the first facial image and the purpose of facial recognition, and is not limited in this embodiment.

[0077] In one possible implementation, the following Figure 2 The flowchart shown is used to recognize facial images. Figure 2 The CNN (convolutional neural network) on the side close to the face image is used as the backbone network, and the CNN on the side close to the recognition feature (ie, facial feature) is used as the feature extraction head.

[0078] In this embodiment, the confidence level of a facial image is defined as the degree of deviation of the facial features of the facial image from the spatial distribution of the facial features of each facial image in the training dataset of the facial feature extraction network. The following explains why the first confidence level calculated using steps S10 to S40 can represent the degree of deviation of the first facial feature from the spatial distribution of the facial features of each facial image in the training dataset.

[0079] In this embodiment, out of distribution (OOD) in a face recognition algorithm is defined as the degree of deviation of the facial features of a face image from the spatial distribution of the facial features of each face image in the training data set of the face feature extraction network. In this embodiment, the degree of out of distribution is defined as the confidence level of the recognized facial features. The definition of OOD in this embodiment is different from that of OOD in general machine learning tasks.

[0080] In common tasks, data that closely resembles the training data is called in-distribution (ID) data, while data that differs from the training data is called out-of-distribution (OOD) data. There are two general interpretations of data "difference." First, semantic differences, where the test data belongs to a new category compared to the training data, such as training for cat-dog classification and testing for a bird. Second, surface statistical differences, similarly called domain differences or style differences, involve the same categories as in training. For example, training for real-world cat-dog classification and testing for a cartoon dog.

[0081] However, face recognition algorithms in actual applications are open-set tasks. The IDs recognized in real-world scenarios are likely to be different from those in the training data, meaning the categories used during testing are different from those used during training. There are also often significant differences in scene styles. For example, daytime RGB data is used for training, while nighttime RGB data or IR data is used for application. Despite this, it is inappropriate to define the OOD of face recognition directly by category or style, because many face recognition algorithms can work properly in these scenarios, rather than producing completely incorrect results as in the usual OOD definition.

[0082] In actual use, the performance of the face recognition algorithm will be affected by various factors, such as environmental factors (light intensity, over / under exposure, rainy and snowy weather, etc.), imaging equipment factors (lens distortion, clarity, etc.), and individual factors (posture, gender, occlusion, etc.). Under the influence of these factors, the face recognition algorithm will produce false alarms to varying degrees, thereby affecting the overall effect of the face recognition system. At the same time, the preceding steps in the face recognition system may also bring certain erroneous data. For example, directly sending a non-face picture to the face feature extractor will also affect the overall effect of the face recognition system. Therefore, in this embodiment, the definition of OOD is different from the above two definitions in the usual tasks.

[0083] From the perspective of the feature space of the face recognition algorithm, these factors ultimately lead to the facial features extracted from the corresponding face image deviating from the feature space distribution originally defined by the face recognition algorithm. Therefore, in the embodiment of the present application, a definition of out-of-distribution data in the face recognition algorithm is given: due to various factors, the extracted facial features deviate from the feature distribution space during training. This feature is called an out-of-distribution feature, and the corresponding data is called out-of-distribution data. It should be emphasized that in the face recognition algorithm, the result of judging whether a facial feature belongs to an out-of-distribution feature is not a discrete value, that is, it is not a relationship between 0 and 1, but a continuous value, which represents the different degrees of influence of various factors on the face recognition algorithm, that is, it represents the degree of deviation of the facial features from the feature space distribution originally defined by the face recognition algorithm. Therefore, this degree of out-of-distribution can be regarded as the confidence level of the facial feature.

[0084] Since the confidence defined in this embodiment can also be interpreted as the distribution difference between facial features and features within the distribution, and a continuous feature space is difficult to directly represent its distribution, the first discretization result is obtained by discretizing the first intermediate layer features, so as to first discretize the continuous space and use a sequence to represent the original continuous space; then, the first discrete result is predicted based on the pre-trained preset prediction network to obtain the first prediction result, so as to realize the distribution fitting of the discrete space based on sequence learning; and since the same data as the facial feature extraction network is used when training the preset prediction network, the prediction result of the preset prediction network always tends to fall within the distribution, so the difference between the first prediction result and the first discrete result can be used to evaluate the degree to which the first facial feature falls within the distribution, that is, the first confidence corresponding to the first facial feature can be obtained.

[0085] Because the first confidence level reflects the degree of deviation of the first facial feature from the spatial distribution of facial features of each facial image in the training dataset, the first confidence level can represent the quality of the first facial feature. The higher the first confidence level, the higher the quality of the first facial feature, that is, the more accurate the first facial feature. The lower the first confidence level, the lower the quality of the first facial feature, that is, the less accurate the first facial feature. Using the first confidence level and the first facial feature, which can represent the quality of the first facial feature, for face recognition can avoid false positives in the face recognition system by filtering out low-quality facial features, thereby improving the overall effectiveness of the face recognition system.

[0086] In the related art, there are also methods of performing face recognition based on confidence based on the confidence calculated based on facial features. One way to calculate confidence is to calculate confidence from the quality of the facial image itself. The quality of the image itself includes clarity, resolution, noise, blur, overexposure, underexposure, etc., and the confidence calculated in this way is not particularly correlated with the confidence of the facial features. There are cases where the image quality is slightly poor but the confidence of the facial features is high. There are also some ways to calculate confidence, but most of them are based on individual information or local information, such as the convergence of facial features and the relationship between facial features in the face database. Compared with the method of calculating confidence by evaluating the confidence of facial features through the difference in overall distribution, the method of calculating confidence in this embodiment covers more comprehensive information, so that the result of face recognition based on confidence is more accurate and avoids false positives.

[0087] In summary, in this embodiment, a new definition of the confidence of facial features is proposed, and the confidence calculated based on this definition is used to guide face recognition. This method does not rely on human judgment of image quality or the process of facial feature training. It can more accurately and objectively reflect the quality of facial features and improve the recognition effect of the face recognition system in various low-quality scenarios.

[0088] In one feasible implementation, step S50 includes:

[0089] Step S501: If the first confidence level is less than a preset threshold, a face recognition result indicating that the first face image recognition fails is obtained.

[0090] In this embodiment, when the first facial image is a facial image that needs to be face recognized, a feasible implementation method for face recognition based on a first confidence level is proposed. The first confidence level can be compared with a preset threshold value. The preset threshold value can be pre-set as needed and is not limited in this embodiment. When the first confidence level is less than the preset threshold value, it means that the quality of the first facial feature is low and cannot accurately characterize the features of the face in the first facial image. At this time, a face recognition result indicating that the first facial image recognition failed can be obtained, thereby avoiding false alarms caused by feature comparison when the quality of the first facial feature is low, thereby improving the overall effect of the face recognition system. Accordingly, in a feasible implementation method, if the first confidence level is greater than or equal to the preset threshold value, the first facial feature of the first facial image can be compared with the facial features in the face database, and then the face recognition result of the first facial image can be determined based on the comparison result.

[0091] In one feasible implementation, step S50 includes:

[0092] Step S502: If the first confidence level is less than a preset threshold, the first facial image is deleted from a preset facial database.

[0093] In this embodiment, when the first facial image is a facial image in a facial database, a feasible implementation method for performing facial recognition based on a first confidence level is proposed. When the first confidence level is less than a preset threshold, the first facial image can be deleted from the facial database to avoid false positives caused by feature comparison when the quality of the first facial features is low, thereby improving the overall effect of the facial recognition system. Accordingly, in a feasible implementation method, if the first confidence level is greater than or equal to the preset threshold, the first facial image can be retained in the facial database, and the facial features of the first facial image can be added to the facial database, so that the facial features of the facial image to be facial recognized can be compared with the facial features in the facial database, and the facial recognition result can be determined based on the comparison result.

[0094] In one feasible implementation, step S50 includes:

[0095] Step S503: Compare the first facial feature corresponding to the first facial image with a third facial feature corresponding to a third facial image in a preset face database to obtain a first comparison result.

[0096] In this embodiment, when the first facial image is a facial image that requires facial recognition, a feasible embodiment of performing facial recognition based on the first confidence level and facial features is proposed.

[0097] The face database includes multiple facial images, corresponding facial images for each facial image, confidence scores for each facial image, and identities for each facial image. Facial images in the face database are referred to as third facial images for distinction, facial features corresponding to third facial images are referred to as third facial features for distinction, and confidence scores corresponding to third facial images are referred to as third confidence scores for distinction.

[0098] In a specific implementation, the third facial feature and the third confidence level corresponding to the third facial image may be calculated using the method described in steps S10 to S40 above.

[0099] The first facial feature is compared with the third facial feature corresponding to the third facial image, and the result obtained is a first comparison result for differentiation. The comparison method is not limited in this embodiment, and for example, cosine similarity can be calculated.

[0100] Step S504 : performing weighted processing on the first comparison result using the first confidence level and a third confidence level corresponding to the third facial image in the face database to obtain a second comparison result.

[0101] The first confidence level and the third confidence level corresponding to the third facial image are used to perform weighted processing on the first comparison result, and the obtained result is referred to as the second comparison result for distinction.

[0102] Step S505 : determining a face recognition result corresponding to the first face image according to the second comparison results respectively corresponding to the first face image and each of the third face images in the face database.

[0103] After comparing the first facial image with each third facial image in the facial database to obtain second comparison results corresponding to the first facial image and each third facial image, the facial recognition result corresponding to the first facial image can be determined based on each second comparison result. For example, in one feasible embodiment, when the first comparison result indicates similarity, the identity corresponding to the third facial image with the highest similarity to the first facial image can be used as the identity of the face in the first facial image.

[0104] In one feasible implementation, the first facial feature and the third facial feature are represented as f(x1) and f(x2) respectively, and the second comparison result can be calculated as follows:

[0105] sim=C1·C2·cos(f(x1),f(x2))

[0106] Wherein, C1 and C2 represent the first confidence and the third confidence, respectively.

[0107] In this embodiment, the first comparison result is weighted by using the first confidence of the first facial image and the third confidence of the third facial image, so that the second comparison result is affected by the confidence of the two facial images, thereby avoiding the generation of erroneous facial recognition results based on facial features of poor quality.

[0108] Based on the above first embodiment, a second embodiment of the face recognition method of the present applicant is proposed. In this embodiment, step S20 includes:

[0109] In step S201, the first intermediate layer features are vector quantized using a preset codebook, and a first discretization result is obtained according to the result of the vector quantization, wherein the preset codebook is trained using the same training data set as the facial feature extraction network.

[0110] In this embodiment, it is proposed to discretize the first intermediate layer features using vector quantization (VQ).

[0111] The preset codebook (also known as the embedding space) can be pre-trained using the same training dataset as the facial feature extraction network. That is, the facial feature extraction network's training dataset includes multiple facial images, and the preset codebook is also trained using these multiple facial images. The purpose of training the preset codebook is to enable vector quantization of intermediate layer features using the preset codebook. In this embodiment, the specific training process of the preset codebook is not limited.

[0112] The first intermediate layer features are vector quantized using a preset codebook, and a first discretization result is obtained according to the vector quantization result. In this embodiment, there is no limitation on the process of vector quantizing the first intermediate layer features using the preset codebook.

[0113] For example, in one feasible implementation, step S201 includes:

[0114] Step S2011: Divide the first intermediate layer features according to the channel dimension to obtain multiple first feature vectors.

[0115] In this embodiment, the preset codebook includes multiple prototype vectors. The pre-trained preset codebook can be a prototype vector in the preset codebook obtained by pre-training. The number and vector dimension of the prototype vectors in the preset codebook are not limited in this embodiment.

[0116] For example, in one feasible implementation, a discrete latent variable space e∈R can be defined K×D, also known as embedding space, or CodeBook. K is the size of the embedding space, that is, the number of prototype vectors, and D is the number of prototype vectors e i The dimension of D can be equal to the number of channels C of the first intermediate layer feature. Therefore, there are a total of K prototype vectors e i ∈R D , i∈1,2,…K。

[0117] The first intermediate layer features can be divided according to the channel dimension to obtain multiple vectors (called first feature vectors for distinction). For example, the first intermediate layer features are tensor features output by a network layer in the backbone network. The dimensions of this tensor are expressed as C×H×W, where C represents the number of channels. By dividing according to the channel dimension, H×W C-dimensional vectors can be obtained.

[0118] Step S2012: For any target first eigenvector among the multiple first eigenvectors, calculate the distance between the target first eigenvector and the multiple prototype vectors respectively.

[0119] The steps for performing vector quantization on multiple first eigenvectors are the same, so a single first eigenvector is used as an example for illustration, and this first eigenvector is referred to as a target first eigenvector for distinction. For the target first eigenvector, the distances between the target first eigenvector and multiple prototype vectors in a preset codebook can be calculated to obtain the distances between the target first eigenvector and each prototype vector. The method for calculating the distance between vectors is not limited herein; for example, the Euclidean distance between vectors can be calculated.

[0120] Step S2013 , obtaining a vector quantization result corresponding to the target first eigenvector according to a prototype vector that is closest to the target first eigenvector among the multiple prototype vectors.

[0121] According to the calculated distance, a prototype vector with the shortest distance to the target first eigenvector can be selected from multiple prototype vectors, and a vector quantization result corresponding to the target first eigenvector can be obtained based on the prototype vector.

[0122] Step S2014: Obtain the first discretization result according to the vector quantization results corresponding to the multiple first eigenvectors.

[0123] It can be understood that, according to the above-mentioned steps S2012 and S2013, the vector quantization results corresponding to the multiple first eigenvectors can be obtained. According to the vector quantization results corresponding to the multiple first eigenvectors, the first discretization result can be obtained. Specifically, the vector quantization results corresponding to the respective first eigenvectors can be arranged according to the arrangement order of the multiple first eigenvectors to obtain a sequence consisting of the respective vector quantization results, and the sequence is used as the first discretization result. For example, after H×W first eigenvectors are vector quantized, a sequence consisting of H×W vector quantization results can be obtained, and the sequence is used as the first discretization result.

[0124] In one feasible implementation, step S2013 includes:

[0125] Step S20131 : taking the prototype vector closest to the target first eigenvector among the multiple prototype vectors as the result of vector quantization corresponding to the target first eigenvector.

[0126] In this embodiment, after determining the prototype vector closest to the target first eigenvector among multiple prototype vectors, this prototype vector can be directly used as the result of vector quantization corresponding to the target first eigenvector. In this case, the result of vector quantization is also a vector, hereinafter referred to as a quantized vector for distinction. For example, after vector quantization of H×W first eigenvectors, a sequence of H×W quantized vectors can be obtained, which is used as the first discretization result.

[0127] Denote the target first eigenvector as z e (x), the prototype vector closest to the first eigenvector of the target is represented as e k . e The result of vector quantization of (x) is expressed as z q (x), then z q (x) can be calculated as follows:

[0128] z q (x) = Quantize(z e (x))=e k where k = argmin||z e (x)-e i ||2

[0129] In one feasible implementation, step S2013 includes:

[0130] In step S20132, the prototype vector among the multiple prototype vectors that is closest to the target first eigenvector is used as the target prototype vector, and the preset index number corresponding to the target prototype vector is used as the result of vector quantization corresponding to the target first eigenvector, wherein the multiple prototype vectors respectively correspond to different preset index numbers.

[0131] In this embodiment, after determining the prototype vector with the closest distance to the target first eigenvector among multiple prototype vectors, the prototype vector can be referred to as the target prototype vector for distinction. Index numbers can be set in advance for multiple prototype vectors in the preset codebook. For example, if there are 1024 prototype vectors, the index numbers of these 1024 prototype vectors can be 1 to 1024. After determining the target prototype vector corresponding to the target first eigenvector, the index number corresponding to the target prototype vector can be used as the result of vector quantization corresponding to the target first eigenvector. Then, the result of vector quantization at this time is the index number. For example, after vector quantization of H×W first eigenvectors, a sequence of H×W index numbers can be obtained, and the sequence is used as the first discretization result.

[0132] Based on the above second embodiment, a third embodiment of the face recognition method of the present applicant is proposed. In this embodiment, before step S201, the method further includes:

[0133] Step S202: Obtain multiple second feature vectors obtained by dividing the second intermediate layer features corresponding to the second facial image according to the channel dimension, wherein the second intermediate layer features are obtained by inputting the second facial image in the training data set into the trained facial feature extraction network for feature extraction, or by inputting the second facial image into the facial feature extraction network to be trained for feature extraction.

[0134] In this embodiment, a feasible implementation method for training a preset codebook is proposed. A training dataset is used to train the preset codebook and includes multiple facial images. This training dataset is also used to train the facial feature extraction network. The facial images in the training dataset are referred to as second facial images for clarity.

[0135] In a specific implementation, the preset codebook can be trained together with the facial feature extraction network, or the facial feature extraction network can be trained first, and then the preset codebook can be trained separately based on the trained facial feature extraction network.

[0136] The preset codebook training process involves multiple rounds. Each round updates the preset codebook based on the codebook obtained in the previous round of training. Specifically, the prototype vectors in the preset codebook are updated. Before the first round of training begins, the prototype vectors in the preset codebook are initialized randomly or based on experience. The prototype vectors during the training phase are referred to as the "trained vectors" to distinguish them from the preset codebook during the training phase, the "trained codebook" to distinguish them from the "trained codebook" to distinguish them from the "trained facial feature extraction network" to distinguish them from the "trained facial feature extraction network" to distinguish them from the "trained facial feature extraction network" to distinguish them from the "trained" facial feature extraction network. The following explanation uses one round of training as an example.

[0137] It should be noted that in the process of training the preset codebook, multiple second face images will be used in each round of training, but the process of processing each second face image is the same. Therefore, in the following description, in order to clearly indicate that multiple second face images are being processed, it is equivalent to taking the processing process of a second face image as an example for explanation.

[0138] When the preset codebook and the facial feature extraction network are trained together, the second facial image can be input into the latest updated facial feature extraction network to be trained for feature extraction, and the intermediate layer features output by at least one network layer in the backbone network (called second intermediate layer features for distinction) are obtained. The second intermediate layer features are divided according to the channel dimension to obtain multiple second feature vectors. The division method can refer to the process of dividing the first intermediate layer features to obtain multiple second feature vectors, which will not be repeated here.

[0139] When the preset codebook is trained alone, second intermediate-layer features corresponding to the second facial image can be obtained. These second intermediate-layer features are intermediate-layer features output by at least one network layer in the backbone network after the second facial image is input into the trained facial feature extraction network for feature extraction. It should be noted that since the network parameters in the trained facial feature extraction network no longer change, the second intermediate-layer features obtained by inputting the second facial image into the trained facial feature extraction network will not change. Therefore, when the preset codebook is trained alone, it is not necessary to input the second facial image into the trained facial feature extraction network to obtain the second intermediate-layer features in every round of training.

[0140] Step S203 : For any target second eigenvector among the plurality of second eigenvectors, respectively calculate distances between the target second eigenvector and a plurality of to-be-trained vectors in the to-be-trained codebook.

[0141] The processing process for multiple second eigenvectors is the same, so one of the second eigenvectors is used as an example for illustration, and this second eigenvector is referred to as the target second eigenvector for distinction. The distance between the target second eigenvector and each of the training vectors in the most recently updated training codebook is calculated to obtain the distance between the target second eigenvector and each training vector. The method for calculating the distance between vectors is not limited here; for example, the Euclidean distance between vectors can be calculated.

[0142] Step S204 : Using the to-be-trained vector having the shortest distance to the target second eigenvector among the plurality of to-be-trained vectors as the target to-be-trained vector corresponding to the target second eigenvector.

[0143] According to the calculated distance, a training vector with the shortest distance to the target second eigenvector can be selected from multiple training vectors, and the training vector can be used as the target training vector corresponding to the target second eigenvector.

[0144] Step S205: Update the codebook to be trained according to the target vectors to be trained corresponding to the multiple second eigenvectors, and return to step S202 based on the updated codebook to be trained until a preset training stop condition is met, and obtain the preset codebook based on the updated codebook to be trained.

[0145] It can be understood that, according to the above-mentioned steps S203 and S204, target vectors to be trained corresponding to multiple second eigenvectors can be obtained. According to the target vectors to be trained corresponding to the multiple second eigenvectors, the codebook to be trained can be updated. After the codebook to be trained is updated, if the preset training stop condition is not currently met, the codebook to be trained is updated for the next round based on the updated codebook to be trained, that is, step S202 is returned to execution; if the preset training stop condition is met after the codebook to be trained is updated, the updated codebook to be trained can be used as the preset codebook for training completion, and each vector to be trained in the updated codebook to be trained is the prototype vector in the preset codebook. The preset training stop condition can be set as needed, for example, it can be that the number of training rounds reaches a certain number of rounds, or that the training duration reaches a preset duration, etc.

[0146] It should be noted that when the preset codebook and the facial feature extraction network are trained together, before returning to step S202, the facial feature extraction network to be trained needs to be updated, and then based on the updated facial feature extraction network to be trained and the updated codebook to be trained, the process returns to step S202. In this embodiment, there is no limitation on the method for updating the facial feature extraction network to be trained. For example, the facial feature extraction network to be trained can be updated based on the facial recognition loss by calculating the facial recognition loss.

[0147] In this embodiment, the specific implementation method of updating the to-be-trained codebook according to the target to-be-trained vectors corresponding to the plurality of second eigenvectors is not limited and can be specifically configured as needed.

[0148] In this embodiment, the main purpose of CodeBook is to effectively represent the original continuous intermediate layer feature space in a discretized manner. Therefore, when training CodeBook, it is necessary to use the same training data as when training the facial feature extraction network, because the training data of the facial feature extraction network constitutes the ID (in distribution) space by default, and non-training data may have OOD to a certain extent. The confidence of facial features is evaluated by the OOD situation of the features. Therefore, it is necessary to first remember the distribution of IDs and then estimate the difference between the facial image and the ID distribution. In the embodiment of this application, the ID space is represented by CodeBook, and then a prediction network is used to learn the distribution law of the features after VQ, that is, the distribution law of the ID data.

[0149] In one feasible implementation, the step of updating the codebook to be trained according to the target vectors to be trained corresponding to the plurality of second eigenvectors in step S205 includes:

[0150] Step S2051 : Calculate the quantization loss corresponding to the second facial image based on the errors between the plurality of second eigenvectors and the corresponding target vectors to be trained.

[0151] In this embodiment, a feasible implementation method of updating the codebook to be trained by using dictionary learning is proposed. After obtaining the target vectors to be trained corresponding to the multiple second eigenvectors, the errors between the multiple second eigenvectors and the corresponding target vectors to be trained can be calculated to obtain the quantization loss corresponding to the second face image. In this embodiment, the specific calculation method of the quantization loss is not limited, and it can be set as needed so that the smaller the error between the multiple second eigenvectors and the corresponding target vectors to be trained, the smaller the quantization loss. There are many ways to calculate the error between the second eigenvector and the corresponding target vector to be trained, and they are not limited here.

[0152] In one feasible implementation, the L2 error (mean square error) corresponding to a second eigenvector may be calculated as follows:

[0153]

[0154] Here, the second eigenvector is represented as z e (x), the vector to be trained that is closest to the second eigenvector is represented as e k sg represents a gradient blocking operation, which means that the L2 error only updates the CodeBook and does not propagate to the facial feature extraction network. The L2 errors of multiple second eigenvectors are summed or averaged to obtain the quantization loss corresponding to the second facial image.

[0155] Step S2052: Calculate the total quantization loss corresponding to the to-be-trained codebook according to the quantization loss corresponding to each second face image in the training data set.

[0156] According to the same method, the quantization loss corresponding to each second face image in the training data set can be calculated, and the total quantization loss corresponding to the codebook to be trained can be calculated based on the quantization loss corresponding to each second face image. For example, the quantization losses corresponding to each second face image can be added or averaged to obtain the total quantization loss corresponding to the codebook to be trained.

[0157] Step S2053: Update the codebook to be trained according to the total quantization loss.

[0158] The method of updating the codebook to be trained according to the total quantization loss can be to calculate the gradient value corresponding to each to-be-trained vector in the codebook to be trained according to the total quantization loss, and update the codebook to be trained according to the gradient value and each to-be-trained vector.

[0159] In one feasible implementation, before returning to execute step S202 based on the updated codebook to be trained, the method further includes:

[0160] Step S206: updating the face feature extraction network to be trained according to the total quantization loss and the face recognition loss corresponding to the face feature extraction network.

[0161] In this embodiment, when the preset codebook and the facial feature extraction network are trained together, the facial feature extraction network to be trained can also be updated by combining the total quantization loss with the facial recognition loss corresponding to the facial feature extraction network. The calculation method of the facial recognition loss is not limited in this embodiment; for example, it can be calculated using softmax, arcface, cosface, etc. The facial recognition loss can be added to the total quantization loss or weighted averaged to obtain the total loss corresponding to the facial feature extraction network, and the facial feature extraction network can be updated based on the total loss.

[0162] For example, in one feasible implementation, the L2 error (mean square error) corresponding to a second eigenvector may be calculated as follows:

[0163]

[0164] Here, the second eigenvector is represented as z e (x), the vector to be trained that is closest to the second eigenvector is represented as e k sg represents a gradient blocking operation, which means that only the facial feature extraction network is updated based on this L2 error, and it is not transmitted to the CodeBook. The L2 errors of multiple second eigenvectors are added or averaged to obtain the quantization loss corresponding to the second face image. The quantization loss is then added or weighted averaged with the face loss corresponding to the facial feature extraction network to obtain the total loss corresponding to the facial feature extraction network. The facial feature extraction network is then updated based on the total loss.

[0165] In this embodiment, the face feature extraction network is updated by quantizing the loss, which can constrain the output of the face feature extraction network to be consistent with the CodeBook space, so as to avoid large changes in the intermediate layer features (from one prototype vector to another).

[0166] In one feasible implementation, the step of updating the codebook to be trained according to the target vectors to be trained corresponding to the plurality of second eigenvectors in step S205 includes:

[0167] Step S2054: For any target vector among the multiple vectors to be trained, count the total number of second eigenvectors of the target vector to be trained as of the current round of update process, and count the cumulative vector of the target vector to be trained as the second eigenvector of the target vector to be trained.

[0168] In this embodiment, a feasible implementation method of using an exponential moving average (EMA) method to update the codebook to be trained is proposed.

[0169] Since the updating method for each vector to be trained in the training codebook is the same, one vector to be trained is taken as an example for description, and the vector to be trained is called a target vector for distinction.

[0170] For the target vector, the total number of second eigenvectors of the target vector used as the target vector to be trained as of the current round of updates can be counted, including the number of second eigenvectors of the target vector used as the target vector to be trained in the current round and the number of second eigenvectors of the target vector used as the target vector to be trained in the training rounds before the current round. The statistical method is not limited in this embodiment.

[0171] For example, in one feasible implementation, the following calculation method can be used for statistics:

[0172]

[0173] Among them, γ∈(0,1), n i t It represents the number of second eigenvectors that use the target vector as the target vector to be trained among the second eigenvectors corresponding to the face images in the training dataset during the t-th update.

[0174] For the target vector, statistics can also be performed on the target vector as of the current round of updates, using the target vector as the second eigenvector of the target vector to be trained. This cumulative vector is obtained by accumulating the cumulative vectors of the current round of updates using the target vector as the second eigenvector of the target vector to be trained, and accumulating the cumulative vectors of the target vector as the second eigenvector of the target vector to be trained in each training round before the current round. The statistical method is not limited in this embodiment.

[0175] For example, in one feasible implementation, the following calculation method can be used for statistics:

[0176]

[0177] Among them, γ∈(0,1), z i，j t It means that in the second eigenvector corresponding to the face image in the training dataset during the t-th update, the target vector is used as the second eigenvector of the target vector to be trained.

[0178] Step S2055 : Divide the accumulated vector by the total number to obtain the updated vector of the target vector in the current round of updating process, so as to update the codebook to be trained.

[0179] Divide the cumulative vector obtained by statistics by the total number obtained by statistics to obtain the updated vector of the target vector in the current round of update process. After updating each vector to be trained, the update of the codebook to be trained is completed. For example, m i t Divide by N i t Get the target vector after the update in round t: i t .

[0180] Based on the above-mentioned first, second and / or third embodiments, a fourth embodiment of the face recognition method of the present applicant is proposed. In this embodiment, step S40 includes:

[0181] Step S401 : After performing multiple masking processes on the first discretization result to obtain multiple first prediction results corresponding to the first discretization result, similarities between the first discretization result and the multiple first prediction results are respectively calculated.

[0182] In this embodiment, a feasible implementation method of calculating the first confidence based on the first discretization result and the first prediction result is proposed.

[0183] The first discretization result can be subjected to multiple masking processes, with the mask position and / or mask value being different for each masking process. Each masking process produces a first masked result, and each first masked result is input into a preset prediction network for prediction to produce a first prediction result. In other words, the first discretization result corresponds to multiple first prediction results. The number of times the first discretization result is masked is not limited in this embodiment and can be set as needed. The mask position and mask value for each masking process can be randomly generated or pre-set, and are not limited in this embodiment.

[0184] By respectively calculating the similarities between the first discretization result and the plurality of first prediction results, a plurality of similarities can be obtained. In this embodiment, the similarity calculation method is not limited.

[0185] Step S402: averaging the similarities to obtain a first confidence level corresponding to the first facial image.

[0186] The result obtained by averaging the similarities can be used as the first confidence level.

[0187] In a feasible implementation, when the first discretization result is a sequence composed of index numbers, the first prediction result is also a sequence composed of index numbers. For any one of the first prediction results, the first discretization result can be compared with the index number of the same mask position in the first prediction result. If they are the same, it is recorded as 1, and if they are different, it is recorded as 0. The comparison results corresponding to the mask processing are added and divided by the number of mask positions in the mask processing process to obtain the comparison result corresponding to the mask processing. The comparison results corresponding to each mask processing are averaged, and the result obtained is used as the first confidence level. It should be noted that one index number is a mask position, and masking K index numbers in the first discrete result indicates that there are K mask positions. The first confidence level C can be calculated using the following calculation method:

[0188]

[0189] in and They represent the predicted value (from the first prediction result) and the original value (from the first discrete processing result) of the i-th mask position during the j-th mask processing, respectively. It indicates that when the condition is met, it is 1; otherwise, it is 0. N indicates the total number of masking operations, and K indicates the total number of mask positions in each masking operation.

[0190] In one feasible implementation, when the first discretization result is a sequence composed of quantization vectors, the first prediction result is also a sequence composed of vectors. For any one of the first prediction results, the cosine similarity between the first discretization result and the vector of the same mask position in the first prediction result can be calculated, and the cosine similarities of the various mask positions can be added and divided by the number of mask positions in the mask processing to obtain the comparison result corresponding to the mask processing. The comparison results corresponding to the various mask processing are averaged, and the result obtained is used as the first confidence level. It should be noted that a quantization vector is a mask position, and masking the K quantization vectors in the first discrete result indicates that there are K mask positions. The first confidence level C can be calculated using the following calculation method:

[0191]

[0192] in and denote the predicted value (from the first prediction result) and original value (from the first discrete processing result) of the i-th mask position in the j-th masking process, cos(.) denotes the cosine similarity between the two vectors, N denotes the total number of masking processes, and K denotes the total number of mask positions in each masking process.

[0193] In one possible implementation, the following Figure 3The flowchart shown is used to recognize facial images. Figure 3 Mid-sequence prediction network and preset prediction network.

[0194] In addition, an embodiment of the present application further provides a face recognition device, comprising:

[0195] a feature extraction module, configured to input the first facial image into a facial feature extraction network for feature extraction, and obtain first intermediate layer features output by at least one network layer in a backbone network and first facial features output by a feature extraction head, wherein the facial feature extraction network includes the backbone network and the feature extraction head;

[0196] a discretization processing module, configured to discretize the first intermediate layer features to obtain a first discretization result;

[0197] a mask prediction module, configured to perform masking on the first discretization result to obtain a first mask result, and input the first mask result into a preset prediction network for prediction to obtain a first prediction result, wherein the preset prediction network is trained using the same training data set as the facial feature extraction network and is configured to predict values ​​of mask positions to restore the discretization result after masking;

[0198] a confidence calculation module, configured to calculate a first confidence level corresponding to the first facial image based on the first discretization result and the first prediction result, wherein the smaller the difference between the first discretization result and the first prediction result, the greater the first confidence level;

[0199] A face recognition module is used to perform face recognition based on the first confidence level and the first face feature corresponding to the first face image.

[0200] In one feasible implementation, the discretization processing module is further configured to:

[0201] The first intermediate layer features are vector quantized using a preset codebook, and a first discretization result is obtained according to the result of the vector quantization, wherein the preset codebook is trained using the same training data set as the facial feature extraction network.

[0202] In one feasible implementation, the preset codebook includes a plurality of prototype vectors, and the discretization processing module is further configured to:

[0203] Dividing the first intermediate layer features according to the channel dimension to obtain multiple first feature vectors;

[0204] For any target first eigenvector among the multiple first eigenvectors, respectively calculating the distance between the target first eigenvector and the multiple prototype vectors;

[0205] Obtaining a vector quantized result corresponding to the target first eigenvector according to a prototype vector closest to the target first eigenvector among the multiple prototype vectors;

[0206] The first discretization result is obtained according to the vector quantization results respectively corresponding to the multiple first eigenvectors.

[0207] In one feasible implementation, the discretization processing module is further configured to:

[0208] The prototype vector closest to the target first eigenvector among the multiple prototype vectors is used as the result of vector quantization corresponding to the target first eigenvector;

[0209] Or, the prototype vector among the multiple prototype vectors that is closest to the target first eigenvector is used as the target prototype vector, and the preset index number corresponding to the target prototype vector is used as the result of vector quantization corresponding to the target first eigenvector, wherein the multiple prototype vectors respectively correspond to different preset index numbers.

[0210] In one feasible implementation, the face recognition device further includes:

[0211] an acquisition module, configured to obtain a plurality of second feature vectors obtained by dividing second intermediate layer features corresponding to the second facial image according to a channel dimension, wherein the second intermediate layer features are obtained by inputting the second facial image in a training data set into the trained facial feature extraction network for feature extraction, or by inputting the second facial image into the facial feature extraction network to be trained for feature extraction;

[0212] a distance calculation module, configured to calculate, for any target second eigenvector among the plurality of second eigenvectors, a distance between the target second eigenvector and a plurality of to-be-trained vectors in the to-be-trained codebook;

[0213] a determination module, configured to select a to-be-trained vector having the shortest distance to the target second eigenvector among the plurality of to-be-trained vectors as a target to-be-trained vector corresponding to the target second eigenvector;

[0214] An updating module is configured to update the codebook to be trained according to the target vectors to be trained corresponding to the multiple second eigenvectors, and return to execute the operation of obtaining the multiple second eigenvectors obtained by dividing the second intermediate layer features corresponding to the second face image according to the channel dimension based on the updated codebook to be trained, until a preset training stop condition is met, and then obtain the preset codebook based on the updated codebook to be trained.

[0215] In one feasible implementation, the update module is further configured to:

[0216] Calculating a quantization loss corresponding to the second facial image based on errors between the plurality of second feature vectors and the corresponding target training vectors;

[0217] Calculating a total quantization loss corresponding to the to-be-trained codebook according to the quantization loss corresponding to each second face image in the training data set;

[0218] The to-be-trained codebook is updated according to the total quantization loss.

[0219] In one feasible implementation, the update module is further configured to:

[0220] The face feature extraction network to be trained is updated according to the total quantization loss and the face recognition loss corresponding to the face feature extraction network.

[0221] In one feasible implementation, the update module is further configured to:

[0222] For any target vector among the plurality of vectors to be trained, counting the total number of second eigenvectors of the target vector to be trained as of the current round of update process, and counting the cumulative vector of the target vector to be trained as the second eigenvector of the target vector to be trained;

[0223] The accumulated vector is divided by the total number to obtain an updated vector of the target vector in the current round of updating process, so as to update the codebook to be trained.

[0224] In one feasible implementation, the confidence calculation module is further configured to:

[0225] After performing multiple masking processes on the first discretization result to obtain multiple first prediction results corresponding to the first discretization result, respectively calculating similarities between the first discretization result and the multiple first prediction results;

[0226] The similarities are averaged to obtain a first confidence level corresponding to the first facial image.

[0227] In one possible implementation, the face recognition module is further configured to:

[0228] If the first confidence level is less than a preset threshold, a face recognition result indicating that the first face image fails to be recognized is obtained, or the first face image is deleted from a preset face database.

[0229] In one possible implementation, the face recognition module is further configured to:

[0230] Comparing the first facial feature corresponding to the first facial image with a third facial feature corresponding to a third facial image in a preset face database to obtain a first comparison result;

[0231] performing weighted processing on the first comparison result using the first confidence level and a third confidence level corresponding to the third facial image in the face database to obtain a second comparison result;

[0232] A face recognition result corresponding to the first face image is determined according to second comparison results respectively corresponding to the first face image and each of the third face images in the face database.

[0233] In addition, the embodiment of the present application also proposes a face recognition device, such as Figure 4 As shown, Figure 4 It should be noted that the face recognition device in the embodiment of the present application can be a smart phone, a personal computer, a server, etc., and is not specifically limited here.

[0234] like Figure 4 As shown, the face recognition device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0235] Those skilled in the art will understand that Figure 4 The device structure shown in the figure does not constitute a limitation on the face recognition device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0236] like Figure 4 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a face recognition program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the face recognition program and other software or programs. Figure 4In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used to establish a communication connection with the server; and the processor 1001 can be used to call the face recognition program stored in the memory 1005 and execute the face recognition method in the above-mentioned embodiments.

[0237] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which a face recognition program is stored. When the face recognition program is executed by a processor, the steps of the face recognition method described above are implemented.

[0238] The various embodiments of the applicant's face recognition device and computer-readable storage medium can refer to the various embodiments of the applicant's face recognition method, and will not be repeated here.

[0239] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0240] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0241] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0242] The above are only preferred embodiments of the present application and do not limit the scope of protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of protection of the present application.

Claims

1. A face recognition method, characterized in that: The face recognition method comprises the following steps: Inputting the first facial image into a facial feature extraction network for feature extraction, obtaining first intermediate layer features output by at least one network layer in a backbone network and first facial features output by a feature extraction head, wherein the facial feature extraction network includes the backbone network and the feature extraction head; Discretizing the first intermediate layer features to obtain a first discretization result; performing masking on the first discretization result to obtain a first mask result, and inputting the first mask result into a preset prediction network for prediction to obtain a first prediction result, wherein the preset prediction network is trained using the same training data set as the facial feature extraction network and is used to predict values ​​of mask positions to restore the discretization result after the masking process; Calculating a first confidence level corresponding to the first facial image based on the first discretization result and the first prediction result, wherein the smaller the difference between the first discretization result and the first prediction result, the greater the first confidence level; Perform face recognition based on the first confidence level and the first facial features corresponding to the first facial image.

2. The face recognition method according to claim 1, wherein: The step of discretizing the first intermediate layer features to obtain a first discretization result includes: The first intermediate layer features are vector quantized using a preset codebook, and a first discretization result is obtained according to the result of the vector quantization, wherein the preset codebook is trained using the same training data set as the facial feature extraction network.

3. The face recognition method according to claim 2, wherein: The preset codebook includes a plurality of prototype vectors, and the step of performing vector quantization on the first intermediate layer feature using the preset codebook and obtaining a first discretization result according to the result of the vector quantization includes: Dividing the first intermediate layer features according to the channel dimension to obtain multiple first feature vectors; For any target first eigenvector among the multiple first eigenvectors, respectively calculating the distance between the target first eigenvector and the multiple prototype vectors; Obtaining a vector quantized result corresponding to the target first eigenvector according to a prototype vector closest to the target first eigenvector among the multiple prototype vectors; The first discretization result is obtained according to the vector quantization results respectively corresponding to the multiple first eigenvectors.

4. The face recognition method according to claim 3, wherein: The step of obtaining a vector quantization result corresponding to the target first eigenvector according to a prototype vector closest to the target first eigenvector among the multiple prototype vectors includes: The prototype vector closest to the target first eigenvector among the multiple prototype vectors is used as the result of vector quantization corresponding to the target first eigenvector; Or, the prototype vector among the multiple prototype vectors that is closest to the target first eigenvector is used as the target prototype vector, and the preset index number corresponding to the target prototype vector is used as the result of vector quantization corresponding to the target first eigenvector, wherein the multiple prototype vectors respectively correspond to different preset index numbers.

5. The face recognition method according to claim 2, wherein: Before the step of performing vector quantization on the first intermediate layer features using a preset codebook and obtaining a first discretization result according to the vector quantization result, the method further includes: Obtaining a plurality of second feature vectors obtained by dividing second intermediate layer features corresponding to the second facial image according to a channel dimension, wherein the second intermediate layer features are obtained by inputting the second facial image in a training data set into the trained facial feature extraction network for feature extraction, or by inputting the second facial image into the facial feature extraction network to be trained for feature extraction; For any target second eigenvector among the plurality of second eigenvectors, respectively calculating distances between the target second eigenvector and a plurality of to-be-trained vectors in the to-be-trained codebook; The to-be-trained vector having the shortest distance to the target second eigenvector among the plurality of to-be-trained vectors is used as the target to-be-trained vector corresponding to the target second eigenvector; The codebook to be trained is updated according to the target vectors to be trained corresponding to the multiple second eigenvectors respectively, and based on the updated codebook to be trained, the step of obtaining the multiple second eigenvectors obtained by dividing the second intermediate layer features corresponding to the second facial image according to the channel dimension is returned to be executed until a preset training stop condition is met, and the preset codebook is obtained based on the updated codebook to be trained.

6. The face recognition method according to claim 5, wherein: The step of updating the codebook to be trained according to the target vectors to be trained corresponding to the plurality of second eigenvectors comprises: Calculating a quantization loss corresponding to the second facial image based on errors between the plurality of second feature vectors and the corresponding target training vectors; Calculating a total quantization loss corresponding to the to-be-trained codebook according to the quantization loss corresponding to each second face image in the training data set; The to-be-trained codebook is updated according to the total quantization loss.

7. The face recognition method according to claim 6, wherein: Before returning to execute the step of obtaining a plurality of second feature vectors obtained by dividing the second intermediate layer features corresponding to the second facial image according to the channel dimension based on the updated codebook to be trained, the method further includes: The face feature extraction network to be trained is updated according to the total quantization loss and the face recognition loss corresponding to the face feature extraction network.

8. The face recognition method according to claim 5, wherein: The step of updating the codebook to be trained according to the target vectors to be trained corresponding to the plurality of second eigenvectors comprises: For any target vector among the plurality of vectors to be trained, counting the total number of second eigenvectors of the target vector to be trained as of the current round of update process, and counting the cumulative vector of the target vector to be trained as the second eigenvector of the target vector to be trained; The accumulated vector is divided by the total number to obtain an updated vector of the target vector in the current round of updating process, so as to update the codebook to be trained.

9. The face recognition method according to claim 1, wherein: The step of calculating a first confidence level corresponding to the first facial image according to the first discretization result and the first prediction result includes: After performing multiple masking processes on the first discretization result to obtain multiple first prediction results corresponding to the first discretization result, respectively calculating similarities between the first discretization result and the multiple first prediction results; The similarities are averaged to obtain a first confidence level corresponding to the first facial image.

10. The face recognition method according to any one of claims 1 to 9, characterized in that: The step of performing face recognition according to the first confidence level and the first face feature corresponding to the first face image includes: If the first confidence level is less than a preset threshold, a face recognition result indicating that the first face image fails to be recognized is obtained, or the first face image is deleted from a preset face database.

11. The face recognition method according to any one of claims 1 to 9, wherein: The step of performing face recognition according to the first confidence level and the first face feature corresponding to the first face image includes: Comparing the first facial feature corresponding to the first facial image with a third facial feature corresponding to a third facial image in a preset face database to obtain a first comparison result; performing weighted processing on the first comparison result using the first confidence level and a third confidence level corresponding to the third facial image in the face database to obtain a second comparison result; A face recognition result corresponding to the first face image is determined according to second comparison results respectively corresponding to the first face image and each of the third face images in the face database.

12. A face recognition device, characterized in that: The face recognition device includes: a memory, a processor, and a face recognition program stored in the memory and executable on the processor. When the face recognition program is executed by the processor, the steps of the face recognition method according to any one of claims 1 to 11 are implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a face recognition program, which, when executed by a processor, implements the steps of the face recognition method according to any one of claims 1 to 11.

Images