Face recognition method and device, electronic equipment, storage medium and program product
By adding face images that simulate wearing masks to the training data set of the face recognition system, the problem of reduced recognition performance of traditional face recognition systems when wearing masks is solved, and higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202410995579.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2025-01-17
AI Technical Summary
When traditional facial recognition systems wear masks, their recognition performance has significantly decreased, their intra-class differences have increased, and their inter-class separability has decreased.
Based on the existing training data set, a set of face key points is used to simulate the mask generation, a face enhancement image set is generated, and it is jointly trained with the original training data set to generate face recognition for wearing a mask. Model.
The recognition accuracy of the face recognition model in the scene of wearing a mask is improved, and the system performance is improved.
Smart Images
Figure CN119323811A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of biometrics, and in particular to a face recognition method, device, electronic device, storage medium and program product. Background Art
[0002] With the increasing demand for wearing masks worldwide, traditional face recognition systems face challenges because masks block most facial features, leading to problems such as increased intra-class differences and reduced inter-class separability during face recognition, which seriously affects the performance of face recognition systems.
[0003] Therefore, how to improve the accuracy of face recognition for obscured faces is a problem that technical personnel in this field urgently need to solve. Summary of the invention
[0004] The present application provides a face recognition method, device, electronic device, storage medium and program product to solve the technical problem of how to improve the recognition performance of a face recognition system.
[0005] In a first aspect, the present application provides a face recognition method, comprising:
[0006] Obtain a face image to be recognized;
[0007] Outputting the face image to a pre-trained face recognition model to obtain a face recognition result output by the face recognition model; the face recognition model is trained based on a face registration image set, a face image set, a face key point set, and a face enhancement image set; the face enhancement image in the face enhancement image set carries a face mask generated according to the face key point set;
[0008] Output the face recognition result.
[0009] Optionally, the method further includes:
[0010] Acquire the face registration image set, the face image set, and the face key point set;
[0011] Generate the face enhancement image set according to the face registration image set, the face image set, and the face key point set;
[0012] The face recognition model is generated by training the initial model using the face registration image set, the face image set, and the face enhancement image set.
[0013] Optionally, the training an initial model using the face registration image set, the face image set, and the face enhancement image set to generate the face recognition model includes:
[0014] Performing quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence;
[0015] Obtaining a comparison score sequence according to the face registration image set, the face image set, and the face enhanced image set;
[0016] The quality score sequence and the comparison score sequence are input into an initial model for collaborative training to generate the face recognition model.
[0017] Optionally, the step of performing quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence includes:
[0018] According to the face enhanced images in the face enhanced image set, obtaining a similarity set of training positive samples of each face enhanced image and a similarity set of training negative samples of each face enhanced image, wherein the training positive samples are image pairs of the same face and the training negative samples are image pairs of different faces;
[0019] Obtaining error and rejection curves of the face enhanced image set according to the similarity set of the training positive samples, the similarity set of the training negative samples, and a recognition threshold set;
[0020] The quality score sequence is generated according to the error and rejection curve.
[0021] Optionally, generating the quality score sequence according to the error and rejection curve includes:
[0022] Determining, according to the error and rejection curve, a target recognition threshold that meets a preset quality requirement from the recognition threshold set;
[0023] According to the target recognition threshold, obtaining a set of false matching rates and a set of false non-matching rates of the face enhanced image set;
[0024] The quality score sequence is generated according to the set of false matching rates and the set of false non-matching rates.
[0025] Optionally, the step of performing quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence includes:
[0026] According to the joint distribution of the training positive samples and the training negative samples, using a similarity distance to measure the quality difference between the images;
[0027] Based on the mass differences, a quality score sequence is obtained.
[0028] Optionally, the step of inputting the quality score sequence and the comparison score sequence into an initial model for collaborative training to generate the face recognition model includes:
[0029] Generate a feature vector sequence according to the comparison score sequence and the quality score sequence, wherein the feature vector sequence represents the uncertainty of the comparison algorithm and the uncertainty of the face recognition process;
[0030] Using the uncertainty in the feature vector sequence as input data, and inputting it into the initial model for Bayesian enhanced collaborative training;
[0031] The face recognition model is generated according to the training results.
[0032] Optionally, the method further includes:
[0033] Color rendering is performed on the face mask to generate a face mask with at least two colors.
[0034] In a second aspect, the present application provides a face recognition device, comprising:
[0035] An acquisition module, used for acquiring a face image to be recognized;
[0036] A processing module, used for outputting the face image to a pre-trained face recognition model, and obtaining a face recognition result output by the face recognition model; the face recognition model is trained based on a face registration image set, a face image set, a face key point set, and a face enhancement image set; the face enhancement image in the face enhancement image set carries a face mask generated according to the key point set;
[0037] An output module is used to output the face recognition result.
[0038] In a third aspect, the present application provides an electronic device, comprising: a processor, a communication interface, and a memory, wherein the processor is communicatively connected to the communication interface and the memory respectively;
[0039] The memory stores computer-executable instructions;
[0040] The display displays an image;
[0041] The communication interface communicates and interacts with external devices;
[0042] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.
[0044] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first aspects.
[0045] The face recognition method, device, electronic device, storage medium and program product provided by the present application are as follows: obtaining a face image to be recognized; outputting the face image to a pre-trained face recognition model to obtain a face recognition result output by the face recognition model; the face recognition model is trained based on a face registration image set, a face image set, a face key point set, and a face enhancement image set; the face enhancement image in the face enhancement image set carries a face mask generated according to the face key point set; and outputting the face recognition result. In the above method, the training process of the face recognition model for the face wearing condition is completed by jointly training the existing training data set and the face enhancement image set, thereby improving the performance of the face recognition model and improving the accuracy of face recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0047] Figure 1 Schematic diagram of the training method of the face recognition model provided in the embodiment of the present application Figure 1 ;
[0048] Figure 2 A schematic diagram of a face recognition method according to an embodiment of the present invention;
[0049] Figure 3 Schematic diagram of the training method of the face recognition model provided in the embodiment of the present application Figure 2 ;
[0050] Figure 4 Schematic diagram of the training method of the face recognition model provided in the embodiment of the present application Figure 3 ;
[0051] Figure 5 Schematic diagram of the training method of the face recognition model provided in the embodiment of the present application Figure 4 ;
[0052] Figure 6 A schematic diagram of the structure of a face recognition device provided in an embodiment of the present application;
[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0056] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0057] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.
[0058] It should be noted that the face recognition method, device, electronic device, storage medium and program product provided by this application can be used in the field of biometrics, and can also be used in any field other than biometrics. This application does not limit the application field of the face recognition method, device, electronic device, storage medium and program product.
[0059] With the sharp increase in the global demand for wearing masks, especially during public health events, conventional face recognition technology is facing unprecedented challenges. Masks are widely used as a protective measure, but this also brings new problems to face recognition technology. Specifically, when people wear masks, most of the features of the face are blocked, including important recognition areas such as the mouth and nose. This situation causes the face recognition system to have a significant increase in intra-class differences when processing faces wearing masks. That is, the difference in facial features of the same person with and without a mask becomes larger, which increases the difficulty of recognition.
[0060] At the same time, due to the occlusion of masks, the differences in facial features between different people (i.e., inter-class differences) become blurred, making it difficult for the face recognition system to accurately distinguish different individuals. These two problems - the increase in intra-class differences and the decrease in inter-class separability - together lead to a significant decline in the performance of the face recognition system.
[0061] In order to improve the performance of the face recognition system in the mask-wearing scenario, a large number of real face samples wearing masks can be added to the training data set. However, collecting such a large amount of real data requires a lot of manpower, material resources and time resources. On the other hand, for privacy and security reasons, people may be reluctant to provide their facial data, especially when wearing masks. Therefore, although adding real mask face samples to the training set is a theoretically feasible solution, it faces many difficulties and challenges in actual operation. This also requires face recognition technology to find more innovative and efficient solutions in response to the new situation of increasing demand for masks worldwide.
[0062] In view of this, the present application provides a face recognition method, in the process of training the face recognition model used in face recognition, by simulating and generating a mask based on an existing training data set and using a set of key points that can represent facial features to simulate a face image wearing a mask, and generating a face enhancement image set. By jointly training the existing training data set and the face enhancement image set, the training process of the face recognition model for the face wearing condition is completed, thereby improving the accuracy of face recognition using the model.
[0063] The execution subject of the embodiment of the present application can be a computing device or a cloud platform with computing capabilities. When the execution subject is a computing device, the computing device can be a server, a virtual machine on a server, a terminal device, etc. The following takes the execution subject as an example to illustrate how the computing device implements the face recognition method.
[0064] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0065] First, the process of training the face recognition model provided in the embodiment of the present application is introduced.
[0066] Figure 1 The following is a flow chart of the face recognition model training method provided in the embodiment of the present application. Figure 1 As shown, the method may include:
[0067] S101, obtaining a face registration image set, a face image set, and a face key point set.
[0068] The face registration image set is a set of clear frontal face images used when registering a user in the face recognition system. The face image set is a set of face images captured in real scenes. The face images in the face image set may include images of various angles, various expressions, and different lighting conditions. The face registration image set and the face image set may be generated based on face images pre-stored in the database, or may be obtained from other devices (such as image generating devices such as cameras, webcams, or electronic devices such as mobile phones and computers that pre-store face images) or the Internet, and this application does not limit this.
[0069] The face key point set (also called the face image key point set) includes key points corresponding to facial features. For example, these key points can be important locations marked on the face image by a specific algorithm (such as a face key point detection algorithm), such as the locations of feature points such as eyes, nose, and mouth. Alternatively, these key points can also be obtained from an existing non-masked dataset using an open source key point model, such as the key points of 106 face images obtained from MS-Celeb1M, which can be connected to form a face region.
[0070] S102: Generate a face enhancement image set according to the face registration image set, the face image set, and the face key point set.
[0071] Among them, the face enhanced images in the face enhanced image set have face masks generated according to the key point set.
[0072] Based on the facial key point set, multiple ways and angles of wearing a mask on a face can be simulated to generate a face mask simulating a face wearing a mask. By covering the face mask on the face registration image set and / or the face image set, a face enhancement image simulating a face wearing a mask is obtained. These masks are used to cover certain parts of the face so as to introduce occlusion factors during the training process and improve the face recognition model's ability to recognize occlusion situations.
[0073] Optionally, when generating a face mask based on a set of face key points, the number of key points corresponding to the generated face mask can be adjusted to generate face masks of different positions and sizes, thereby simulating more ways of mask occlusion of the face. For example, if only key points around the mouth are selected to generate a face mask, a smaller mask occlusion can be simulated, covering only the mouth part, and a corresponding face enhancement image can be generated. Alternatively, if more key points including the nose and mouth are selected, a larger mask can be generated to simulate a more comprehensive mask occlusion and generate a corresponding face enhancement image.
[0074] Optionally, the face enhancement images included in the face enhancement image set can be further enriched by various data enhancement technologies such as rotation, scaling, and color transformation. Exemplarily, the face mask can be color rendered to generate a face mask of at least two colors to simulate the situation of wearing masks of different colors to cover the face, thereby improving the applicable scenarios and recognition accuracy of the face recognition model.
[0075] S103, collaboratively training an initial model using the face registration image set, the face image set, and the face enhancement image set to generate the face recognition model.
[0076] By simultaneously inputting the face registration image set, face image set, and face enhancement image set into the initial model, comprehensive feature learning is performed. The model is trained with the face registration image set to capture clear, positive facial features. Combined with the face image set, the model can adapt to the changing real-world influencing factors. Based on the augmented image set of humans who simulate wearing masks, the model's recognition ability for partially occluded faces is enhanced. During the training process, by fusion and complementation of features between multiple data sets, the face recognition model can more comprehensively learn and recognize facial features in various scenarios.
[0077] The collaborative training method may be, for example, Bayesian collaborative training, or other collaborative training methods that can be used to train a face recognition model, and this application does not impose any limitation on this.
[0078] A possible implementation method is to determine the degree of fluctuation of the quality score and the degree of fluctuation of the comparison score according to the quality score of the face enhancement image in the face enhancement image set, the comparison score between the face registration image and the face enhancement image, and the comparison score between the face live image and the face enhancement image. By co-training the degree of fluctuation of the quality score and the degree of fluctuation of the comparison score, the recognition parameters in the initial model are adjusted to generate a face recognition model that can recognize face images wearing masks with higher performance.
[0079] The method provided in the embodiment of the present application simulates and generates a mask based on an existing training data set using a set of key points that can represent facial features to simulate a face image of a person wearing a mask and generate a face enhancement image set. By jointly training the existing training data set and the face enhancement image set, the training process of the face recognition model for the case of wearing a mask is completed, thereby improving the performance of the face recognition model.
[0080] The face recognition method provided in the embodiment of the present application is introduced below.
[0081] Figure 2 A flow chart of a face recognition method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, including:
[0082] S201: Obtain a face image to be recognized.
[0083] The computing device may obtain the facial image to be recognized input from the outside, or obtain the facial image to be recognized from an internal preset space.
[0084] S202: Output the facial image to a pre-trained facial recognition model to obtain a facial recognition result output by the facial recognition model.
[0085] The face recognition model is obtained by collaborative training based on a face registration image set, a face image set, a face key point set, and a face enhancement image set; the face enhancement images in the face enhancement image set carry face masks generated according to the face key point set. The process of training the face recognition model can be found in Figure 1 As shown, no further description is given here.
[0086] When the computing device obtains the face image to be identified, it can call the face recognition model to identify the face image to be identified to obtain the face recognition result.
[0087] S203: Output the face recognition result.
[0088] The face recognition method provided in the embodiment of the present application can effectively improve the accuracy of face recognition for faces with obstructed faces.
[0089] exist Figure 1 Based on the illustrated embodiment, a detailed description is given of how to input the face registration image set, the face image set, and the face enhancement image set into the initial model for collaborative training to generate a face recognition model in the aforementioned step S103. Figure 3 A flowchart of another face recognition model training method provided in an embodiment of the present application. Figure 3 As shown, the aforementioned step S103 may include:
[0090] S301. Perform quality evaluation on face-enhanced images in a face-enhanced image set to obtain a quality score sequence.
[0091] Among them, the quality score sequence is used to effectively screen out high-quality face enhancement images from the face enhancement image set for model training, so as to reduce the impact of noise and interference factors during model training, thereby improving the training efficiency and accuracy of the model. The quality score sequence can also be used to evaluate model performance, and understand the model's recognition ability for images of different qualities through the performance of the model on images with different quality scores.
[0092] One possible implementation method is to input the face enhanced image set into a quality assessment model to generate a quality score set corresponding to the face enhanced image set, and generate a quality score sequence corresponding to the face enhanced image set based on the quality score set.
[0093] Another possible implementation method is to use a pre-trained deep learning model (such as a convolutional neural network model) to extract features of face enhanced images, and train a regression model for predicting image quality scores based on the extracted features to obtain a quality score sequence corresponding to the face enhanced image set.
[0094] Taking the input of a face enhancement image set into a quality assessment model as an example, specifically, the present application can generate a quality score sequence through the following steps S3011 to S3013:
[0095] S3011. According to the face-enhanced images in the face-enhanced image set, obtain a similarity set of training positive samples of each face-enhanced image and a similarity set of training negative samples of each face-enhanced image.
[0096] Among them, the training positive samples are image pairs of the same face, that is, the face images corresponding to the image pair belong to the same person; the training negative samples are image pairs of different faces, that is, the face images corresponding to the image pair do not belong to the same person.
[0097] By calculating the similarity between the image pairs of the training positive samples of each face enhanced image, a similarity set of the training positive samples is generated; by calculating the similarity between the image pairs of the training negative samples of each face enhanced image, a similarity set of the training negative samples is generated. The similarity between each pair of images can be calculated by a similarity measurement method (such as cosine similarity, Euclidean distance, etc.). The similarity value of the positive sample pair is higher than the similarity value of the negative sample pair, for example, the similarity value of the positive sample pair is higher than the first similarity threshold, the similarity value of the negative sample pair is lower than the second similarity threshold, and the first similarity threshold is greater than or equal to the second similarity threshold.
[0098] For example, assuming that X represents the feature representation of the face registration image set, Y represents the feature representation of the face image set, and F represents the feature representation of the face enhancement image set, then the training data set D of the model training method can be represented as a triple data set as shown in the following formula (1):
[0099] D= {(x i ,y i ,f(x i ),…,(x n ,y n .f(x n )) )} (1)
[0100] Among them, x i is the i-th face registration image in the face registration image set, y i is the face image of the ith person in the face image set, f(x i ) is the i-th face enhanced image in the face enhanced image set. At this time, the face enhanced image is generated by simulating and adding a face mask according to the corresponding face registration image.
[0101] Based on formula (1), the training positive sample can be expressed as The training negative sample can be expressed as where x j is the jth face image. When the i-th face image and the j-th face image are the same person, they are positive training samples, i.e., y i =y j ; When the i-th face scene image and the j-th face scene image are not the face images of the same person, they are training negative samples, that is, y i ! =y j .in, <f(x i ),f(x j )> represents the similarity score between the i-th face enhanced image in the face enhanced image set and the j-th face enhanced image in the face enhanced image set.
[0102] Based on the above similarity scores, it can be assumed that and Respectively represent the similarity sets of training positive samples and training negative samples, represents the set of positive samples that are less than the recognition threshold, Represents the set of negative samples that are greater than the recognition threshold.
[0103] S3012. Obtain error and rejection curves of the face enhancement image set according to the similarity set of the training positive samples, the similarity set of the training negative samples, and the recognition threshold set.
[0104] After obtaining the similarity set of positive and negative samples, a recognition threshold set can be combined to evaluate the recognition performance of the face enhanced image set. By continuously adjusting the recognition threshold, the Error-Reject Curve (EVRC) can be obtained. EVRC describes the relationship between the error rate and rejection rate of the face recognition model under different recognition thresholds.
[0105] Specifically, when we increase the recognition threshold, the system will more strictly judge whether two images are of the same face, which will cause more legitimate users to be rejected (i.e., the false rejection rate (False No-Match Rate, FNMR) increases), but at the same time it will also reduce the possibility of falsely accepting illegal users (i.e., the false recognition rate (False Match Rate, FMR) decreases).
[0106] S3013. Generate a quality score sequence based on the error and rejection curve.
[0107] Based on the error and rejection curve, a quality score can be generated for each face enhancement image. This quality score reflects the reliability and recognition performance of the face image in the face recognition model. Images with high quality scores have higher confidence in face recognition.
[0108] A possible implementation method is to determine a target recognition threshold that meets the preset quality requirements from a recognition threshold set through an error and rejection curve, and calculate the FMR set and FNMR set of the face enhanced image set based on the target recognition threshold, thereby generating a quality score sequence.
[0109] Another possible implementation is to measure the quality difference between images by using a similarity distance (for example, Wasserstein distance), and generate a quality score sequence according to the quality difference.
[0110] S302: Obtain a comparison score sequence according to the face registration image set, the face image set, and the face enhancement image set.
[0111] A face registration image is selected from the face registration image set as a reference, and then it is compared with the corresponding face enhancement image in the face enhancement image set, and the similarity between the two images is calculated to obtain the comparison score of the two images. By performing the above steps on all the face registration images in the face registration image set, a comparison score sequence of the face registration image set and the face enhancement image set is obtained.
[0112] A face live image is selected from the face image set as a reference, and then compared with the corresponding face enhanced image in the face enhanced image set, and the similarity between the two images is calculated to obtain the comparison score of the two images. By performing the above steps on all face live images in the face image set, a comparison score sequence of the face image set and the face enhanced image set is obtained.
[0113] The method for calculating the similarity between the two images may use any current algorithm for calculating image similarity, for example, a similarity algorithm for calculating the similarity of facial features of facial images, which is not limited in the present application.
[0114] For example, assume that the face registration image set is {x1,x2,x3,…,x n}, the face image set is {y1,y2,y3,…,y n}, then the comparison score sequence between the face registration image set and the face enhancement image set is F1 = {s 11 ,s 12 ,s 13 ,..,s 1n}, the comparison score sequence of the face image set and the face enhanced image set is F2 = {s 21 ,s 22 ,s 23 ,..,s 2n}. The length of the quality score sequence is twice that of each comparison score sequence.
[0115] S303: Input the quality score sequence and the comparison score sequence into the initial model for collaborative training to generate a face recognition model.
[0116] The fluctuation of the comparison score in the comparison score sequence can indirectly reflect the uncertainty of the initial model for face recognition, and the fluctuation of the quality score in the quality score sequence can reflect the uncertainty of the initial model for face recognition when the face mask is added. The quality score sequence and the comparison score sequence are combined to generate a feature vector sequence, and the uncertainty of the feature vector sequence is used as the input of the collaborative training and input into the initial model for collaborative training. The initial model can be a deep learning network, such as a convolutional neural network model.
[0117] During the collaborative training process, the model continuously adjusts its parameters to minimize recognition errors. Through collaborative training, the model can learn how to make more accurate recognition decisions based on the quality of the image and its similarity to other images. After training, the face recognition model not only considers the quality factors of the image, but also combines the similarity information between images, thereby improving the accuracy and robustness of face recognition.
[0118] Next, how to generate the quality score sequence according to the FMR set and the FNMR set in the aforementioned step S3013 is described in detail. Figure 4 A flowchart of another face recognition model training method provided in an embodiment of the present application. Figure 4 As shown, including:
[0119] S401 , determining a target recognition threshold that meets a preset quality requirement from a recognition threshold set according to an error and rejection curve.
[0120] Since the error and rejection curve is intended to test the FNMR of the highest face quality subset with σ percentage under a fixed FMR, and when the FMR is fixed, the recognition threshold needs to be constantly changed to keep the highest face quality subset with σ percentage unchanged. Therefore, the recognition threshold corresponding to the point can be determined from the corresponding recognition threshold set on the error and rejection curve according to the preset quality requirements, such as the point that meets the preset FMR and / or preset FNMR quality requirements, and the recognition threshold is the target recognition threshold.
[0121] S402: Obtain a set of false matching rates FMR and a set of false non-matching rates FNMR of the face enhanced image set according to the target recognition threshold.
[0122] According to the error and rejection curve, as well as the target recognition threshold, the face enhanced image set is tested for face recognition. During the test, all false matches (i.e., faces of different people are recognized as the same person) are recorded and the FMR is calculated. At the same time, all false non-matches (i.e., faces of the same person are recognized as different people) are recorded and the FNMR is calculated. Thus, the false match rate FMR set and the false non-match rate FNMR set of the face enhanced image set are obtained.
[0123] S403: Generate a quality score sequence according to the FMR set and the FNMR set.
[0124] According to the EVRC standard, image quality can be represented by the gradient of FNMR attenuation. If a sample is deleted from the test set, the lower the attenuation of FNMR, the lower the image quality.
[0125] For each face enhanced image, a quality score can be calculated based on its FMR and FNMR values. When FMR and FNMR are lower, the quality score is higher.
[0126] Exemplarily, referring to the example of the above embodiment, FMR can be calculated by the following formula (2), and FNMR can be calculated by the following formula (3):
[0127]
[0128] Among them, ∈ is the recognition threshold mentioned above, that is, the similarity threshold for dividing training positive samples and training negative samples.
[0129] Assumptions is the subset with the highest face quality with σ percentage. The error and rejection curves are designed to test σ and subsets with fixed FMR. When the FMR is fixed, the recognition threshold ∈ needs to be constantly changed to maintain constant.
[0130] According to the error and rejection curve metric, the image quality can be expressed using the gradient of the FNMR attenuation. If a sample x i The lower the FNMR attenuation is when the image is deleted from the test set, the lower the image quality of the image is. Therefore, the change of FNMR can be used as a label to measure the image quality.
[0131] In this implementation, the quality score sequence of the face enhanced image set can be calculated by the following formula (4):
[0132]
[0133] in, For sample x i The quality score of the corresponding face enhanced image, Characterize face enhancement image set elimination x i The following subset, ∈x represents X and The recognition threshold of Only with sample x i is related to the recognition threshold ∈. Since the above contents are all quality scores calculated under a fixed FMR, in order to calculate the quality score sequence composed of the actual quality scores of the face enhanced image set, we can take The expected value of FMR is used as the actual quality score constituting the quality score sequence.
[0134] Alternatively, the FMR can be treated as a random variable uniformly distributed in the interval [0,1]. In this case,
[0135] For a given set of face-enhanced images, for a sample x in the face-enhanced image set i For example, and are independent of each other, so the quality score sequence of the face enhanced image set can also be expressed by the following formula (5):
[0136]
[0137] Among them, F() is the implicit function. It indicates that high-quality face enhanced images are easy to be recognized, that is, the intra-class distance of the face enhanced images in the face enhanced image set is short, and the inter-class distance is long.
[0138] Next, how to generate the quality score sequence according to the Wasserstein distance in the aforementioned step S3013 is described in detail. Figure 5 A flowchart of another face recognition model training method provided in an embodiment of the present application. Figure 5 As shown, including:
[0139] S501. According to the joint distribution of training positive samples and training negative samples, the Wasserstein distance is used to measure the quality difference between images.
[0140] Since high-quality images are easy to recognize, the intra-class distance of the face enhanced images in the face enhanced image set is short and the inter-class distance is long, so the quality difference between any two face images can be measured by the Wasserstein distance.
[0141] S502. Obtain a quality score sequence according to the quality difference.
[0142] In this implementation, the quality score sequence of the face enhanced image set can be calculated by the following formula (6):
[0143]
[0144] in, Characterizes the Wasserstein distance between training positive samples and training negative samples, represents the set of joint distribution γ of training positive samples and training negative samples, Represents the expectation of training positive samples and training negative samples.
[0145] Optionally, the quality scores in the quality score sequence may be subjected to standard regularization, and the value of the quality score is standardly regularized to the interval [0,100]. When the quality score is 100, the quality of the face enhanced image is the highest, and when the quality score is 0, the quality of the face enhanced image is the lowest.
[0146] Based on the above embodiments, the embodiments of the present application also provide a face recognition device.
[0147] Figure 6 A structural diagram of a face recognition device 60 provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, including:
[0148] The acquisition module 601 is used to acquire a face image to be recognized.
[0149] Processing module 602 is used to output the facial image to a pre-trained facial recognition model to obtain a facial recognition result output by the facial recognition model; the facial recognition model is trained based on a facial registration image set, a facial image set, a facial key point set, and a facial enhancement image set; the facial enhancement image in the facial enhancement image set carries a facial mask generated based on the key point set.
[0150] The output module 603 is used to output the face recognition result.
[0151] Optionally, the acquisition module 601 is further used to acquire the face registration image set, the face image set, and the face key point set.
[0152] Optionally, the processing module 602 is also used to generate the face enhanced image set based on the face registration image set, the face image set, and the face key point set; and train the initial model through the face registration image set, the face image set, and the face enhanced image set to generate the face recognition model.
[0153] Optionally, the processing module 602 is further used to perform quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence; obtain a comparison score sequence based on the face registration image set, the face image set, and the face enhanced image set; and input the quality score sequence and the comparison score sequence into an initial model for collaborative training to generate the face recognition model.
[0154] Optionally, the processing module 602 is further used to obtain a similarity set of training positive samples of each face enhanced image and a similarity set of training negative samples of each face enhanced image based on the face enhanced images in the face enhanced image set, wherein the training positive samples are image pairs of the same face, and the training negative samples are image pairs of different faces; obtain an error and rejection curve of the face enhanced image set based on the similarity set of the training positive samples, the similarity set of the training negative samples, and a recognition threshold set; and generate the quality score sequence based on the error and rejection curve.
[0155] Optionally, the processing module 602 is also used to determine a target recognition threshold that meets preset quality requirements from the recognition threshold set based on the error and rejection curve; obtain an error match rate set and an error non-match rate set of the face enhanced image set based on the target recognition threshold; and generate the quality score sequence based on the error match rate set and the error non-match rate set.
[0156] Optionally, the processing module 602 is further configured to measure the quality difference between images using a similarity distance according to the joint distribution of the training positive samples and the training negative samples; and obtain a quality score sequence according to the quality difference.
[0157] Optionally, the processing module 602 is also used to generate a feature vector sequence based on the comparison score sequence and the quality score sequence, wherein the feature vector sequence represents the uncertainty of the comparison algorithm and the uncertainty of the face recognition process; use the uncertainty in the feature vector sequence as input data and input it into the initial model for Bayesian enhanced collaborative training; and generate the face recognition model based on the training results.
[0158] Optionally, the processing module 602 is further configured to perform color rendering on the face mask to generate a face mask with at least two colors.
[0159] The face recognition device provided in the embodiment of the present application can execute the technical solution of the face recognition processing method provided in any of the above embodiments, and its principles and technical effects are similar, which will not be repeated here.
[0160] Figure 7 Schematic diagram of the structure of the electronic device 70 provided in the embodiment of the present application. Figure 7 As shown, the electronic device may include: a transceiver 701 , a processor 702 , and a memory 703 .
[0161] The processor 702 executes the computer execution instructions stored in the memory, so that the processor 702 executes the scheme in the above embodiment. The processor 702 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0162] The memory 703 is connected to the processor 702 via a system bus and completes communication between them. The memory 703 is used to store computer program instructions.
[0163] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.
[0164] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the face recognition method in the above embodiment.
[0165] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the face recognition method in the above embodiment.
[0166] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the face recognition method in the above embodiment.
[0167] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0168] It should be further noted that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0169] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0170] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0171] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0172] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0174] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0175] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A face recognition method, characterized in that: The method comprises: Obtain a face image to be recognized; Output the face image to a pre-trained face recognition model to obtain a face recognition result output by the face recognition model; the face recognition model is obtained by collaborative training based on a face registration image set, a face image set, a face key point set, and a face enhancement image set; the face enhancement image in the face enhancement image set carries a face mask generated according to the face key point set; the face mask is used to simulate a scene in which a user wears a mask; Output the face recognition result.
2. The method according to claim 1, characterized in that The method further comprises: Acquire the face registration image set, the face image set, and the face key point set; Generate the face enhancement image set according to the face registration image set, the face image set, and the face key point set; The face recognition model is generated by training the initial model using the face registration image set, the face image set, and the face enhancement image set.
3. The method according to claim 2, characterized in that The method of generating the face recognition model by collaboratively training the initial model with the face registration image set, the face image set, and the face enhancement image set comprises: Performing quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence; Obtaining a comparison score sequence according to the face registration image set, the face image set, and the face enhanced image set; the comparison score sequence is used to indicate the similarity between images in different image sets; The quality score sequence and the comparison score sequence are input into an initial model for collaborative training to generate the face recognition model.
4. The method according to claim 3, characterized in that The step of performing quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence includes: According to the face enhanced images in the face enhanced image set, obtaining a similarity set of training positive samples of each face enhanced image and a similarity set of training negative samples of each face enhanced image, wherein the training positive samples are image pairs of the same face and the training negative samples are image pairs of different faces; Obtaining an error and rejection curve of the face enhanced image set according to the similarity set of the training positive samples, the similarity set of the training negative samples, and the recognition threshold set; the error and rejection curve is used to indicate whether the face enhanced image in the face enhanced image set and other face images are the same face image; The quality score sequence is generated according to the error and rejection curve.
5. The method according to claim 4, characterized in that Generating the quality score sequence according to the error and rejection curve comprises: Determining, according to the error and rejection curve, a target recognition threshold that meets a preset quality requirement from the recognition threshold set; According to the target recognition threshold, obtaining a set of false matching rates and a set of false non-matching rates of the face enhanced image set; The quality score sequence is generated according to the set of false matching rates and the set of false non-matching rates.
6. The method according to claim 4, characterized in that The step of performing quality evaluation on the face enhanced images in the face enhanced image set to obtain a quality score sequence includes: According to the joint distribution of the training positive samples and the training negative samples, using a similarity distance to obtain the quality difference between the face enhanced images; The quality score sequence is obtained according to the quality difference.
7. The method according to any one of claims 2 to 6, characterized in that: The step of inputting the quality score sequence and the comparison score sequence into an initial model for collaborative training to generate the face recognition model comprises: Generate a feature vector sequence according to the comparison score sequence and the quality score sequence, wherein the feature vector sequence represents the uncertainty of the comparison algorithm and the uncertainty of the face recognition process; Using the uncertainty in the feature vector sequence as input data, and inputting it into the initial model for Bayesian enhanced collaborative training; The face recognition model is generated according to the training results.
8. The method according to any one of claims 2 to 6, characterized in that: The method further comprises: Color rendering is performed on the face mask to generate a face mask with at least two colors.
9. A face recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a face image to be recognized; A processing module, used for outputting the face image to a pre-trained face recognition model, and obtaining a face recognition result output by the face recognition model; the face recognition model is obtained by collaborative training based on a face registration image set, a face image set, a face key point set, and a face enhancement image set; the face enhancement image in the face enhancement image set carries a face mask generated according to the key point set; the face mask is used to simulate a scene in which a user wears a mask; An output module is used to output the face recognition result.
10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.
12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.