A real-time face key point quality evaluation method based on deep learning

Abstract

The application discloses a real-time face key point quality evaluation method based on deep learning, comprising the following steps: modifying a face key point detection model structure based on a convolutional neural network; designing a loss function in model training; setting a confidence loss function cutoff use threshold in model training; finally, the sum of the face key point loss function with weight and the confidence loss function is the total loss function; evaluating the face key point quality in the model inference stage. The application uses one model to complete face key point positioning and face key point quality evaluation, realizes the evaluation of the face key point quality with lower computing power, is more real-time, and has important engineering significance.

Description

Technical Field

[0001] This invention relates to the field of face recognition and detection technology, specifically, to a real-time face key point quality assessment method based on deep learning. Background Technology

[0002] Currently, facial landmark detection remains a key technology in the field of facial recognition and detection. It uses the facial region detected by a facial detection model as input, then passes the input through the facial landmark detection model to obtain a certain number of landmarks on the facial image. The quality of these facial landmarks plays a decisive role in the accuracy of subsequent liveness detection and face comparison stages in facial recognition, directly affecting the effectiveness of facial recognition in scenarios such as mobile phones (computers), turnstiles, access control systems, attendance machines, and facial payment. Poor-quality facial landmarks will directly lead to facial recognition failure. Besides facial recognition, facial landmark detection is increasingly being applied to head pose estimation, 3D reconstruction technology, fatigue driving detection in traffic safety (such as eye opening / closing and yawning), and beauty cameras (such as beauty filters and AI face swapping). The accuracy of these related functions is directly determined by the quality of the facial landmarks.

[0003] Poor-quality facial landmarks typically exhibit features such as blurriness, occlusion, profile views, inaccurate face bounding boxes, and erroneous facial images. Figure 1 As shown (from left to right, the images are: "Incorrect Face", "Blurred", "Occluded", "Side View", and "Inaccurate Face Outline").

[0004] To reduce the impact of facial landmarks on overall functionality, commonly used facial landmark evaluation techniques include: 1) Adding a facial image quality assessment module between face detection and facial landmark detection, using a facial image quality assessment model to evaluate the quality of the face image output by the face detection model, ensuring high quality of the face image input to the facial landmark detection model. 2) Adding a facial landmark quality assessment module after facial landmark detection, using a facial landmark quality assessment model to evaluate the quality of the facial landmark image to be evaluated, ensuring high quality of the face image input to the next module, such as face recognition, and relatively accurate facial landmarks. Existing facial landmark quality assessment methods add an image quality assessment model to the facial landmark detection model, resulting in a corresponding increase in performance overhead and severely impacting the real-time performance of the algorithm. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a real-time facial key point quality assessment method based on deep learning. This method assesses the quality of facial key points with almost no increase in computing power, ensuring high data quality flowing into the next module, such as the face recognition module, thereby improving the accuracy of related functions, such as increasing the success rate of face recognition.

[0006] This invention can be achieved through the following technical solutions:

[0007] A real-time facial landmark quality assessment method based on deep learning includes the following steps:

[0008] 1) Modify the structure of the face landmark detection model based on convolutional neural network to output face landmarks and their confidence scores;

[0009] 2) In model training, a loss function is designed. The loss function for facial key points adopts the smooth L1 loss function based on geometric constraints, and the confidence loss function adopts the L2 loss function to calculate the mean square error between the predicted confidence (i.e., the network output confidence) and the true confidence. The true confidence is obtained from the real facial key points and the predicted facial key points. Finally, the sum of the weighted facial key point loss function and the confidence loss function is the total loss function.

[0010] 3) Set a cutoff threshold for the confidence loss function during model training to reduce the imbalance between positive and negative samples related to confidence.

[0011] 4) During the model inference stage, the confidence level of the prediction is used to evaluate the quality of facial landmarks.

[0012] Further, step 1) specifically involves changing the original 1×136 fully connected layer for facial landmarks to a 1×(136+1) fully connected layer for facial landmarks and confidence, where 1×136 represents 68 facial landmarks and 1×(136+1) represents adding a facial landmark confidence score to the 68 facial landmarks. The facial landmark confidence score is used to evaluate the quality of the current facial landmarks. That is, each facial image, after being processed by the improved convolutional neural network-based facial landmark detection and confidence evaluation model, simultaneously obtains 68 facial landmarks and 1 confidence score to evaluate the quality of the facial landmarks.

[0013] Optionally, step 1) specifically involves adding a new fully connected layer branch to the face landmark detection network that originally outputs 1×136, to output the confidence of the face landmarks. The new fully connected layer branch is set to a 1×1 or 1×68 dimensional feature vector. 1×1 means outputting a total confidence of the face landmarks, and 1×68 means outputting the confidence of each of the 68 face landmarks separately.

[0014] Further, step 2) specifically involves: the total loss function includes a face key point loss function and a confidence loss function, designed as shown in equation (1):

[0015] Loss_total=w1×Loss_lms+w2×Loss_conf (1)

[0016] In equation (1), w1 and w2 are the weights of the loss functions Loss_lms and Loss_conf, respectively, with w1 = 1 and w2 = 0.5.

[0017] In equation (1), Loss_lms is the loss function for 68 facial key points, which is a smooth L1 loss function based on geometric constraints, obtained from equation (2):

[0018]

[0019] In equation (2), N is the number of samples in the current batch, and Loss_lms i The loss of the facial landmarks for the i-th sample is obtained from equation (3);

[0020]

[0021] In equation (3), M is the number of facial landmark values ​​(M = 136). The loss for the j-th value of the facial keypoint of the i-th sample is obtained by equations (4) and (5);

[0022]

[0023] In equations (4) and (5), w is the weighting parameter for each sample, which can be adjusted according to the proportion of samples belonging to class C. Class C is divided into attribute classes such as frontal face, side face, head up, head down, expression, blur and occlusion. Here, w is simply set to 10; w_esp is the fixed weight for calculating the loss of each sample, and w_esp is set to 2.

[0024] In equations (4) and (5), error i j Let $\frac{j}{i}$ be the absolute error between the predicted and actual values ​​of the $j$-th facial landmarks of the $i$-th sample. Let $\frac{j}{i}$ be the absolute error between $j$ and $j$. The loss is calculated using the logarithm of the absolute error, which is equivalent to the absolute error at (w, ∞). i j It can be obtained from equation (6);

[0025]

[0026] In equation (6), Let j be the predicted value of the face landmark of the i-th sample. Let j be the j-th value of the real i-th sample face keypoint;

[0027] In equation (2), euler_angle_w i The geometric constraint weights are obtained from equation (7);

[0028]

[0029] In equation (7), K represents the index in Euler angles (pitch, yaw, roll). Let represent the deviation angle between the true value of the i-th sample and the predicted yaw, pitch, and roll angles. Obviously, as the deviation angle increases, the degree of penalty also increases. It is obtained from equation (8);

[0030]

[0031] In equation (8), The k-th angle in the predicted i-th sample Euler angles is obtained by the Euler angle network attached during training and is not involved in the testing phase.

[0032] In equation (8), For the k-th angle in the true i-th sample Euler angle, the true Euler angle is solved using the solvePnP function in OpenCV to obtain the rotation vector, and then the rotation vector is converted into Euler angle;

[0033] In equation (1), Loss_conf is the confidence loss function for evaluating the quality of facial landmarks. The L2 loss function, i.e., the mean squared error loss function, is used, as shown in equation (9):

[0034]

[0035] In equation (9), N is the number of samples in a batch, x i Let y be the prediction confidence of the facial landmark of the i-th sample, i.e., the confidence of the network output. i Let y be the true confidence score of the facial landmark of the i-th sample. i The standard errors of the selected predicted keypoints and actual keypoints are given by equation (10):

[0036]

[0037] In equation (10), n is the nth value of the facial landmark, m is the mth value of the facial landmark, j∈[n, m], Lms_pre j Lms_gt is the j-th value of the predicted facial landmark. jLet j be the j-th value of the real facial landmark.

[0038] Furthermore, step 3) includes the following steps:

[0039] 1) During the first model training iteration, the change process of confidence loss is recorded. After the model training is completed, the time when the threshold for use needs to be calculated is determined based on the confidence loss decreasing to a certain level. That is, the number of training iterations. Here, the number of training iterations is simply set to 10, i.e., epoch = 10.

[0040] 2) During the second model training, in the last iteration of the 10th epoch of model training, the confidence threshold calculation method of the model inference stage is used to start calculating the first cutoff threshold.

[0041] 3) At the beginning of the 11th epoch of the current model training, i.e. the first iteration, the cutoff threshold is used. At the end of the 11th epoch, i.e. the last iteration, the first calculation method is used again to obtain the second cutoff threshold.

[0042] 4) Starting from the 12th epoch of the current model training, repeat step 3, that is, use the cutoff threshold calculated in the previous epoch in the current epoch, and update the cutoff threshold in the last iteration of the current epoch, and repeat this until the end of training.

[0043] Further, step 4) specifically involves: during the model inference stage, inputting a face image to be detected, and outputting the face key points and their confidence scores based on the face key point detection and confidence score evaluation model using a convolutional neural network. The confidence score is used to evaluate the quality of the face key points. If the confidence score is less than a threshold, the current face key point is considered to be of good quality and is applied to the face recognition module or other modules. Otherwise, the current face key point is considered to be of poor quality, and the system directly returns to the image input stage to obtain the next frame of face image data and execute the next inference.

[0044] Beneficial effects

[0045] Existing technologies add an image quality assessment model to the facial landmark detection model, resulting in increased performance consumption. In contrast, this invention uses a single model to simultaneously output facial landmarks and their confidence scores, where the confidence scores can be used to assess the quality of the facial landmarks. This invention does not increase performance consumption and is more real-time. Attached Figure Description

[0046] Figure 1 Feature maps of facial landmark images of poor quality;

[0047] Figure 2This is a structural diagram of a face landmark detection and confidence evaluation model based on a convolutional neural network.

[0048] Figure 3 This is a structural diagram of the second type of face keypoint detection and confidence evaluation model based on convolutional neural networks;

[0049] Figure 4 This is a diagram of the Euler angle network structure.

[0050] Figure 5 Flowchart for facial landmark quality assessment;

[0051] Figure 6 The images show key facial features of both low-quality and normal faces, along with their confidence scores. Detailed Implementation

[0052] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0053] This invention proposes a real-time facial landmark quality assessment method based on deep learning. First, it modifies the structure of the facial landmark detection model based on a convolutional neural network. An additional fully connected output is added to the original 1×136 fully connected layer output, enabling the model to output 1×136 (68) facial landmarks while simultaneously outputting corresponding confidence scores. These confidence scores are used to assess the quality of the current facial landmarks.

[0054] Then, loss functions are designed during model training. The loss function for facial landmarks uses a smoothed L1 loss function based on geometric constraints. The smoothed L1 loss function reduces the impact of outliers on facial landmark detection. The confidence loss function uses a mean squared error (MSE) loss function, which increases the impact of outliers on the predicted confidence score. The MSE loss function is used to calculate the mean squared error between the predicted confidence score (the network output confidence score) and the true confidence score. The true confidence score is obtained from the real facial landmarks and the predicted facial landmarks. Finally, the sum of the weighted facial landmark loss function and the confidence loss function is the total loss function.

[0055] Next, a cutoff threshold for the confidence loss function is set during model training. This cutoff threshold is used after the model has been trained for a period of time. It is obtained using the same confidence threshold calculation method used during the model inference phase, applied during the testing phase of training, and updated after each training iteration and used in the next training iteration. The cutoff threshold is applied as follows: if any true confidence in a batch exceeds the cutoff threshold, the confidence loss function is applied to that batch. In this case, the total loss, consisting of the facial landmark loss and the confidence loss, is backpropagated. If no true confidence in a batch exceeds the cutoff threshold, the confidence loss function is not applied, and only facial landmark backpropagation occurs. This method can reduce the imbalance between positive and negative samples related to confidence during model training.

[0056] Finally, the quality of facial landmarks is evaluated using the predicted confidence scores during the model inference phase. In this phase, a single frame of a face image is input into the new model, which outputs the facial landmarks and their confidence scores. The confidence scores are used to assess the quality of the facial landmarks. If the confidence score is within a threshold, the current facial landmark is considered to be of good quality and is used; otherwise, it is considered to be of poor quality and is discarded.

[0057] Specifically, the steps include:

[0058] Step 1: Modify the structure of the face landmark detection model based on convolutional neural networks

[0059] The original fully connected layer for facial landmark detection, which outputs 1×136, is changed to a fully connected layer for facial landmark detection and confidence, which outputs 1×(136+1). 1×136 represents 68 facial landmarks, and 1×137 represents adding a facial landmark confidence score to these 68 landmarks. The facial landmark confidence score is used to evaluate the quality of the current facial landmarks. In other words, each face image processed by the improved convolutional neural network-based facial landmark detection and confidence evaluation model can simultaneously obtain 68 facial landmarks and one confidence score to evaluate the quality of the facial landmarks.

[0060] Among them, the structure of the facial landmark detection and confidence evaluation model based on convolutional neural networks is as follows: Figure 2 As shown:

[0061] In addition to the methods mentioned above, another way to modify the facial landmark detection model structure is to add a new fully connected layer branch to the original 1×136 facial landmark detection network. This new fully connected layer branch can be set to a 1×1 or 1×68 dimensional feature vector. 1×1 represents outputting a total confidence score for each facial landmark, while 1×68 represents outputting a confidence score for each of the 68 facial landmarks individually. The facial landmark and confidence detection model structure after using this method is as follows: Figure 3 As shown; or, after the original 1×136 fully connected layer output at the keypoints, add a 1×1 or 1×68 fully connected layer confidence output.

[0062] Step 2: Design the loss function for model training

[0063] The total loss function includes the face landmark loss function and the confidence loss function, designed as shown in equation (1):

[0064] Loss_total=w1×Loss_lms+w2×Loss_conf (1)

[0065] In equation (1), w1 and w2 are the weights of the loss functions Loss_lms and Loss_conf, respectively, with w1 = 1 and w2 = 0.5.

[0066] In equation (1), Loss_lms is the loss function for 68 facial key points, which is a smooth L1 loss function based on geometric constraints, obtained from equation (2):

[0067]

[0068] In equation (2), N is the number of samples in the current batch, and Loss_lms i The loss of the facial landmarks for the i-th sample is obtained from equation (3);

[0069]

[0070] In equation (3), M is the number of facial landmark values ​​(M = 136). The loss for the j-th value of the facial keypoint of the i-th sample is obtained by equations (4) and (5);

[0071]

[0072] In equations (4) and (5), w is the weighting parameter for each sample, which can be adjusted according to the proportion of samples belonging to class C. Class C is divided into attribute classes such as frontal face, side face, head up, head down, expression, blur and occlusion. Here, w is simply set to 10; w_esp is the fixed weight for calculating the loss of each sample, and w_esp is set to 2.

[0073] In equations (4) and (5), error i j Let $\frac{j}{i}$ be the absolute error between the predicted and actual values ​​of the $j$-th facial landmarks of the $i$-th sample. Let $\frac{j}{i}$ be the absolute error between $j$ and $j$. The loss is calculated using the logarithm of the absolute error, which is equivalent to the absolute error at (w, ∞). i j It can be obtained from equation (6);

[0074]

[0075] In equation (6), Let j be the predicted value of the face landmark of the i-th sample. Let j be the j-th value of the real i-th sample face keypoint;

[0076] In equation (2), euler_angle_w i The geometric constraint weights are obtained from equation (7);

[0077]

[0078] In equation (7), K represents the index in Euler angles (pitch, yaw, roll). Let represent the deviation angle between the true value of the i-th sample and the predicted yaw, pitch, and roll angles. Obviously, as the deviation angle increases, the degree of penalty also increases. It is obtained from equation (8);

[0079]

[0080] In equation (8), The k-th angle in the predicted i-th sample Euler angles is obtained from the Euler angle network attached during training and does not participate in the testing phase. The Euler angle network structure is as follows: Figure 4 As shown;

[0081] In equation (8), For the k-th angle in the true i-th sample Euler angle, the rotation vector is solved using the solvePnP function in OpenCV, and then the rotation vector is converted into Euler angles.

[0082] In equation (1), Loss_conf is the confidence loss function for evaluating the quality of facial landmarks. The L2 loss function, i.e., the mean squared error loss function, is used, as shown in equation (9):

[0083]

[0084] In equation (9), N is the number of samples in a batch, x i Let y be the prediction confidence of the facial landmark of the i-th sample, i.e., the confidence of the network output. i Let y be the true confidence score of the facial landmark of the i-th sample. i The standard errors of the selected predicted keypoints and actual keypoints are given by equation (10):

[0085]

[0086] In equation (10), n is the nth value of the facial landmark, m is the mth value of the facial landmark, j∈[n, m], Lms_pre j Lms_gt is the j-th value of the predicted facial landmark. j Let j be the j-th value of the real facial landmark.

[0087] In equation (10), m and n make the selection of important facial key points more flexible when calculating the confidence loss function. For example, m = 34 and n = 136 means selecting key points of eyebrows, eyes, nose and mouth other than facial contours to calculate the true confidence.

[0088] In addition to the above design methods, the face key point loss function can also adopt the Huber loss function based on geometric constraints. Essentially, the Huber loss is an absolute error, but when the error is very small, it becomes an L2 loss function. This loss function is not as sensitive to outliers as the L2 loss function. Based on the above methods, equations (4) and (5) are changed to equation (11);

[0089]

[0090] In equation (11), Let j be the predicted value of the face landmark of the i-th sample. Let j be the j-th value of the real i-th sample face keypoint;

[0091] In equation (11), when the hyperparameter δ control error is reduced to a certain level, the loss function becomes the L2 loss function, which can be obtained through multiple training sessions; when the Huber loss is between [-δ, δ], it is equivalent to the L2 loss function, and when it is between (-∞, -δ) and (δ, ∞), it is equivalent to the L1 loss function.

[0092] Alternatively, based on the above-mentioned Huber loss function based on geometric constraints, equation (11) can be modified into equation (12);

[0093]

[0094] In equation (12), when the hyperparameter δ control error decreases to a certain level, the loss function becomes the L2 loss function and the exponential function; when the loss is in [-e -δ e -δ Between ], using the exponential function, in [-δ, -e -δ ) and (e -δ When the distance is between (-∞, -δ) and (δ, ∞), the equivalent L2 loss function is used; when the distance is between (-∞, -δ) and (δ, ∞), the equivalent L1 loss function is used.

[0095] In addition to the above design methods, the confidence loss function can also use the L1 loss function, i.e., the mean absolute error. Based on the above methods, Equation (9) can be changed to Equation (13).

[0096]

[0097] In equation (13), N is the number of samples in a batch, x i Let y be the prediction confidence of the facial landmark of the i-th sample, i.e., the confidence of the network output. i Let be the true confidence level of the facial landmarks of the i-th sample;

[0098] In addition, the confidence loss function can also adopt a smooth L2 loss function similar to Huber's. Based on the above method, Equation (9) is changed to Equation (14).

[0099]

[0100] In equation (14), N is the number of samples in a batch, x i Let y be the prediction confidence of the facial landmark of the i-th sample, i.e., the confidence of the network output. i Let be the true confidence level of the facial landmarks of the i-th sample;

[0101] In equation (14), when the control error of the hyperparameter δ decreases to a certain extent, the loss function becomes the L1 loss function; when the loss is between [-δ, δ], it is equivalent to the L1 loss function, and when it is between (-∞, -δ) and (δ, ∞), it is equivalent to the L2 loss function.

[0102] In equation (10), the true confidence level y i In addition to the methods described above, absolute error can also be used. Based on the methods described above, equation (10) can be changed to equation (15):

[0103]

[0104] In equation (15), n is the nth value of the facial landmark, m is the mth value of the facial landmark, j∈[n, m], Lms_pre j Lms_gt is the j-th value of the predicted facial landmark. j Let j be the j-th value of the real facial landmark.

[0105] In addition, the true confidence level y i Alternatively, a smooth L2 loss function method similar to Huber can be used. Based on the above method, equation (15) can be changed to equation (16):

[0106]

[0107] In equation (16), when the control error of the hyperparameter δ decreases to a certain extent, the loss function becomes the L1 loss function; when the loss is between [-δ, δ], it is equivalent to the L1 loss function, and when it is between (-∞, -δ) and (δ, ∞), it is equivalent to the L2 loss function.

[0108] In addition, based on the confidence loss function being a smooth L2 loss function, equation (16) can also be changed to equation (17);

[0109]

[0110] In equation (17), when the control error of the hyperparameter δ decreases to a certain extent, the loss function becomes the L2 loss function; when the loss is between [-δ, δ], the standard error is used, and the equivalent L2 loss function is used in (-∞, -δ) and (δ, ∞).

[0111] In addition, based on the confidence loss function being a smooth L2 loss function, equation (17) can also be changed to equation (18);

[0112]

[0113] In equation (18), when the control error of the hyperparameter δ decreases to a certain level, the loss function becomes the L2 loss function and the exponential function; when the loss is between [-δ, δ], it is equivalent to the L1 loss function, and in [-e δ ,-δ) and [δ,e δ The equivalent L2 loss function is used, and the exponential function is used in (-∞, -eδ) and (eδ, ∞).

[0114] Step 3: Set the cutoff threshold for the confidence loss function during model training.

[0115] At the beginning of model training, due to the inaccuracy of its predicted facial landmarks, the calculated true confidence score is relatively high. At this stage, the number of images with inaccurate facial landmark predictions far exceeds the number of images with accurate predictions, resulting in an imbalance between positive and negative samples. As training progresses, the model's predicted facial landmarks become more accurate, but the calculated true confidence score decreases, leading to a gradual decrease in positive samples and a gradual increase in negative samples, again resulting in an imbalance. Therefore, a threshold is proposed to be set during model training to reduce the confidence-related imbalance between positive and negative samples.

[0116] The cutoff threshold is obtained by the following method:

[0117] First, during the first model training iteration, the change in confidence loss is recorded. After the model training is completed, the time to calculate the cutoff threshold is determined based on the confidence loss decreasing to a certain level, i.e., the number of training iterations. Here, the number of training iterations is simply set to 10, i.e., epoch = 10.

[0118] Then, during the second model training, in the last iteration of the 10th epoch of model training, the confidence threshold calculation method used in the model inference phase is used to start calculating the first cutoff threshold.

[0119] Next, at the start of the 11th epoch of the current model training (the first iteration), the cutoff threshold is used. At the end of the 11th epoch (the final iteration), the first calculation method is used again to obtain the second cutoff threshold. The cutoff threshold is applied as follows: if any true confidence score in a batch exceeds the cutoff threshold, a confidence loss function is applied to that batch. In this case, the total loss, consisting of the facial landmark loss and the confidence loss, is backpropagated. If no true confidence score in a batch exceeds the cutoff threshold, the confidence loss function is not used, and only facial landmark backpropagation occurs.

[0120] Finally, starting from the 12th epoch of the current model training, the third step is repeated, that is, the cutoff threshold calculated in the previous epoch is used in the current epoch, and the cutoff threshold is updated in the last iteration of the current epoch, and so on until the end of training.

[0121] This method can reduce the imbalance between positive and negative samples related to confidence during model training.

[0122] Step 4: Evaluate the quality of facial landmarks during the model inference stage.

[0123] Facial landmark quality assessment process as follows Figure 5As shown, in the model inference stage, an image of a face to be detected is input. The facial landmark detection and confidence evaluation model based on a convolutional neural network outputs the facial landmarks and their confidence scores. The confidence scores are used to evaluate the quality of the facial landmarks. If the confidence score is less than a threshold, the current facial landmark is considered to be of good quality and can be applied to the face recognition module or other modules. Otherwise, the current facial landmark is considered to be of poor quality, and the system directly returns to the image input stage to obtain the next frame of facial image data and perform the next inference.

[0124] Figure 6 The diagram shows how to use a facial landmark detection and confidence evaluation model to infer six types of images: erroneous faces, blurred faces, occluded faces, profile faces, inaccurate face bounding boxes, and normal faces. The facial landmarks and their confidence scores are obtained and then plotted on the corresponding face images.

[0125] according to Figure 6 It can be seen that facial landmarks with high confidence are actually facial landmarks of poor quality, which are usually generated by facial feature images of poor quality. Therefore, removing facial landmark images with high confidence can reduce the flow of poor quality facial images into the next stage, thereby improving the accuracy of related modules or functions.

[0126] When using confidence scores to assess the quality of facial landmarks, the required threshold is typically between [0, 1], and is obtained by the following method.

[0127] The first step is to prepare 1000 or more test images, including both high-quality and low-quality images of faces. Figure 1 The errors shown (blurred, occluded, side profile, and incomplete bounding boxes) are negative and positive samples, respectively, and are stored in the "good_test_images" and "bad_test_images" folders. At the same time, the facial key points of the face images with better quality are labeled, while the face images with poor quality do not need to be labeled.

[0128] The second step involves dividing the threshold into 9 groups, such as [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. Alternatively, the threshold can be divided more finely, with one group selected sequentially for each test, such as choosing a threshold of 0.1 for the first test. Then, the trained model is used to test all the test images, and the model outputs the facial landmarks and their confidence scores for each image. At this point, the threshold selected in step two is used. Four counters are defined and initialized: num_bad_right = 0, num_bad_false = 0, num_good_right = 0, and num_good_false = 0. When the confidence level is greater than the threshold and the input face image quality is poor, num_bad_right is incremented by 1; when the confidence level is less than the threshold and the input face image quality is poor, num_bad_false is incremented by 1; when the confidence level is less than the threshold and the input face image quality is good and the true confidence level is greater than the threshold, num_good_right is incremented by 1; and when the confidence level is greater than the threshold and the input face image quality is good and the true confidence level is less than the threshold, num_good_false is incremented by 1. The true confidence level is obtained from the actual (annotated) facial landmarks and the predicted facial landmarks.

[0129] The third step is to calculate the comprehensive metric F1-Score, which is a weighted harmonic average of precision and recall. It can balance the shortcomings of individual metrics such as precision and recall. The larger the F1-Score, the better the performance. The calculation formula is shown in equation (19):

[0130]

[0131] In equation (19), precision reflects the model's ability to correctly predict positive samples, that is, how many positive samples are correctly predicted as positive samples, also known as accuracy. The larger the value, the better the precision. Precision is obtained from equation (20):

[0132]

[0133] In equation (20), TP is the positive sample predicted as a positive sample, which is equal to num_bad_right here, and FP is the positive sample predicted as a negative sample, which is equal to num_bad_false here. Thus, equation (20) transforms into equation (21):

[0134]

[0135] In equation (19), recall reflects the model's ability to correctly predict the full extent of positive samples, that is, how many of the predicted positive samples are actually positive samples. It is also called recall rate. The larger the value, the better the performance. Recall is obtained from equation (22):

[0136]

[0137] In equation (22), TP is the positive sample predicted as a positive sample, which is equal to num_bad_right here, and FN is the negative sample predicted as a positive sample, which is equal to num_good_false here. Thus, equation (22) is transformed into equation (23):

[0138]

[0139] Finally, in the decision-making phase, steps two and three are repeated for each threshold in the set to select the threshold with the highest F1-Score, i.e., the one with the best performance. This threshold is the one required in the model inference phase when using confidence to evaluate the quality of facial landmarks.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time facial landmark quality assessment method based on deep learning, characterized in that, Includes the following steps: 1) Modify the structure of the face landmark detection model based on convolutional neural network to output face landmarks and their confidence scores; 2) In model training, a loss function is designed. The loss function for facial key points adopts the smooth L1 loss function based on geometric constraints, and the confidence loss function adopts the L2 loss function to calculate the mean square error between the predicted confidence (i.e., the network output confidence) and the true confidence. The true confidence is obtained from the real facial key points and the predicted facial key points. Finally, the sum of the weighted facial key point loss function and the confidence loss function is the total loss function. 3) Set a cutoff threshold for the confidence loss function during model training to reduce the imbalance between positive and negative samples related to confidence. 4) Step 3) includes the following steps: a) During the first model training iteration, the change process of confidence loss is recorded. After the model training is completed, the time when the threshold for use needs to be calculated is determined based on the reduction of confidence loss to a certain level. That is, the number of training iterations. Here, the number of training iterations is simply set to 10, i.e., epoch=10. b) During the second model training, in the last iteration of the 10th epoch of model training, the confidence threshold calculation method of the model inference stage is used to start calculating the first cutoff threshold. c) At the beginning of the 11th epoch of the current model training, i.e., the first iteration, the cutoff threshold is used, and at the end of the 11th epoch, i.e., the last iteration, the first calculation method is used again to obtain the second cutoff threshold. d) Starting from the 12th epoch of the current model training, repeat step 3, that is, use the cutoff threshold calculated in the previous epoch in the current epoch, and update the cutoff threshold in the last iteration of the current epoch, and repeat this process until the end of training. 5) During the model inference stage, the confidence level of the prediction is used to assess the quality of facial landmarks.

2. The real-time facial landmark quality assessment method based on deep learning according to claim 1, characterized in that, Step 1) specifically involves changing the original 1×136 fully connected layer for facial landmarks to a 1×(136+1) fully connected layer for facial landmarks and confidence. Here, 1×136 represents 68 facial landmarks, and 1×(136+1) represents adding a facial landmark confidence score to the 68 facial landmarks. The facial landmark confidence score is used to evaluate the quality of the current facial landmarks. That is, each facial image, after being processed by the improved convolutional neural network-based facial landmark detection and confidence evaluation model, simultaneously obtains 68 facial landmarks and 1 confidence score to evaluate the quality of the facial landmarks.

3. The real-time facial landmark quality assessment method based on deep learning according to claim 1, characterized in that, Step 1) specifically involves adding a new fully connected layer branch to the face landmark detection network that originally outputs 1×136, to output the confidence of the face landmarks. The new fully connected layer branch is set to a 1×1 or 1×68 dimensional feature vector. 1×1 means outputting a total confidence for the face landmarks, and 1×68 means outputting the confidence for each of the 68 face landmarks separately.

4. The real-time facial landmark quality assessment method based on deep learning according to claim 1, characterized in that, Step 2) specifically involves the following: The total loss function includes the face key point loss function and the confidence loss function, designed as shown in equation (1): In equation (1), , Loss functions and Weights, set =1, =0.5; In equation (1), The loss function for 68 facial key points is a smooth L1 loss function based on geometric constraints, obtained from equation (2): In equation (2), This represents the number of samples in the current batch. For the first The loss of facial landmarks for each sample is obtained by equation (3); In equation (3), Number of facial landmark values , For the first The loss of the j-th value of the facial key points of each sample is obtained by equations (4) and (5); In equations (4) and (5), The weighting parameter for each sample can be adjusted based on the proportion of samples belonging to class C. Class C is divided into frontal face, side face, head tilt, head tilt, expression, blur, and occlusion attribute categories. Here, we set it simply. ; To calculate the fixed weights for the loss of each sample, set... ; In equations (4) and (5), For the first The absolute error between the j-th predicted value and the true value of the facial landmark of a sample is given when the absolute error is within a certain range. between, The loss is calculated using the logarithm of the absolute error. Time equivalent absolute error From equation (6), we obtain: In equation (6), For the predicted first The j-th value of the facial landmarks of a sample. For the true first The j-th value of the facial landmarks of a sample; In equation (2), The geometric constraint weights are obtained from equation (7); In equation (7), Euler angles Index in Indicates the first The deviation angles between the actual values ​​and the predicted yaw, pitch, and roll angles of each sample are shown. Clearly, as the deviation angle increases, the penalty also increases. It is obtained from equation (8); In equation (8), For the predicted first The first of the sample Euler angles The angles are obtained by an Euler angle network attached during training and are not used in the testing phase. In equation (8), For the true first The first of the sample Euler angles For each angle, the true Euler angles are solved using the solvePnP function in OpenCV to obtain the rotation vector, and then the rotation vector is converted into Euler angles. In equation (1), To evaluate the confidence loss function for facial landmark quality, the L2 loss function, i.e., the mean squared error loss function, is used, as shown in equation (9): In equation (9), The number of samples in a batch For the first The prediction confidence of facial landmarks for each sample, i.e., the confidence of the network output. For the first True confidence of facial landmarks in a sample, True confidence The standard errors of the selected predicted keypoints and actual keypoints are given by equation (10): In equation (10), The first key point of the face One value, The first key point of the face One value, , Let j be the predicted value of the facial landmark. Let j be the j-th value of the real facial landmark.

5. The real-time facial landmark quality assessment method based on deep learning according to claim 1, characterized in that, Step 4) specifically involves: In the model inference stage, a face image to be detected is input, and the face key point detection and confidence evaluation model based on the convolutional neural network outputs the face key points and their confidence scores. The confidence scores are used to evaluate the quality of the face key points. When the confidence score is less than the threshold, the current face key point is considered to be of good quality and is applied to the face recognition module or other modules. Otherwise, the current face key point is considered to be of poor quality, and the system directly returns to the image input stage to obtain the next frame of face image data and execute the next inference.

Images