Face alignment and pose evaluation methods and systems, storage media, electronic devices
By training the face alignment branch to obtain mask-invariant features, and combining the data mining methods learned in the course, the coupling between facial feature points and head pose is reduced. This solves the problems of instability under mask conditions and accuracy decline under large-angle poses in traditional methods, and achieves higher evaluation accuracy.
Patent Information
- Application Number
- CN202211729227.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-12-30
AI Technical Summary
In existing technologies, facial feature point and head pose evaluation are highly coupled in traditional methods, which leads to instability in head pose evaluation when wearing masks, and the dataset exhibits a trailing phenomenon, resulting in a decline in accuracy at large pose angles.
By training the face alignment branch to obtain mask-invariant features, freezing the data, training the head pose evaluation branch, and using a course-based data mining method to retrain the face alignment and head pose evaluation branches, the coupling is reduced and the accuracy is improved.
It improves the accuracy of head posture assessment when wearing a mask, and enhances the accuracy of face alignment and head posture assessment under large-angle postures.
Smart Images

Figure CN116343286B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more particularly to a face alignment and pose assessment method and system, storage medium, and electronic device. Background Technology
[0002] Face alignment, also known as face feature point detection, aims to predict the coordinates of a series of predefined points in a face image, such as face contour points, eyebrow corners, and mouth corners. Face alignment is widely used in computer vision and machine vision, including tasks such as face recognition, expression recognition, face correction, and 3D face modeling. Head pose assessment is the evaluation of the pose of a face and head based on a face image, including horizontal angle, pitch angle, and rotation angle. Pose assessment is widely used in human-computer interaction scenarios and driver monitoring scenarios.
[0003] Since both face alignment and pose assessment are calculated based on face images, they are closely related. That is, when the head pose changes, the position of facial feature points will also change. Therefore, there are two traditional methods for these two methods. One is to first calculate facial feature points through a model, and then calculate the head pose based on the feature points and the 3D face model (e.g., patent document: CN114882574A). The other is to use multi-task learning, which puts both into the same model for learning, that is, the output of the model is the result of both (e.g., patent documents: CN14360041A, CN115205926A). These two traditional methods result in a high degree of coupling between facial feature points and head pose. When facial expression changes, facial feature points will change, and the head pose will also change accordingly, but the actual head pose does not change.
[0004] The currently available head pose datasets lack cases with masks, and it is difficult to manually label the head pose of samples with masks, resulting in unstable head pose evaluation by the model under mask conditions. In addition, the previously released face datasets often have data tailing, that is, there are fewer large-angle pose samples and more small-angle pose samples, which leads to a decline in the accuracy of face alignment and head pose evaluation under large-angle pose conditions. Summary of the Invention
[0005] To address the aforementioned issues, the main objective of this invention is to provide a face alignment and pose assessment method that overcomes the high coupling between facial feature points and head pose in traditional methods, improves the accuracy of head pose assessment when wearing a mask, and enhances the accuracy of face alignment and head pose assessment under large-angle pose conditions.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A method for face alignment and pose assessment includes the following steps:
[0008] Step 1: Based on the facial feature point dataset, train the model's face alignment branch to obtain facial features that are invariant even when wearing a mask.
[0009] Step 2: Freeze the face alignment branch, train the head pose evaluation branch based on the head pose dataset, and obtain the head pose of the face feature point dataset.
[0010] Step 3: Based on the head pose of the face feature point dataset, a data mining method based on course learning is used to retrain the face alignment branch.
[0011] Step 4: Freeze the face alignment branch again and add the feature fusion branch. Use a data mining method based on course learning to retrain the head pose evaluation branch.
[0012] Furthermore, step 1 specifically includes the following steps:
[0013] Based on the facial feature point dataset and feature point labels, data augmentation operations are performed on facial images, and then the facial alignment branch is trained.
[0014] The face alignment branch outputs a first face feature point heatmap and a second face feature point heatmap. An evaluation function is calculated based on the first face feature point heatmap, the second face feature point heatmap, and the real face feature point heatmap in the dataset.
[0015] Furthermore, the data augmentation operations include cropping, rotation, scaling, and color changes.
[0016] Furthermore, the face alignment branch consists of a convolution module, a MobileNetV2 module, and a MobileNetV2 funnel module.
[0017] Furthermore, in step 2, the second feature is extracted from the face image sample through the face alignment branch. Based on the head pose dataset, the weights of the face alignment branch in the framework are frozen. The head pose evaluation branch is trained with the second feature as input. The head pose evaluation branch consists of a series of MobileNetV2 modules and finally generates the head pose.
[0018] Furthermore, step 3 includes:
[0019] Pose evaluation is performed on face images from a face feature point dataset to obtain pose labels;
[0020] The obtained attitude labels are classified to obtain attitude classification labels for the samples;
[0021] Data mining of samples is performed using sample pose classification labels, and course learning principles are introduced into the data mining process.
[0022] Finally, the head pose evaluation branch is frozen, and the face alignment branch is retrained based on the face feature point dataset.
[0023] Furthermore, step 4 includes:
[0024] The face alignment branch is frozen again, and the second feature of face alignment and the features of different dimensions of the two MobileNetV2 funnels are fused together as inputs to different dimensions of the head pose evaluation branch. The head pose evaluation branch is then retrained to generate models for face alignment and head pose evaluation.
[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above.
[0026] The present invention also provides an electronic device, including: a processor, a memory, and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program including methods as described above.
[0027] The present invention also provides a face alignment and pose evaluation system, which includes the methods described above.
[0028] The beneficial effects of this invention are as follows:
[0029] The present invention includes the following steps: Step 1, based on a facial feature point dataset, train a model face alignment branch to obtain facial features invariant even when wearing a mask; Step 2, freeze the face alignment branch, train a head pose evaluation branch based on a head pose dataset, and obtain the head pose from the facial feature point dataset; Step 3, based on the head pose from the facial feature point dataset, retrain the face alignment branch using a course-based data mining method; Step 4, freeze the face alignment branch again, add a feature fusion branch, and retrain the head pose evaluation branch using a course-based data mining method. This invention reduces the coupling between face alignment and head pose assessment, thereby improving the accuracy of both. It improves the accuracy of head pose assessment when masked by first training the face alignment model to obtain mask-invariant features, and then training the head pose assessment model. To overcome dataset tailing, data mining is used during training to calculate weights based on the pose classification distribution of each batch; categories with smaller percentages receive larger weights. Furthermore, this invention employs a curriculum-based learning principle, focusing on samples with small-angle poses in the early stages of training and shifting to samples with large-angle poses in the later stages, resulting in better convergence of the training process and improved accuracy of face alignment and head pose assessment under large-angle conditions. Attached Figure Description
[0030] Figure 1This is a flowchart of the face alignment and head pose evaluation method described in this invention.
[0031] Figure 2 This is a schematic diagram of the entire training framework of the present invention.
[0032] Figure 3 This is a schematic diagram illustrating an example of face alignment and head pose evaluation according to the present invention. Detailed Implementation
[0033] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.
[0034] To overcome the high coupling between facial feature points and head pose in traditional methods, this invention proposes a face alignment and pose evaluation method and system, storage medium, and electronic device. To improve the accuracy of head pose evaluation when wearing masks, a face alignment task is first trained to obtain mask-invariant features, and then a head pose task is trained based on these mask-invariant features. To mitigate the tailing phenomenon caused by the dataset, a course-based data mining method is used in the training method to improve the accuracy of face alignment and head pose evaluation under large-angle poses.
[0035] The face alignment and pose evaluation method includes the following steps:
[0036] Step 1: Based on the facial feature point dataset, train the model's face alignment branch to obtain facial features that are invariant even when wearing a mask.
[0037] Step 2: Freeze the face alignment branch, train the head pose evaluation branch based on the head pose dataset, and obtain the head pose of the face feature point dataset.
[0038] Step 3: Based on the head pose of the face feature point dataset, a data mining method based on course learning is used to retrain the face alignment branch.
[0039] Step 4: Freeze the face alignment branch again and add the feature fusion branch. Use a data mining method based on course learning to retrain the head pose evaluation branch.
[0040] Furthermore, step 1 specifically includes the following steps:
[0041] Based on the facial feature point dataset and feature point labels, data augmentation operations are performed on facial images, and then the facial alignment branch is trained.
[0042] The face alignment branch outputs a first face feature point heatmap and a second face feature point heatmap. An evaluation function is calculated based on the first face feature point heatmap, the second face feature point heatmap, and the real face feature point heatmap in the dataset.
[0043] Furthermore, the data augmentation operations include cropping, rotation, scaling, and color changes.
[0044] Furthermore, the face alignment branch consists of a convolutional module, a MobileNetV2 module, and a MobileNetV2 funnel module.
[0045] Furthermore, in step 2, the second feature (feal) is extracted from the face image sample through the face alignment branch. 2nd Based on head pose dataset S head Freeze the face alignment branch weights in the frame, as featured 2nd As input, a head pose evaluation branch is trained; the head pose evaluation branch consists of a series of MobileNetV2 modules, which finally generate the head pose.
[0046] Further, step 3 includes:
[0047] For the facial feature point dataset S fan Facial images are used to assess pose and obtain pose labels.
[0048] The obtained attitude labels are classified to obtain attitude classification labels for the samples;
[0049] Data mining of samples is performed using sample pose classification labels, and course learning principles are introduced into the data mining process.
[0050] Finally, the head pose evaluation branch is frozen, based on the face feature point dataset S. fan Retrain the face alignment branch.
[0051] Further, step 4 includes:
[0052] The face alignment branch is frozen again, and the second feature of face alignment and the features of different dimensions of the two MobileNetV2 funnels are fused together as inputs to different dimensions of the head pose evaluation branch. The head pose evaluation branch is then retrained to generate models for face alignment and head pose evaluation.
[0053] A specific embodiment of the present invention is as follows:
[0054] Step 1: Based on the facial feature point dataset S fan(Including mask-wearing situations) and feature point marking, a series of data augmentation operations are performed on the face image, including cropping, rotation, scaling, and color changes, and then the framework is trained ( Figure 2 As shown in the diagram, the face alignment branch consists of a convolutional module, a MobileNetV2 module, and a MobileNetV2 funnel module. Samples generated by data augmentation, with dimensions of 256×256×3, are input into the face alignment branch of the model. After passing through the first MobileNetV2 funnel module, the first feature is obtained, with dimensions of 64×64×152. This first feature then passes through two branches, one of which generates a heatmap of the first facial feature points. Then, the two branches are fused with the input of the first MobileNetV2 funnel module and used as the input of the second MobileNetV2 funnel module. Finally, the second feature is generated by the second MobileNetV2 funnel module. znd The second feature is then processed by a convolutional module to generate a heatmap of the second facial feature points. During training, the heatmap of the first facial feature points output by the face alignment branch is used. Second facial feature heatmap and the dataset of real facial feature point heatmaps Calculate the evaluation function loss fan As shown in the following formula,
[0055]
[0056] Where N is the batch size of samples during training, α1 and α2 are weighting values, and loss is... hm The heatmap loss function can be the L1 function, L2 function, or Awing function;
[0057] Step 2: After the face image samples have been trained as described above, the face alignment branch is used to obtain the second feature. 2nd Since the facial feature point dataset includes both masked and unmasked face samples, the second feature... 2nd It concerns features that remain unchanged when wearing a mask; based on the head pose dataset S head (Excluding mask-wearing cases), freeze the face alignment branch weights within the frame, using feature... 2nd As input, a head pose evaluation branch is trained; the head pose evaluation branch consists of a series of MobileNetV2 modules, which finally generate the head pose. During the training of the head branch, the evaluation function is:
[0058]
[0059] Where lossrog For regression function, This represents the actual head posture of the sample.
[0060] Step 3: Since the head pose evaluation branch mentioned above uses mask-invariant features as input, it can stably evaluate the head pose while wearing a mask. Therefore, for the face feature point dataset S... fan Facial images are used for pose assessment to obtain pose labels. 0t p 0t r 0t These are the horizontal angle, pitch angle, and rotation angle of the head posture; then, the three posture angles are classified according to the following rules.
[0061]
[0062]
[0063]
[0064] Thus, the pose classification labels of the samples are obtained. Due to the tailing phenomenon in the dataset, that is, the data is concentrated at small angles, the pose classification labels of the samples can be used for data mining. For each sample, its pose classification label is: The weighted values are calculated as follows;
[0065]
[0066] Where N is the number of training batch samples, and count(x) is the number of samples in the batch with pose classification x, the weighted weight can be calculated based on the distribution of pose types in each batch. The smaller the distribution proportion, the larger the weight.
[0067] Performing data mining early in the training process, and prematurely increasing the influence of large-angle pose samples, is detrimental to the convergence of the network model. Therefore, data mining incorporates the principle of curriculum learning, which involves first learning relatively easy cases and then learning relatively difficult cases. Thus, the sample weighting is adjusted as follows:
[0068]
[0069] Where S is the current training iteration number, S all The total number of iterations is [number missing]. Early training focuses on samples with small-angle poses, while later training emphasizes samples with large-angle poses. Finally, the head pose evaluation branch is frozen, based on the face feature point dataset S. fan Retrain the face alignment branch, where the training evaluation function is:
[0070]
[0071] Step 4: Since the previous head pose evaluation branch was based on the second feature output from the face alignment branch as input, this resulted in a high coupling between face alignment and head pose evaluation. Therefore, the face alignment branch was frozen again, and the second feature of face alignment was fused with features from different dimensions of the two MobileNetV2 funnels, serving as input to different dimensions of the head pose evaluation branch, such as... Figure 2 The red dashed line indicates the feature fusion branch. The head pose assessment branch is retrained using the same course-based data mining approach, and the training evaluation function is:
[0072]
[0073] The final result is a model for face alignment and head pose evaluation. Since face alignment outputs a heatmap of facial feature points, the heatmap needs to undergo post-processing (softargmax) to obtain the coordinates of the facial feature points (x, y, y). i y i ), as shown below:
[0074]
[0075] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above.
[0076] The present invention also provides an electronic device, including: a processor, a memory, and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program including methods as described above.
[0077] The present invention also provides a face alignment and pose evaluation system, which includes the methods described above.
[0078] The technical effects of this invention are as follows:
[0079] This invention reduces the coupling between face alignment and head pose assessment, thereby improving the accuracy of both. It improves the accuracy of head pose assessment when masked by first training the face alignment model to obtain mask-invariant features, and then training the head pose assessment model. To overcome dataset tailing, data mining is used during training to calculate weights based on the pose classification distribution of each batch; categories with smaller percentages receive larger weights. Furthermore, this invention employs a curriculum-based learning principle, focusing on samples with small-angle poses in the early stages of training and shifting to samples with large-angle poses in the later stages, resulting in better convergence of the training process and improved accuracy of face alignment and head pose assessment under large-angle conditions.
[0080] It should be noted that, in Figure 3 In the image, the first four rows are test set instances of the WFLW dataset, and the last two rows are instances with masks on. Below each instance are three numbers representing head pose assessment: the first is the pitch angle, the second is the horizontal angle, and the third is the rotation angle.
[0081] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Those skilled in the art can readily implement the present invention based on the accompanying drawings and the description above. However, any modifications, alterations, or variations made by those skilled in the art without departing from the scope of the present invention, utilizing the disclosed technical content, are equivalent embodiments of the present invention. Furthermore, any equivalent changes, alterations, or variations made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the present invention.
Claims
1. A method for face alignment and pose evaluation, characterized in that, Includes the following steps: Step 1: Based on the facial feature point dataset, train the model's face alignment branch to obtain facial features that are invariant even when wearing a mask. Specifically, based on the facial feature point dataset and feature point labels, data augmentation operations are performed on facial images, and then the facial alignment branch is trained. Step 2: Freeze the face alignment branch. Based on the head pose dataset, train the head pose evaluation branch and obtain the head pose of the face feature point dataset. Specifically, extract the second feature from the face image sample through the face alignment branch. Based on the head pose dataset, freeze the weights of the face alignment branch in the framework. Use the second feature as input and the real head pose in the head pose dataset as the supervision signal to train the head pose evaluation branch. The head pose evaluation branch consists of a series of MobileNetV2 modules and finally generates the head pose. The face alignment branch consists of a convolutional module, a MobileNetV2 module, and a MobileNetV2 funnel module. Step 3: Based on the head pose of the face feature point dataset, a data mining method based on course learning is used to retrain the face alignment branch. Step 4: Freeze the face alignment branch again, and fuse the second feature output by the face alignment branch with the features of different dimensions of the two MobileNetV2 funnel modules. Use this as the input of different dimensions of the head pose evaluation branch. Use a data mining method based on course learning to retrain the head pose evaluation branch and generate the face alignment and head pose evaluation models.
2. The face alignment and pose evaluation method of claim 1, wherein: Step 1 specifically includes the following steps: The face alignment branch outputs a first face feature point heatmap and a second face feature point heatmap. An evaluation function is calculated based on the first face feature point heatmap, the second face feature point heatmap, and the real face feature point heatmap in the dataset.
3. The face alignment and pose evaluation method of claim 2, wherein: The data augmentation operations include cropping, rotating, scaling, and color changing.
4. The face alignment and pose evaluation method of claim 1, wherein: Step 3 includes: Pose evaluation is performed on face images from a face feature point dataset to obtain pose labels; The obtained attitude labels are classified to obtain attitude classification labels for the samples; Data mining of samples is performed using sample pose classification labels, and course learning principles are introduced into the data mining process. Finally, the head pose evaluation branch is frozen, and the face alignment branch is retrained based on the face feature point dataset.
5. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by a processor according to any one of claims 1-4.
6. An electronic device, comprising: include: A processor, a memory, and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program including methods for performing the method as described in any one of claims 1-4.
7. A face alignment and pose estimation system, characterized by: The face alignment and pose assessment system includes the method described in any one of claims 1-4.
Citation Information
Patent Citations
Head posture analysis method, system and equipment based on face key point detection
CN114882574A
Lightweight robust face alignment method and system based on multi-task learning
CN115205926A
Face alignment method based on thick-to-thin face shape estimation
CN108446672A
Facial expression recognition method based on space attention and deep neural network
CN115171180A
