A lightweight face pose estimation method based on multi-task learning

Abstract

The present invention discloses a lightweight face pose estimation method based on multi-task learning. By utilizing the strong correlation between face pose and feature points, a lightweight face alignment and pose estimation model with inverted residual as the backbone network is designed. In the model, the pose estimation task network uses a pose rotation matrix with low-dimensional information as the output to represent the face pose, which can solve the problem of information loss caused by continuous rotation of Euler angles and the problem of information redundancy of ordinary rotation matrices. At the same time, a single-branch network with a sequence sharing strategy is introduced in the model, which consists of a face alignment task network and a pose estimation task network, to continue to perform inverted residual downsampling and convolution processing on the intermediate feature map obtained by the pose estimation task network, and obtain the face feature point coordinate matrix. Based on this, face alignment and pose estimation can be carried out, which can reduce the complexity of the multi-task learning model, and through multi-task learning, the accuracy of face pose estimation and alignment can be well guaranteed.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision image processing technology and artificial intelligence interaction technology, and particularly relates to a lightweight face pose estimation method based on multi-task learning. Background Art

[0002] In computer vision, head pose estimation is generally defined as using machine learning methods to estimate the relative deflection angle between the head in a digital image containing the head and the camera. Usually, a person's head pose has three degrees of freedom, namely the yaw angle in the horizontal direction, the pitch angle in the vertical direction, and the rotation angle in the image plane. In the context of requirements in aspects such as identity verification, safe driving, and human-computer interaction, head pose estimation, as a key issue in these practical applications, has received increasing attention in the fields of computer vision and machine learning in recent years. For example: in the field of identity verification, head pose estimation pre-estimates the orientation of the human face, which is beneficial for the subsequent positioning of feature points and subsequent processing; in the field of safe driving, head pose estimation can assist in gaze estimation to monitor the driver's attention; in the field of human-computer interaction, head pose estimation can assist in facial expression recognition for behavior prediction, etc.

[0003] In the research of face images, the pose estimation task is rarely a direct end-to-end task and is usually an auxiliary product of tasks such as face alignment and 3D reconstruction. Due to the close correlation between facial feature points and face pose, traditional methods calculate the head pose from facial feature points, and the accuracy depends on the face alignment algorithm. Extreme poses have a great impact on the accuracy of face alignment, resulting in large pose errors in the estimation of large poses.

[0004] As a classic method for estimating face pose, Perspective-n-point (PnP) (see the literature "Lepetit V, Moreno-Noguer F, Fua P. Epnp: An accurate o(n) solution to the pnp problem [J]. International journal of computer vision, 2009") utilizes the dependency relationship between feature points in a 2D image and face pose, constructs a general face model based on feature points, and solves the rotation matrix to obtain face pose information; the accuracy of the PnP method is related to feature point localization and the selection of general face model parameters, so the predicted pose error is relatively large. FDN (see the literature "Zhang H, Wang M, Liu Y, et al. FDN: Feature decoupling network for head pose estimation [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020") uses a three-branch network with a feature decoupling module to predict each pose angle and designs a new loss function based on the discreteness between the true label and the predicted label, and its results are better than advanced methods with a single RGB image as input. FSA-Net (see the literature "Yang T Y, Chen Y T, Lin Y Y, et al. Fsa-net: Learning fine-grained structure aggregation for head pose estimation from a single image [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019") utilizes the fine-grained structure mapping of spatially grouped features combined with different activation functions to obtain image features for predicting Euler angles.HopeNet (see the literature "Ruiz N, Chong E, Rehg J M. Fine-grained head pose estimation without keypoints[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018") designed a neural network with multiple losses, which extracts common features for Euler angles, uses three different branches for regression to obtain different angles. Each branch consists of a fully connected layer and a SoftMax activation function, and uses two different loss functions to adjust the coarse-grained and fine-grained regressions respectively. QuatNet (see the literature "Hsu H W, Wu T Y, Wan S, et al. Quatnet: Quaternion-based head pose estimation with multiregression loss[J]. IEEE Transactions on Multimedia, 2018") divides classification and regression into separate network branches, represents pose information using quaternions, one branch is used to classify Euler angles, and the other branch regresses the pose. WHENet (see the literature "Yijun Zhou and James Gregson, "Whenet: Real-time fine-grained estimation for wide range head pose," in 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020") established a new dataset to predict the Euler angle range of 360°. MNN (see the literature "Valle R, Buenaposada J M, Baumela L. Multi-task head pose estimation in-the-wild[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020") uses multi-task learning to train face alignment and pose estimation on a relabeled dataset to obtain better results.SynergyNet (see the literature "Wu C Y, Xu Q, Neumann U. Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry[C] / / 2021 International Conference on 3D Vision(3DV). IEEE, 2021") uses 3D reconstruction to minimize the average error of face pose estimation.

[0005] The above deep learning methods pursue high accuracy in face pose estimation, but the information used is overly redundant, resulting in a huge number of parameters in the estimation model. The data capacity occupied by the model usually reaches hundreds of megabytes, making it difficult to be deployed on lightweight computing carriers such as embedded devices, which limits the wider application of face pose estimation technology. Summary of the Invention

[0006] Aiming at the deficiencies of the existing technology, the present invention provides a lightweight face pose estimation method based on multi-task learning to solve the problems of excessive information redundancy and huge data capacity occupation in the face pose estimation model, which is more conducive to being deployed on lightweight computing carriers such as embedded devices for wider application.

[0007] To solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A lightweight face pose estimation method based on multi-task learning, which acquires a face image to be processed, extracts the corresponding convolutional feature map from the face image to be processed by using a convolutional network, and then inputs the convolutional feature map of the face image to be processed into a pre-trained lightweight face alignment and pose estimation model to obtain the face alignment and pose estimation information of the face image to be processed.

[0009] The lightweight face alignment and pose estimation model includes a pose estimation task network based on inverted residuals and a face alignment task network based on inverted residuals.

[0010] The pose estimation task network is used to perform inverted residual processing and convolutional processing on the convolutional feature map of the face image to be processed, output the intermediate feature map of the face image to be processed obtained by the inverted residual processing to the face alignment task network, and perform fully connected multi-scale fusion processing on each processing result in the pose estimation task network to obtain the pose rotation matrix of the face image to be processed.

[0011] The face alignment task network is used to further perform inverted residual processing and convolutional processing on the intermediate feature map, and perform fully connected multi-scale fusion processing on the processing results within the face alignment task network to obtain the face feature point coordinate matrix of the face image to be processed;

[0012] The pose rotation matrix and the face feature point coordinate matrix of the face image to be processed are used as face alignment and pose estimation information for face pose estimation.

[0013] As a preferred solution, the pose estimation task network includes two first inverted residual modules and a first convolutional module cascaded in sequence, and a first fully connected layer connecting the two first inverted residual modules and the first convolutional module;

[0014] The two first inverted residual modules are used to perform two inverted residual processes on the convolutional feature map of the face image to be processed, perform two downsampling feature extractions on the convolutional feature map to obtain the intermediate feature map of the face image to be processed, and output it to the face alignment task network;

[0015] The first convolutional module is used to perform convolutional feature extraction on the intermediate feature map to obtain the intermediate convolutional feature map of the face image to be processed;

[0016] The first fully connected layer is used to perform fully connected multi-scale fusion processing on the processing results of the two first inverted residual modules and the first convolutional module to obtain the pose rotation matrix of the face image to be processed.

[0017] As a preferred solution, the face alignment task network includes a second inverted residual module and two second convolutional modules cascaded in sequence, and a second fully connected layer connecting the second inverted residual module and the two second convolutional modules;

[0018] The second inverted residual module is used to further perform inverted residual processing on the intermediate feature map of the face image to be processed and perform downsampling feature extraction on the intermediate feature map again;

[0019] The two second convolutional modules are used to further perform convolutional feature extraction processing on the downsampling feature extraction results of the intermediate feature map;

[0020] The second fully connected layer is used to perform fully connected multi-scale fusion processing on the processing results of the second inverted residual module and the two second convolutional modules to obtain the face feature point coordinate matrix of the face image to be processed.

[0021] As a preferred solution, both the first inverted residual module or the second inverted residual module are inverted residual networks; the inverted residual network includes a 1×1 convolutional upsampling layer, a 3×3 depth convolutional layer, and a 1×1 convolutional downsampling layer cascaded in sequence.

[0022] As a preferred solution, the training method of the lightweight face alignment and pose estimation model is as follows:

[0023] Input a training set composed of face sample images that have completed face pose estimation and face feature point marking into the lightweight face alignment and pose estimation model, construct a total loss function including a pose estimation task loss function and a face alignment task loss function, and optimize and update the model parameters of the lightweight face alignment and pose estimation model with the goal of minimizing the total loss function, thereby training the lightweight face alignment and pose estimation model.

[0024] As a preferred solution, the pose estimation task loss function is:

[0025]

[0026] where R p is the pose rotation matrix predicted by the pose estimation task network for the face sample image, and R gt is the true label pose rotation matrix of the face sample image; tr(·) is the trace operator of the matrix; T is the transpose symbol;

[0027] The face alignment task loss function is:

[0028]

[0029] where is the position coordinate of the i-th feature point in the face feature point coordinate matrix predicted by the pose estimation task network for the face sample image, and P i (x,y) is the position coordinate of the i-th feature point in the true label face feature point coordinate matrix of the face sample image, i ∈ {1, 2,..., N}, and N is the number of feature points; ||·|| represents the L2 norm operation;

[0030] The total loss function L is the weighted sum of the pose estimation task loss function L1 and the face alignment task loss function L2:

[0031] L = L1 + λL2;

[0032] λ is the weight coefficient.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The lightweight face pose estimation method based on multi-task learning of the present invention designs a lightweight face alignment and pose estimation model by utilizing the strong correlation between face pose and feature points. Both the pose estimation task network and the face alignment task network in the model are constructed based on inverted residuals, which can effectively reduce the computational amount and data information redundancy.

[0035] 2. In the lightweight face alignment and pose estimation model proposed by the present invention, the pose estimation task network uses a low-dimensional information pose rotation matrix as the output to represent the face pose, which can solve the information loss problem caused by continuous rotation of Euler angles and the information redundancy problem of ordinary rotation matrices. At the same time, a single-branch network with a sequence sharing strategy is introduced by combining the face alignment task network and the pose estimation task network to further perform inverted residual downsampling and convolution processing on the intermediate feature map obtained by the pose estimation task network, resulting in a face feature point coordinate matrix. Based on this, face alignment and pose estimation are carried out, improving the utilization efficiency of the feature map and also reducing the complexity of the multi-task learning model.

[0036] 3. The lightweight face pose estimation method based on multi-task learning proposed by the present invention uses a lightweight face alignment and pose estimation model with an inverted residual as the backbone network for estimation. With the lightweight and multi-scale feature map fusion processing of this model, it can well solve the problems of excessive information redundancy and huge data capacity occupation of the face pose estimation model while ensuring the accuracy of face pose estimation, and is more conducive to being deployed on lightweight computing carriers such as embedded devices for wider applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the structure and processing flow of the lightweight face alignment and pose estimation model in the present invention.

[0038] Figure 2 It is a schematic diagram of the structure of the inverted residual network.

[0039] Figure 3 It is the pose results of some samples in the AFLW2000 test set in the embodiment.

[0040] Figure 4 It is the pose results of some samples in the BIWI test set in the embodiment.

[0041] Figure 5 It is the feature point localization results of some samples in the WFLW dataset in the embodiment.

[0042] Figure 6 It is the CED graph of the WFLW test set in the embodiment.

[0043] Figure 7 It is a curve graph of the change of the FPA-Net loss function under different sharing strategies in the embodiment.

[0044] Figure 8 It is a schematic diagram of the structure and processing flow of the ordinary hard sharing version FPA-1H shown in the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention generally described and illustrated in the accompanying drawings herein can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0046] The present invention provides a lightweight face pose estimation method based on multi-task learning. After obtaining a face image to be processed, the method uses a convolutional network to extract the corresponding convolutional feature map of the face image to be processed, and then inputs the convolutional feature map of the face image to be processed into a pre-trained lightweight face alignment and pose estimation model to obtain face alignment and pose estimation information of the face image to be processed.

[0047] The lightweight face alignment and pose estimation model is the key design of the present invention. The lightweight face alignment and pose estimation model proposed by the present invention is a multi-task learning model based on a convolutional neural network, which can be abbreviated as FPA-Net (Face Pose Estimation and Alignment Network). It includes a pose estimation task network based on inverted residuals and a face alignment task network based on inverted residuals; wherein:

[0048] The pose estimation task network is used to perform inverted residual processing and convolutional processing on the convolutional feature map of the face image to be processed, output the intermediate feature map of the face image to be processed obtained by the inverted residual processing to the face alignment task network, and perform fully connected multi-scale fusion processing on each processing result in the pose estimation task network to obtain the pose rotation matrix of the face image to be processed.

[0049] The face alignment task network is used to further perform inverted residual processing and convolutional processing on the intermediate feature map, and perform fully connected multi-scale fusion processing on each processing result in the face alignment task network to obtain the face feature point coordinate matrix of the face image to be processed.

[0050] Thus, the pose rotation matrix and the face feature point coordinate matrix of the face image to be processed are obtained, which are used as face alignment and pose estimation information for face pose estimation.

[0051] Specifically, the model structure of the lightweight face alignment and pose estimation model in the present invention is asFigure 1 as shown

[0052] The pose estimation task network includes two first inverted residual modules (Block1, Block2) and a first convolutional module (Conv2) cascaded in sequence, and a first fully connected layer connecting the two first inverted residual modules and the first convolutional module; the two first inverted residual modules are used to perform two inverted residual processes on the convolutional feature map obtained by processing the face image to be processed through a convolutional network (Conv1), perform two downsampling feature extractions on the convolutional feature map, obtain an intermediate feature map (F2) of the face image to be processed, and output it to the face alignment task network; the first convolutional module is used to perform convolutional feature extraction on the intermediate feature map to obtain an intermediate convolutional feature map (F3) of the face image to be processed; the first fully connected layer is used to perform fully connected multi-scale fusion processing on the processing results (F1, F2, F3) of the two first inverted residual modules and the first convolutional module to obtain a pose rotation matrix (Matrix) of the face image to be processed. The obtained pose rotation matrix (Matrix) is a 3×2 matrix and contains 6 parameters.

[0053] The face alignment task network includes a second inverted residual module (Block3) and two second convolutional modules (Conv3, Conv4) cascaded in sequence, and a second fully connected layer connecting the second inverted residual module and the two second convolutional modules; the second inverted residual module is used to further perform an inverted residual process on the intermediate feature map (F2) of the face image to be processed, and perform another downsampling feature extraction (F4) on the intermediate feature map; the two second convolutional modules are used to further perform convolutional feature extraction processing (F5, F6) on the downsampling feature extraction result of the intermediate feature map; the second fully connected layer is used to perform fully connected multi-scale fusion processing on the processing results (F4, F5, F6) of the second inverted residual module and the two second convolutional modules to obtain a face feature point coordinate matrix (Landmarks) of the face image to be processed. The number of feature points contained in the obtained face feature point coordinate matrix (Landmarks) is N, and each feature point is represented by 2 values (x, y) for its coordinates. Therefore, the face feature point coordinate matrix contains 2N parameters.

[0054] In the solution of the present invention, both the first inverted residual module and the second inverted residual module are inverted residual networks. The structure of the inverted residual network is as Figure 2As shown in the figure, it includes a cascaded 1×1 convolutional upsampling layer, a 3×3 depthwise separable convolutional layer, and a 1×1 convolutional downsampling layer in sequence. In this way, the dimension is first increased through 1×1 convolution, and then the 3×3 depthwise separable convolution is used. The input and output channels of the depthwise separable convolution are higher, so more information can be extracted. Finally, 1×1 convolution is used for downsampling to restore the data dimension; moreover, since the 3×3 depthwise separable convolution is performed in a low-dimensional space, this operation can reduce the computational complexity.

[0055] Therefore, it can be seen that the lightweight face pose estimation method based on multi-task learning of the present invention designs a lightweight face alignment and pose estimation model by utilizing the strong correlation between face pose and feature points. The pose estimation task network and the face alignment task network in this model are both based on inverted residuals, which can effectively reduce the computational complexity and data information redundancy; among them, the pose estimation task network uses a pose rotation matrix with low-dimensional information as the output to represent the face pose, which can solve the problem of information loss caused by continuous rotation of Euler angles and the problem of information redundancy of ordinary rotation matrices; at the same time, a single-branch network with a sequence sharing strategy is introduced by combining the face alignment task network and the pose estimation task network to continue to perform inverted residual downsampling and convolutional processing on the intermediate feature map obtained by the pose estimation task network to obtain the face feature point coordinate matrix; based on this, face alignment and pose estimation are performed, which improves the utilization efficiency of the feature map, can also reduce the complexity of the multi-task learning model, and through multi-task learning, can well guarantee the accuracy of face pose estimation and alignment.

[0056] That is to say, the lightweight face pose estimation method based on multi-task learning proposed by the present invention uses a lightweight face alignment and pose estimation model with inverted residuals as the backbone network for estimation. With the lightweight and multi-scale feature map fusion processing of this model, it can well solve the problems of excessive information redundancy and huge data capacity occupation of the face pose estimation model on the premise of ensuring the accuracy of face pose estimation, and is more conducive to being deployed on lightweight computing carriers such as embedded devices for wider applications.

[0057] Model parameters

[0058] Next, through examples, the network parameters of the pose estimation task network and the face alignment task network in the lightweight face alignment and pose estimation model proposed by the present invention will be shown.

[0059] Table 1 Pose Estimation Task Network Parameters

[0060]

[0061] Table 2 Face Alignment Task Network Parameters

[0062]

[0063] Tables 1 and 2 respectively list and show examples of the network parameters of the pose estimation task network and the face alignment task network. Each row represents different types of convolutional operations and parameters. Conv is a normal convolution, and Bottleneck is an inverted residual block. Each operation in each row is repeated n times. The repeated layers have the same number of output channels c, stride s, and expansion factor t. In Table 1, the convolutional network Conv1 that the face image initially passes through can include two 3×3 convolutional layers; the first inverted residual module Block1 can include three inverted residual blocks, and Block2 can include one inverted residual block, which are respectively processed to obtain feature maps F1 and F2. The feature map F2 is respectively used as the input of the first convolutional module Conv2 and the face alignment task network; the first convolutional module Conv2 can include two convolutional layers of 3×3 and 7×7, and is processed to obtain feature map F3; the three feature maps F1, F2, and F3 are the feature maps participating in multi-scale fusion for the pose estimation task. After passing through the first fully connected layer including two fully connected (Full Connection) layers, a 3×2 rotation matrix and an intermediate feature map are output. In Table 2, the input of the face alignment task network is the feature map F2 in Table 1. After further inverted residual downsampling by the second inverted residual module Block3 and further convolutional processing by two second convolutional modules Conv3 (3×3 convolution) and Conv4 (7×7 convolution), three feature maps F4, F5, and F6 are generated to participate in the multi-scale fusion of the face alignment task, and the face feature point coordinates are obtained after full connection. The model parameters of FPA-Net are adjustable. By changing the input image size (Input), the output channels (c), and the number of feature points (N), models with different effects can be obtained, but the model size will also change, which is reflected in the code we have made public. Tables 1 and 2 show the parameters corresponding to the smallest model network in the code, which are also the parameters of the subsequent embodiment FPA-1X of the invention.

[0064] Loss function

[0065] In the present invention, the training method of the lightweight face alignment and pose estimation model is as follows: input a training set composed of face sample images that have been pre-completed with face pose estimation and face feature point marking into the lightweight face alignment and pose estimation model, construct a total loss function including a pose estimation task loss function and a face alignment task loss function, and optimize and update the model parameters of the lightweight face alignment and pose estimation model with the goal of minimizing the total loss function, and then train the lightweight face alignment and pose estimation model.

[0066] In the research process of the present invention's solution, different loss functions were used, and it was found that the design of the loss function had a significant impact on the results. As the most commonly used representation method of pose information, Euler angles are composed of pitch, yaw, and roll. The result after continuous rotation of pitch, yaw, and roll in a fixed order is the true face pose. Rotation information is lost during the continuous rotation process, and using Euler angles as the parameters of the loss function is not an optimal method. The 3×3 rotation matrix, as a representation method of the eigenvector of the face coordinate system in three-dimensional space, consists of three 3×1 unit vectors a1, a2, and a3, without the problem of information loss caused by continuous rotation, and is more accurate for describing face poses. For comparison with previous research methods, Euler angles are still used to represent the original data input and the final result display in the example part, and Euler angles and rotation matrices can be converted to each other. Due to the orthogonality of the rotation matrix, the cross product of any two unit vectors in the three-dimensional rotation matrix can obtain the third unit vector a3. Therefore, the third unit vector a3 is discarded, and a 3×2 rotation matrix is used as the parameter of the loss function for the pose estimation task.

[0067]

[0068]

[0069]

[0070] Equation (1) is to obtain a 3×2 rotation matrix by deleting the last column of the 3×3 rotation matrix, denoted by g GS Expression. Equation (2) is the 3×3 rotation matrix, obtained from the 3×2 rotation matrix output by the model, denoted by f GS Expression. The corresponding relationship between the column vectors a i and b i (i = 1, 2, 3) is shown in Equation (3), where N(·) is vector normalization.

[0071]

[0072] Equation (4) is the loss function of the pose estimation task network, represented by the geodesic error L1 of the rotation matrix. Among them, R p is the pose rotation matrix of the face sample image predicted by the pose estimation task network, and R gt is the true label pose rotation matrix of the face sample image; tr(·) is the trace operator of the matrix; T is the transpose symbol.

[0073]

[0074] Equation (5) is the loss function of the face alignment task network, represented by the sum of the errors L2 of all feature points. Among them, For the pose estimation task network to predict the position coordinates of the \(i\)-th feature point in the facial feature point coordinate matrix of the facial sample image, \(P\) i (x, y) are the position coordinates of the \(i\)-th feature point in the true label facial feature point coordinate matrix of the facial sample image, \(i\in\{1, 2, \ldots, N\}\), where \(N\) is the number of feature points; \(\|\cdot\|\) represents the L2 norm operation;

[0075] \(L = L_1+\lambda L_2\ (6)\)

[0076] Equation (6) is the total loss function \(L\) of the entire network, which is expressed as the weighted sum of the pose estimation task loss function \(L_1\) and the face alignment task loss function \(L_2\); \(\lambda\) is the weight coefficient, and its value can be obtained through experimental prior experience.

[0077] Examples and Analyses:

[0078] In order to study the technical advantages of the proposed solution in this application, three experimental objectives are designed for the accuracy, robustness of the lightweight face alignment and pose estimation model in this invention, and the effectiveness of the sequence sharing strategy, namely the pose estimation accuracy experiment, the face alignment accuracy experiment, and the ablation experiment of the ordinary hard sharing strategy and the sequence sharing strategy.

[0079] Pose Estimation Experiment

[0080] In the pose estimation experiment, different datasets and the same dataset are designed to compare and verify the robustness of the model under different dataset samples.

[0081] This experiment involves three datasets: 300W-LP (see the literature "Zhu X, Lei Z, Yan J, et al. High-fidelity pose and expression normalization for face recognition in the wild [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2015:787-796"), AFLW2000 (see the literature "Fanelli G, Dantone M, Gall J, et al. Random forests for real time 3d face analysis [J]. International journal of computer vision, 2013, 101(3):437-458"), and BIWI (see the literature "Wu W, Qian C, Yang S, et al. Look at boundary: A boundary-aware face alignment algorithm [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018:2129-2138"). 300W-LP is a large-pose face dataset generated from the 300W dataset, including a total of 66,225 faces collected from multiple databases, and 122,450 faces after image flipping. The true labels are Euler angles. The AFLW2000 dataset contains the first 2,000 images from the AFLW dataset, and these images are labeled with true Euler angles as labels. The BIWI dataset has a total of 15,678 images, which were created in an indoor environment with 20 participants. The face poses were captured by a depth camera, and the Euler angles were calculated based on the given information.The pose labels use their original data. The feature point labels and face bounding boxes are generated by 3D-FAN (see the literature "Bulat A, Tzimiropoulos G. How far are we from solving the 2d&3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)[C] / / Proceedings of the IEEE International Conference on Computer Vision. 2017:1021-1030"). The size of the face bounding box is uniformly expanded to 120%, and the size of the cropped face image is adjusted to 112×112, and samples with Euler angles greater than 99° are excluded. Two common experimental strategies are used in this experiment: Experimental strategy 1 is to train and test on different datasets; Experimental strategy 2 is to train and test on the same dataset.

[0082] Verification between different datasets

[0083] The 300W-LP dataset is used for training, and two real datasets AFLW2000 and BIWI are used for testing. The minimum parameter model FPA-1X (parameters are shown in Table 1 and Table 2) and the more accurate FPA-8X in the lightweight face alignment and pose estimation model proposed by the present invention are selected for experiments. FPA-1X model: The optimizer is Adam, the learning rate is 3×10 -3 , the weight decay is 10 -6 , the momentum is 0.9, the number of feature points N is 68, and the weight λ is 100. FPA-8X model: The optimizer is Adam, the learning rate is 3×10 -3 , the weight decay is 10 -7 , the momentum is 0.9, the number of feature points N is 68, the weight λ is 100, and the number of output channels is 8 times that of the FPA-1X output channels. The evaluation metrics are the number of parameters of the model in MB, the mean absolute error and the total mean error (MEAN) of the Euler angles in degrees in the three directions of face Pitch, Yaw and Roll.

[0084] Meanwhile, some comparison models in the prior art were also selected to participate in the experimental comparison. As shown in Table 3, on the AFLW2000 dataset, FPA-8X has the smallest values among all methods in terms of the three indicators of yaw, pitch, and total average error. Taking the total average error MEAN as the measurement standard, compared with the original error-minimum model Img2pose, the accuracy of the FPA-8X model has increased by 0.3%. When the MEAN accuracy of FPA-1X only decreases by 0.2°, the model size is reduced to 1 / 300 of the Img2pose model, and its number of parameters reaches the minimum.

[0085] Table 3 Comparison of Pose Estimation Errors on the AFLW2000 Dataset (All Training Samples are from 300W-LP)

[0086]

[0087] Among the above-listed comparison models:

[0088] For the FAN-12points model, see the literature "Bulat A, Tzimiropoulos G. How far are we from solving the 2d&3d face alignment problem (and a dataset of 230,000 3d facial landmarks) [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2017:1021-1030";

[0089] For the Hopenet model, see the literature "Ruiz N, Chong E, Rehg J M. Fine-grained head pose estimation without keypoints [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018:2074-2083";

[0090] For the SSR-Net-MD model, see the literature "Yang T Y, Huang Y H, Lin Y Y, et al. SSR-Net: A Compact Soft Stagewise Regression Network for Age Estimation [C] / / IJCAI. 2018, 5(6):7";

[0091] The FSA-NET model can be found in the literature "Yang T Y, Chen Y T, Lin Y Y, et al. Fsa-net: Learning fine-grained structure aggregation for head pose estimation from a single image[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019:1087-1096";

[0092] The WHENet-V model can be found in the literature "Yijun Zhou and James Gregson, "Whenet: Real-time fine-grained estimation for wide range head pose," in 31st British Machine Vision Conference 2020, BMVC2020, Virtual Event, UK, September 7-10, 2020. 2020, BMVA Press";

[0093] The FDN model can be found in the literature "Zhang H, Wang M, Liu Y, et al. FDN: Feature decoupling network for head pose estimation[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07):12789-12796";

[0094] The Img2pose model can be found in the literature "Albiero V, Chen X, Yin X, et al. img2pose: Face alignment and detection via 6dof, face pose estimation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:7617-7627";

[0095] The 6DRepNet model can be found in the literature "Hempel T, Abdelrahman AA, Al-Hamadi A. 6D Rotation Representation For Unconstrained Head Pose Estimation[J / OL]. arXiv preprint arXiv:2202.12555".

[0096] Table 4 shows the error comparison results of Experimental Strategy 1 with the optimal models in the past five years on the BIWI dataset. There are obvious differences between the images in the BIWI dataset and those in the 300W-LP dataset used for model training. Also measured by the total average error MEAN, although the MEAN of FPA-8X is 0.27° higher than that of the model with the smallest error, 6DRepNet, its model size is only 15.5% of the latter. And in the case of a 0.51° loss, the model size of FPA-1X is also only 1 / 300 of the latter, and its number of parameters is also the smallest. The lightweight face alignment and pose estimation model FPA-Net proposed in the present invention is multi-task. After discarding the face alignment branch model, the pose estimation still works normally. The single-task model size of the FPA-1X pose estimation model is 0.4 Mb, and the model is more suitable for deployment on embedded devices.

[0097] Table 4 Comparison of Pose Estimation Errors on the BIWI Dataset (All Training Samples are from 300W-LP)

[0098]

[0099] Internal Experiments on the Same Dataset

[0100] Randomly divide 70% of BIWI as the training set and 30% as the test set. Select the model with the smallest number of parameters, FPA-1X, and the parameters are the same as those in Experimental Strategy 1. BIWI ( Figure 4 ) is more single in terms of illumination and color compared to AFLW-2000 ( Figure 3 ), and it is prone to overfitting. To avoid this situation, we perform internal random padding and random shearing on the images before input.

[0101] Table 5 Comparison of Pose Estimation Errors on the BIWI Dataset (70% Samples for Training, 30% Samples for Validation)

[0102]

[0103] As shown in Table 5, on the BIWI dataset, the Yaw, Pitch, and the total average error MEAN of FPA-1X are smaller than those of 6DRepNet. Its MEAN index is 2.16°, which is 19% higher than 2.66° of 6DRepNet. Figure 3and Figure 4 shows some samples in the AFLW-2000 and BIWI datasets, as well as the pose prediction results of the FPA-1X model; Figure 3 In, the upper row is the labeled pose, and the lower row is the predicted pose (FPA-1X trained on 300W-LP) Figure 4 In, the upper row is the labeled pose, the middle row is the predicted pose (FPA-1X trained on 300W-LP), and the lower row is the predicted pose (FPA-1X trained on 70% BIWI). The red is the facial normal vector, pointing directly in front of the face, the green is the horizontal vector on the left side of the face, and the blue points vertically downward of the face.

[0104] Face alignment experiment

[0105] The network is mainly designed for pose estimation, and the exclusive model size for the face alignment task is only 0.1MB. To test the extreme performance of the face alignment branch, we still conduct experiments on the most challenging WFLW dataset. Taking the current model SLPT with the smallest error as the reference basis, we compare it with the lightweight models MobileNet-V2 and ASMNet. The WFLW dataset contains the positions of 98 accurate facial feature points, as well as information on different facial image attributes. As Figure 5 shown, from left to right are the samples of the WFLW test set for: Pose, Expression, Illumination, Make-Up, Occlusion, Blur. The upper row is the real feature points, and the lower row is the facial feature points predicted by FPA-1X. The pose information of the WFLW dataset is relabeled using the PnP method (see the literature "Lepetit V, Moreno-Noguer F, Fua P. Epnp: An accurate o(n) solution to the pnp problem[J]. International journal of computer vision, 2009"), and the original feature point labels are used to experiment on the face alignment task.

[0106] Experimental strategy: Divide WFLW into 75% as the training set and 25% for the test set. Train using the FPA-1X model, with the parameters consistent with Experimental Strategy 1, and the number of feature points N is 96. Use the Normalized Mean Error (NME), the Failure Rate (FR) with a maximum error of 0.1, and the area under the curve (AUC) to measure the face alignment task.

[0107] Formula (7) is the calculation method of NME (%), pi and are the true coordinates and predicted coordinates of the \(i\)-th feature point respectively, \(N\) is the number of feature points, \(d\) is the distance between the outer corners of the eyes. \(FR\) is the proportion of samples in the test set with NME greater than a certain threshold \(e\). The smaller its value, the better the result. \(AUC\) is the area under the cumulative error distribution (CED) curve. Taking \(e = 0.1\) as an example, values \(e_i\) (\(e_0, e_1, \cdots, e_{1000}\)) are taken at intervals of \(0.0001\) from \(0\) to \(0.1\), the proportion of samples less than \(e_i\) is calculated and connected in sequence to obtain the CED curve as shown in Figure 6 . The area under the curve of the graph is \(AUC\). The higher the \(AUC\), the better the result.

[0108]

[0109] Table 6 shows the performance of different methods on all test set samples of WFLW (Test set in the table) and sample sets with different attribute labels. SLPT is the model with the smallest error in the current literature, and the model size is 51MB. Taking SLPT as the benchmark, the performance of FPA-Net and similar lightweight models ASMNet and MobileNetV2 is explored. From the perspective of the number of model parameters, FPA-1X has the smallest model with 0.5MB of parameters, which is 35% of ASMNet and only 1% of SLPT. In the performance on the entire test set Test set, the normalized mean errors of ASMNet and MobileNetV2 are 10.77% and 9.41% respectively, which are 2.37 times and 2.27 times the error of SLPT. The error of FPA-Net is only 1.64 times that of SLPT, and the accuracy is improved by 27.63% compared with MobileNetV2. Taking the failure rate with the maximum error of 0.1 as an index, the failure rate of FPA-1X is 15.15%, which is better than 39.12% of ASMNet and 30.04% of MobileNetV2. Figure 6 is the CED graph of FPA-1X. The area under the curve is \(AUC\). It can be seen from Table 6 that the \(AUC\) of FPA-1X is 0.404, which is close to SLPT and better than ASMNnet and MobileNetV2.

[0110] Table 6 Comparison of face alignment errors on the WFLW dataset

[0111]

[0112]

[0113] Among the above-listed comparison models:

[0114] The MobileNetV2 model and the ASMNet model can be found in the literature "Bulat A, Tzimiropoulos G. How far are we from solving the 2d&3d face alignment problem (and a dataset of 230,000 3d facial landmarks) [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2017:1021-1030";

[0115] The SLPT model can be found in the literature "Fanelli G, Dantone M, Gall J, et al. Random forests for real time 3d face analysis [J]. International journal of computer vision, 2013, 101(3):437-458".

[0116] Ablation experiments on sequence sharing strategy and hard sharing strategy

[0117] The sequence sharing strategy can effectively reduce the number of model parameters. However, from the perspective of traditional coarse / fine-grained feature classification, this strategy is radical because there is no subordinate relationship between face pose and facial feature points. Therefore, in this part, ablation experiments are conducted to explore the influence of ordinary hard sharing and sequence sharing on the accuracy of face pose estimation. By adding inverted residual layers and changing the position of the face alignment branch task, the ordinary hard sharing version FPA-1H as shown in Figure 8 can be obtained. The model size of FPA-1H (ordinary hard sharing) is 0.9MB, which is higher than 0.5MB of FPA-1X (sequence sharing). The number of parameters of sequence sharing is reduced by 44% compared with ordinary hard sharing.

[0118] The experiment is carried out with the same parameters as in Example FPA-1X. The changes of the loss functions of the FPA-1X model training and FPA-1H with the number of training rounds are as shown in Figure 7 It can be seen that there is no obvious difference in the changes of the loss functions of the two strategies, indicating that the networks of the two strategies both fit the training set samples well, and the sequence sharing strategy does not affect the model training process.

[0119] Table 7 Comparison of error results between FPA-1X and FPA-1H

[0120]

[0121] Table 7 shows the accuracy comparison of the pose estimation tasks of FPA-1X and FPA-1H on the AFLW2000 dataset. The total average error (MEAN) of the model proposed by the method of the present invention is 4.11°, which is lower than the average error of 4.23° of the ordinary hard sharing version. It proves that the model of the method of the present invention improves the model accuracy while reducing the model parameters, and verifies the effectiveness of the strategy.

[0122] General description

[0123] The present invention proposes a lightweight face alignment and pose estimation model (FPA-Net network) to solve the face pose estimation and face alignment tasks. The model size is much smaller than the current mainstream methods. In the pose estimation task, when the model img2pos and 6DResNet with the smallest errors on the AFLW2000 and BIWI datasets have similar accuracies, the smallest model size is only 0.33% of theirs. In the face alignment task, both the number of parameters and the accuracy exceed those of similar methods. The lightweight face alignment and pose estimation model proposed by the present invention adopts a sequential sharing strategy, which can improve the model accuracy while reducing the model parameters, introduces multi-scale feature fusion to extract information from feature maps of different sizes, and uses a 3×2 rotation matrix as the network output to make the model more accurate.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Those of ordinary skill in the art should understand that any modifications or equivalent replacements of the technical solutions of the present invention without departing from the purpose and scope of the present technical solution shall be covered by the scope of the claims of the present invention.

Claims

1. A lightweight face pose estimation method based on multi-task learning, characterized in that, Obtain a face image to be processed, extract the convolution feature map of the face image to be processed using a convolutional network, and then input the convolution feature map of the face image to be processed into a pre-trained lightweight face alignment and pose estimation model to obtain the face alignment and pose estimation information of the face image to be processed; The lightweight face alignment and pose estimation model includes a pose estimation task network based on inverted residuals and a face alignment task network based on inverted residuals; The pose estimation task network is used to perform inverted residual processing and convolutional processing on the convolution feature map of the face image to be processed, output the intermediate feature map of the face image to be processed obtained by the inverted residual processing to the face alignment task network, and perform fully connected multi-scale fusion processing on each processing result in the pose estimation task network to obtain the pose rotation matrix of the face image to be processed; The face alignment task network is used to further perform inverted residual processing and convolutional processing on the intermediate feature map, and perform fully connected multi-scale fusion processing on each processing result in the face alignment task network to obtain the face feature point coordinate matrix of the face image to be processed; The pose rotation matrix and the face feature point coordinate matrix of the face image to be processed are used as face alignment and pose estimation information for face pose estimation; the training method of the lightweight face alignment and pose estimation model is: Input a training set composed of face sample images with pre-completed face pose estimation and face feature point marking into the lightweight face alignment and pose estimation model, construct a total loss function including a pose estimation task loss function and a face alignment task loss function, and optimize and update the model parameters of the lightweight face alignment and pose estimation model with the goal of minimizing the total loss function, thereby training the lightweight face alignment and pose estimation model; The pose estimation task loss function is: where R p is the pose rotation matrix of the face sample image predicted by the pose estimation task network, and R gt is the true label pose rotation matrix of the face sample image; tr(·) is the trace operator of the matrix; T is the transpose symbol; The face alignment task loss function is: Among them, is the position coordinate of the i-th feature point in the facial feature point coordinate matrix predicted by the pose estimation task network for the face sample image, P i (x, y) is the position coordinate of the i-th feature point in the true label facial feature point coordinate matrix of the face sample image, i ∈ {1, 2, …, N}, where N is the number of feature points; ||·|| represents the L2 norm operation; The total loss function L is the weighted sum of the pose estimation task loss function L1 and the face alignment task loss function L2: L = L1 + λL2; λ is the weight coefficient.

2. The lightweight face pose estimation method based on multi-task learning according to claim 1, wherein The pose estimation task network includes two first inverted residual modules and a first convolutional module cascaded in sequence, and a first fully connected layer connecting the two first inverted residual modules and the first convolutional module; The two first inverted residual modules are used to perform two inverted residual processes on the convolution feature map of the face image to be processed, perform two downsampling feature extractions on the convolution feature map to obtain the intermediate feature map of the face image to be processed, and output it to the face alignment task network; The first convolutional module is used to perform convolutional feature extraction on the intermediate feature map to obtain the intermediate convolution feature map of the face image to be processed; The first fully connected layer is used to perform fully connected multi-scale fusion processing on each processing result of the two first inverted residual modules and the first convolutional module to obtain the pose rotation matrix of the face image to be processed.

3. The lightweight face pose estimation method based on multi-task learning according to claim 1, wherein, The face alignment task network includes a second inverted residual module and two second convolutional modules cascaded in sequence, and a second fully connected layer connecting the second inverted residual module and the two second convolutional modules; The second inverted residual module is used to further perform inverted residual processing on the intermediate feature map of the face image to be processed, and to perform downsampling feature extraction on the intermediate feature map again; The two second convolutional modules are used to further perform convolutional feature extraction processing on the downsampling feature extraction result of the intermediate feature map; The second fully connected layer is used to perform fully connected multi-scale fusion processing on the processing results of the second inverted residual module and the two second convolutional modules to obtain the face feature point coordinate matrix of the face image to be processed.

4. The lightweight face pose estimation method based on multi-task learning according to claim 2 or 3, characterized in that The first inverted residual module or the second inverted residual module is an inverted residual network; the inverted residual network includes a 1×1 convolutional upsampling layer, a 3×3 depth convolutional layer, and a 1×1 convolutional downsampling layer cascaded in sequence.

Images