A lightweight robust face alignment method and system based on multi-task learning

Abstract

The application relates to a lightweight robust face alignment method and system based on multi-task learning, which comprises the following steps: S1, collecting face images, and performing translation, scaling and rotation preprocessing on the face images to obtain a training set; S2, obtaining face key points L of the face images in the training set; using a pose estimation algorithm to mark 3D head pose Euler angles Theta of the face images in the training set, and converting the 3D head pose Euler angles Theta into a head pose rotation matrix R Θ ; S3, inputting the training set into a face alignment network to output face key point prediction values P and head pose rotation matrix prediction values R Φ , and respectively constructing a head pose loss function, a face key point loss function and a total loss function for training the face alignment network. The method provided by the application simultaneously aligns face key points and head poses using a lightweight model, and uses a head pose alignment task to assist the positioning of the face key points, thereby enhancing the robustness of the lightweight model to large-pose faces.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a lightweight robust face alignment method and system based on multi-task learning. BACKGROUND

[0002] Face alignment, also known as facial landmark localization, aims to locate the precise positions of a set of predefined facial landmarks (distributed in the regions of facial contour, left / right eyes, left / right eyebrows, nose, and upper / lower lip) from a face image. Face alignment is the basis of face recognition, expression analysis, face special effect, and fatigue detection, etc., and is designed to provide more abundant face information for subsequent tasks, and its detection accuracy plays a crucial role in subsequent tasks. However, it is still a great challenge to align faces in extreme cases such as large-angle head poses, exaggerated expressions, and occlusions.

[0003] Current face alignment methods mainly fall into two categories: machine learning methods and deep learning methods. For example, machine learning methods such as level regression trees have high detection accuracy and high computational efficiency, and are widely used in embedded systems. However, this kind of method is too dependent on statistical models, and the detection effect is poor for various extreme cases, so its application is limited to simple scenes such as frontal face recognition.

[0004] Deep learning-based methods mainly use convolutional neural network (CNN) models, and the specific detection methods can be divided into direct regression methods and heat map prediction methods. Direct regression methods directly use fully connected layers to regress the horizontal and vertical coordinates of all key points in the original image after CNN, which has high computational efficiency but is sensitive to the position distribution of the face. Heat map prediction methods use a specific CNN structure to output a high-resolution feature map, and then predict the planar distribution response map for each key point, taking the position with the highest response as the predicted coordinates of each key point. This method has high detection accuracy, but the calculation of high-resolution feature maps consumes a lot of memory and computational resources, making it impossible to be used as a pre-processing step for other face tasks.

[0005] For application scenarios such as 3D face special effect and 3D face reconstruction, using face key point detection or head pose estimation as a pre-processing step can reduce the computational overhead of using a generative model. Existing methods often use face key points and a set of predefined 3D key points to calculate the projection relationship to estimate the head pose, or use an independent head pose detection algorithm to obtain the head pose of the face image separately. Face key point detection and head pose estimation are completed as two steps, ignoring the strong correlation between the two. Therefore, how to realize face key point alignment and head pose alignment in a single model has become a problem to be solved. SUMMARY

[0006] In order to solve the above technical problems, the application provides a lightweight robust face alignment method based on multi-task learning and a system thereof.

[0007] The technical solution of the application is as follows: a lightweight robust face alignment method based on multi-task learning, comprising:

[0008] Step S1: collecting face images, and performing translation, scaling and rotation preprocessing on the face images, expanding samples and constructing a training set;

[0009] Step S2: labeling face key points L of the face images in the training set; using a pose estimation algorithm to label 3D head pose Euler angles Θ of the face images in the training set, and converting the 3D head pose Euler angles Θ into a head pose rotation matrix R Θ ;

[0010] Step S3: inputting the training set into a face alignment network, outputting face key point prediction value P and head pose rotation matrix prediction value R Φ , respectively constructing a head pose loss function, a face key point loss function and a total loss function, and training the face alignment network.

[0011] Compared with the prior art, the application has the following advantages:

[0012] 1. Face key point alignment and head pose alignment belong to two sub-technical fields, but in actual application, the two often appear at the same time. The head pose alignment method is usually independent of the face key point alignment method, or performs linear estimation based on the results of the face key point alignment method. The application adopts a deep learning method, takes the linear correlation of the two as the basis, and uses two fully connected layers to align the face key points and the head pose from the multi-scale features of the same model, thereby enhancing the practicability of the face alignment method.

[0013] 2. The parameter quantity of the lightweight convolutional network model is much smaller than that of the conventional convolutional network, so that it can be applied to embedded devices, but the robustness is poor. The application starts from the perspective of multi-task learning, and through the training of the head pose alignment of the model, it ensures that the model can learn the potential overall features of the face, assists the training of the face key point alignment according to the non-reversible linear constraint between the two fully connected layers, guides the consistency of the overall deformation of the face key points from the head pose, ensures the robustness of the face key point alignment of the model in the case of face occlusion and large-angle head pose, and improves the reliability of the lightweight face alignment model. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The flowchart of the lightweight robust face alignment method based on multi-task learning in the embodiments of the application;

[0015] Figure 2 A structure diagram of a face alignment network in an embodiment of the present application is shown in the figure.

[0016] Figure 3 A structure diagram of a face alignment network in an embodiment of the present application is shown in the figure.

[0017] Figure 4 A structure diagram of a face alignment network in an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0018] The present application provides a lightweight robust face alignment method based on multi-task learning, which uses a lightweight model to align face key points and head posture at the same time, and uses the head posture alignment task to assist the positioning of face key points, thereby enhancing the robustness of the lightweight model to large posture faces.

[0019] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with specific embodiments and in conjunction with the accompanying drawings.

[0020] Embodiment one

[0021] As shown in the figure, the present application provides a lightweight robust face alignment method based on multi-task learning, which includes the following steps: Figure 1

[0022] Step S1: Collect face images, and perform translation, scaling and rotation preprocessing on the face images to expand samples and construct a training set.

[0023] Step S2: Obtain face key points L of the face images in the training set; use a pose estimation algorithm to label 3D head posture Euler angles Θ of the face images in the training set, and convert the 3D head posture Euler angles Θ into a head posture rotation matrix R Θ .

[0024] Step S3: Input the training set into a face alignment network, and output face key point prediction value P and head posture rotation matrix prediction value R Φ , respectively, to construct a head posture loss function, a face key point loss function and a total loss function, which are used to train the face alignment network.

[0025] In one embodiment, the above step S1: collect face images, and perform translation, scaling and rotation preprocessing on the face images to expand samples and construct a training set.

[0026] ​The embodiment of the application uses two public data sets 300W (300 Faces in the Wild) and 300VW (300 Videos in the Wild) as face image samples, and each face image sample in the two data sets contains 68 face key point annotations. Through data augmentation of the face image samples, including scale scaling within 0.8 to 1.25 times, translation within ±10 pixels, rotation within ±30 degrees, and random brightness and saturation adjustment, the adjusted face image is constructed into a training set for training the face alignment network.

[0027] The training set used in the embodiment of the application already contains 68 key point annotations for each face image, so the coordinate values L of the key points can be directly obtained. However, it is difficult to manually annotate the head pose angle of the face image, and the embodiment of the application uses an existing high-precision pose detection algorithm to estimate the head pose of the face image and uses it as an annotation label.

[0028] In one embodiment, the pose estimation algorithm is used to annotate the 3D head pose Euler angle Θ of the face image in the training set in step S2 above, and the 3D head pose Euler angle Θ is converted into a head pose rotation matrix R Θ , specifically comprising:

[0029] The face image is input into the pose estimation algorithm to obtain the 3D head pose Euler angle Θ, which includes the pitch angle α, the yaw angle β and the roll angle γ. According to the following formulas (1)-(4), the head pose rotation matrix R Θ is calculated:

[0030]

[0031]

[0032]

[0033] R Θ =R z *R y *R x (4).

[0034] The embodiment of the application uses a high-precision head pose detection algorithm HopeNet based on ResNet50 to estimate the head pose of the face image. First, the sample set is uniformly scaled to 224x224 pixel size, and then input into HopeNet to obtain the corresponding pitch angle α, yaw angle β and roll angle γ three head pose Euler angles Θ, and then converted into a head pose rotation matrix R Θ .

[0035] In one embodiment, the step S3 described above: inputting the training set into the face alignment network, outputting the face key point prediction value P and the head pose rotation matrix prediction value R Φ , respectively, a head pose loss function, a face key point loss function and a total loss function are constructed for training the face alignment network, specifically including:

[0036] As shown in Figure 2 , the face alignment network constructed in the embodiment of the present application reduces the face image to 112x112 pixels as the input of the lightweight face alignment network to reduce the operation cost. The present application modifies the lightweight convolutional network MobileNetV3-small, removes the down-sampling operation in the first convolution down-sampling module to correspond to the lower resolution image input, and keeps the size of the subsequent calculation feature map consistent; and modifies the 1x1 convolution with an output channel number of 576 in the last layer of the feature extraction network to a 7x7 convolution with an output channel number of 144 to obtain global features. The last three scale features are selected, corresponding to the spatial channel size of 14x14x48, 7x7x96 and 1x1x144, the three features are globally pooled to obtain three features with a size of 1x1x48, 1x1x96 and 1x1x144, and then spliced into an output feature of 1x1x288, and finally two fully connected layers are used to predict the horizontal and vertical coordinates P of 68 face key points and the head pose rotation matrix R Φ containing 9 parameters.

[0037] The face alignment network constructed in the present application modifies the original network output structure on the basis of the lightweight convolutional network. Most lightweight convolutional networks have a structure of multi-level convolution and down-sampling. The face alignment network only uses the convolution module output feature maps corresponding to the last three down-sampling scales, splices the feature maps after global pooling operation, and uses two fully connected layers for prediction. The face alignment network can predict from multiple scale features, fully utilizing the global and local features at different resolution scales. Global features are beneficial to learning the overall information of the face, which is used to predict the head pose; and the local feature information is introduced on the basis of the overall information, so as to accurately predict the position of each face key point.

[0038] Meanwhile, the present application respectively constructs a head pose loss function, a face key point loss function and a total loss function for training the face alignment network, specifically including:

[0039] Step S31: based on the head pose rotation matrix R Θ , a head pose loss function loss pose is constructed, as shown in formula (5):

[0040] loss pose =||R Φ -RΘ ||1 (5)

[0041] wherein, R Φ is the head pose rotation matrix predicted by the face alignment network; ||·||1 is the L1 norm operator;

[0042] Step S32: based on the 3D head pose Euler angle Θ and the labeled face key point L, a face key point loss function loss points , as shown in formula (6):

[0043]

[0044] wherein, K is the number of head pose Euler angles, i.e. K=3; σ is a pose weight adjustment parameter, in the embodiment of the present application, σ=0.1; N is the number of predefined face key points; P j is the jth face key point plane coordinate value predicted by the face alignment network; L j is the jth face key point plane coordinate value in reality;

[0045] Step S33: a total loss function loss total is constructed, as shown in formula (7), which is used to train the face alignment network:

[0046] loss total =loss pose +loss points (7)

[0047] The training of the face alignment network in the embodiment of the present application adopts a multi-task learning mode, a batch of face images are input, and the prediction results of the corresponding 68 face key points and head pose rotation matrix are synchronously trained. The face alignment network learning the head pose information can assist the positioning of the face key points, especially for the face images with large angle poses, the feature parameters corresponding to the head pose can directly determine the approximate deformation state of the face key points. In fact, due to the similarity of the face, the plane key points of any face image can be linearly approximated and aligned according to the projection transformation by a group of invariant predefined 3D key points, so that the face alignment network in the present application adopts two fully connected layers to respectively predict the face key points and the pose rotation matrix, so as to ensure that the required features of the two have strong coupling. The feature of predicting the head pose can linearly predict the approximate plane key points, and then the multi-scale local features are used to individually and differently adjust the key points.

[0048] In the actual training calculation, the loss pose and the loss points are respectively constructed to form the loss functions of the head pose rotation matrix and the face key points, and the parameter training of the face alignment network is completed by minimizing the total loss function loss total .

[0049] The embodiment of the present application uses the above-mentioned 300W and 300VW data sets as a training set to train a face alignment network, and tests the trained face alignment network on the face samples of the WFLW data set. The detection results of part of the face key points and the head posture are as shown in Figure 3 It can be seen that the face alignment network proposed in the present application has high detection accuracy in face key point alignment and head posture alignment under the conditions of large angle posture, exaggerated expression, face occlusion and image blur, and has good robustness for face alignment in complex environments.

[0050] The present application provides a lightweight robust face alignment method based on multi-task learning, which is used to realize strong robust face alignment of a lightweight convolutional network for face images, including face key point alignment and head posture alignment. The multi-task output structure of the lightweight convolutional network model is improved. The structure of directly pooling the last layer features as the output feature vector and then using a fully connected layer for prediction is changed to pool the intermediate layer features of multiple scales and the output features of the last layer respectively, splice the features of each scale as the output feature vector, and connect two fully connected layers to predict the outputs of the face key points and the head posture respectively, so as to construct a multi-task face alignment network. This network structure simultaneously utilizes the local detail information of high-resolution shallow features and the overall facial information of low-resolution deep features, which helps to maintain the spatial consistency of the deep neural network in predicting local face key points and overall facial orientation.

[0051] Meanwhile, the planar coordinate values of the face key points and the head posture rotation matrix are used as the prediction values of the two fully connected layers, instead of directly predicting the 3D Euler angles of the head posture relative to the camera coordinate system. According to the linear projection relationship from the camera coordinate system to the image coordinate system, the two linear fully connected layers ensure the partial linear relationship between the planar key points and the head posture rotation matrix, and the strong constraint relationship enables the face alignment network to more accurately identify the face key points under large angle head posture.

[0052] Embodiment two

[0053] As shown in Figure 4 The embodiment of the present application provides a lightweight robust face alignment system based on multi-task learning, which includes the following modules:

[0054] The training set construction module 41 is used to collect face images, and pre-process the face images by translation, scaling and rotation, expand the samples, and construct a training set;

[0055] The face key point and head pose labeling module 42 is configured to label face key points L of face images in a training set; a pose estimation algorithm is used to label 3D head pose Euler angles Θ of face images in the training set, and the 3D head pose Euler angles Θ are converted into a head pose rotation matrix R Θ ;

[0056] The face alignment network training module 43 is configured to input the training set into a face alignment network, and output a face key point prediction value P and a prediction value R of the head pose rotation matrix. Φ A head pose loss function and a face key point loss function are respectively constructed to train the face alignment network.

[0057] The above embodiments are provided only for the purpose of describing the present application, and are not intended to limit the scope of the present application. The scope of the present application is defined by the appended claims. Various equivalent replacements and modifications made without departing from the spirit and principles of the present application shall be encompassed within the scope of the present application.

Claims

1. A lightweight robust face alignment method based on multi-task learning, characterized in that, The application relates to a face recognition method and device. Step S1: collecting a face image, performing translation, scaling and rotation preprocessing on the face image, expanding samples and constructing a training set; Step S2: obtaining the face key points of the face images in the training set ; using a pose estimation algorithm to label the 3D head pose Euler angles of the face images in the training set , and converting the 3D head pose Euler angles into a head pose rotation matrix ; Step S3: inputting the training set into a face alignment network to output a face key point prediction value and a prediction value of a head pose rotation matrix , respectively constructing a head pose loss function, a face key point loss function and a total loss function, for training the face alignment network, specifically comprising: Step S31: constructing a head pose loss function based on the head pose rotation matrix , constructing a head pose loss function As shown in equation (5): (5) wherein, a head pose rotation matrix predicted by the face alignment network; is an LI norm operator; Step S32: Based on 3D head pose Euler angles and annotated facial landmarks , construct a facial landmark loss function As shown in equation (6): (6) wherein K is the number of head pose Euler angles, i.e., K = 3; is a pose weight adjustment parameter, and N is the number of predefined facial landmarks; is the jth facial landmark plane coordinate value predicted by the face alignment network; is the jth facial landmark plane coordinate value of the real face. Step S33: constructing a total loss function As shown in equation (7), the face alignment network is trained. (7)。 2. A lightweight robust face alignment system based on multi-task learning, characterized in that, The application relates to a face recognition method and device. The application relates to a face recognition method and device. The application relates to a face recognition method and device. The face key point and head pose labeling module is configured to label face key points of face images in the training set ; label 3D head pose Euler angles of face images in the training set using a pose estimation algorithm , and convert the 3D head pose Euler angles into a head pose rotation matrix ; The training face alignment network module is configured to input the training set into a face alignment network to output a face key point prediction value and a prediction value of a head pose rotation matrix , and construct a head pose loss function and a face key point loss function, respectively, to train the face alignment network, specifically including: Step S31: constructing a head pose loss function based on the head pose rotation matrix , constructing a head pose loss function As shown in equation (5): (5) wherein, a head pose rotation matrix predicted by the face alignment network; is an LI norm operator; Step S32: Based on 3D head pose Euler angles and annotated facial landmarks , construct a facial landmark loss function as shown in equation (6): (6) wherein K is the number of head pose Euler angles, i.e., K = 3; is a pose weight adjustment parameter, and N is the number of predefined facial landmarks; is the jth facial landmark plane coordinate value predicted by the face alignment network; is the jth facial landmark plane coordinate value of the real face. Step S33: constructing a total loss function As shown in equation (7), the face alignment network is trained. (7)。

Images