Head pose estimation method and system

Abstract

The application provides a head pose estimation method and system, the method comprising: acquiring a two-dimensional face image to be recognized; performing face calibration on the two-dimensional face image to obtain a calibrated two-dimensional face image; acquiring face feature points on the calibrated two-dimensional face image; inputting the face feature points into a regression network to obtain a 6D rotation matrix, the regression network being a regression model based on 6D rotation representation; converting the 6D rotation matrix into a 3*3 rotation matrix according to orthogonal mapping; and converting the 3*3 rotation matrix into Euler angles, the Euler angles being used to represent the rotation angle of the head pose, so as to solve the problem that the recognition accuracy and robustness of the model-based pose estimation method and the appearance-based pose estimation method cannot be accurately represented at the same time.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a head pose estimation method and system. Background Technology

[0002] Human head pose estimation is the process of estimating a user's head pose in three-dimensional space from a two-dimensional image. Simply put, it estimates the turning angles of the user's head in a two-dimensional image when it is rotated along the X, Y, and Z axes in three-dimensional space. Head pose estimation has a wide range of applications, including virtual reality, human-computer interaction systems, driver assistance systems, classroom attention estimation, and rehabilitation games for children. In the AVATEA (Advanced Virtual Adaptive Technologies health) project, researchers plan to develop an interactive visual interface presented as a game to help children with neuromotor disorders with their rehabilitation training. One rehabilitation task involves improving children's head tracking ability. When therapists move objects in space, these children need to move their heads to track them. These exercises aim to improve children's ability to control their neck and head muscles.

[0003] The training described above can be transformed into a human-computer interactive game presented on a large screen, where various objects can move through animation, and children need to control the character's movement with correct head postures. With the development of computer vision technology, many research methods for head pose estimation have emerged. These methods can be divided into two main categories: model-based pose estimation methods and appearance-based pose estimation methods.

[0004] Model-based pose estimation methods, while yielding more accurate results, rely heavily on the accuracy of facial landmark detection. Factors such as occlusion and significant rotation can drastically reduce the accuracy of facial landmark detection, indirectly impacting the overall accuracy of head pose estimation. Appearance-based pose estimation methods directly extract texture features from 2D face images and correlate these features with 3D head pose. While appearance-based methods exhibit greater robustness to occlusion and significant rotation, their accuracy is lower than model-based methods. Furthermore, they are more cumbersome in applications requiring real-time estimation. In conclusion, the accuracy and robustness of both model-based and appearance-based pose estimation methods cannot be accurately represented simultaneously. Summary of the Invention

[0005] This application provides a head pose estimation method and system to solve the problem that the recognition accuracy and robustness of model-based pose estimation methods and appearance-based pose estimation methods cannot be accurately represented at the same time.

[0006] This application provides a head pose estimation method, including:

[0007] Obtain a 2D face image to be identified;

[0008] The two-dimensional face image is calibrated to obtain a calibrated two-dimensional face image;

[0009] Obtain facial feature points on the calibrated two-dimensional face image;

[0010] The facial feature points are input into the regression network to obtain a 6D rotation matrix. The regression network is a regression model based on the 6D rotation representation.

[0011] The 6D rotation matrix is ​​converted into a 3×3 rotation matrix according to the orthogonal mapping.

[0012] The 3×3 rotation matrix is ​​converted into Euler angles, which are used to represent the rotation angles of the head posture.

[0013] Optionally, the step of performing face calibration on the two-dimensional face image to obtain a calibrated face image includes:

[0014] The head position of the two-dimensional face image is located using the MTCNN face detection algorithm;

[0015] The two-dimensional face image is cropped and / or expanded according to the head position to obtain the face region;

[0016] Each of the facial regions is adjusted to the same image size to obtain a calibrated facial image.

[0017] Optionally, the step of obtaining facial feature points on the calibrated two-dimensional face image includes:

[0018] The calibrated two-dimensional face image is passed into a convolutional network to obtain a multidimensional matrix output by the convolutional network.

[0019] Transform the multidimensional matrix into a vector matrix;

[0020] The vector matrix is ​​compressed and then input into the fully connected layer;

[0021] The fully connected layer outputs 136-dimensional vector features, which are facial feature points, and the facial feature points are two-dimensional coordinate points.

[0022] Optionally, the step of inputting the facial feature points into a regression network to obtain a 6D rotation matrix includes: inputting the facial feature point sequence into a multilayer perceptron (MLP) to obtain a 6D vector, wherein the 6D vector is a feature vector with six nodes.

[0023] The 6D rotation matrix is ​​extracted from the 6D vector.

[0024] Optionally, the step of inputting the facial feature point sequence into a multilayer perceptron (MLP) to obtain a 6D vector includes: sequentially inputting the facial feature point sequence into the input layer, multiple fully connected hidden layers, and the output layer of the MLP to obtain a 6D vector.

[0025] Optionally, the step of converting the 6D rotation matrix into a 3×3 rotation matrix according to the orthogonal mapping includes: performing orthogonalization processing on the first and second columns of the 6D rotation matrix using Gram-Schmidt orthogonalization to obtain an orthogonal matrix that satisfies the constraints.

[0026] Optionally, after obtaining the orthogonal matrix that satisfies the constraints, the method further includes: mapping the orthogonal matrix that satisfies the constraints to a 3×3 rotation matrix.

[0027] Optionally, the step of converting the 3×3 rotation matrix into Euler angles includes: converting the 3×3 rotation matrix into Euler angles; and rotating the Euler angles along the X, Y, and Z axes to obtain the 3×3 rotation matrix.

[0028] Optionally, the step of obtaining the two-dimensional face image to be identified includes: obtaining the two-dimensional face image to be identified based on a continuous pose dataset, a discrete pose dataset, and a yaw angle label dataset, wherein the continuous pose dataset is the BIWI dataset, the discrete pose dataset is the PRIMA dataset, and the yaw angle label dataset is the Drive Face dataset.

[0029] This application also provides a head pose estimation system, comprising: an acquisition unit, a calibration unit, and a training unit. The acquisition unit is used to acquire a two-dimensional face image to be identified. The calibration unit is used to perform face calibration on the two-dimensional face image to obtain a calibrated two-dimensional face image. The training unit is used to acquire facial feature points on the calibrated two-dimensional face image, input the facial feature points into a regression network to obtain a 6D rotation matrix, convert the 6D rotation matrix into a 3×3 rotation matrix according to orthogonal mapping, and convert the 3×3 rotation matrix into Euler angles.

[0030] This application provides a head pose estimation method and system. The method includes: acquiring a two-dimensional face image to be identified; performing face calibration on the two-dimensional face image to obtain a calibrated two-dimensional face image; acquiring facial feature points on the calibrated two-dimensional face image; inputting the facial feature points into a regression network to obtain a 6D rotation matrix, wherein the regression network is a regression model based on the 6D rotation representation; converting the 6D rotation matrix into a 3×3 rotation matrix according to an orthogonal mapping; and converting the 3×3 rotation matrix into Euler angles, wherein the Euler angles are used to represent the rotation angles of the head pose. This addresses the problem that model-based pose estimation methods and appearance-based pose estimation methods cannot simultaneously accurately represent both recognition accuracy and robustness. Attached Figure Description

[0031] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of a head pose estimation method.

[0033] Figure 2 A diagram illustrating face calibration.

[0034] Figure 3 A schematic diagram for extracting facial feature points;

[0035] Figure 4 This is a schematic diagram of the formation of a 6D rotation matrix;

[0036] Figure 5 This is a diagram comparing the results with and without face calibration.

[0037] Figure 6 A schematic diagram of the facial feature point extraction process;

[0038] Figure 7 A schematic diagram of a set of facial feature points;

[0039] Figure 8 This is a schematic diagram illustrating the training of a 6D rotation matrix.

[0040] Figure 9 This is a schematic diagram of the BIWI dataset, PRIMA dataset, and Drive Face dataset. Detailed Implementation

[0041] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0042] Head pose estimation can be applied to scenarios such as driver fatigue detection, smart classrooms, and human-computer interaction. Furthermore, head pose estimation is a fundamental task for many face-related tasks, such as face recognition and gaze estimation. Methods for head pose estimation can be divided into two categories: model-based pose estimation methods and appearance-based pose estimation methods.

[0043] Model-based pose estimation methods first detect facial feature points in 2D face images, and then establish a correspondence between the 2D face and the 3D head using these feature points. This allows the face in the 2D image to be restored to a 3D head pose representation. Although this method can obtain relatively accurate results, it depends on the accuracy of facial feature point detection. Factors such as occlusion and large-scale rotation can lead to lower accuracy in facial feature point detection, which in turn affects the accuracy of head pose estimation.

[0044] Appearance-based methods directly extract texture features from 2D face images and associate these texture features with 3D head pose. Compared to model-based methods, appearance-based methods are more robust to factors such as occlusion or large rotation, but have lower accuracy.

[0045] To address the aforementioned problems, some embodiments of this application provide a head pose estimation method, see [link to relevant documentation]. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 1 This is a schematic diagram of a head pose estimation method. Figure 2 A diagram illustrating face calibration. Figure 3 A schematic diagram for extracting facial feature points; Figure 4 This is a schematic diagram of the formation of a 6D rotation matrix; the method includes:

[0046] S101: Obtain the two-dimensional face image to be identified;

[0047] S102: Perform face calibration on the two-dimensional face image to obtain the calibrated two-dimensional face image;

[0048] S103: Obtain facial feature points on the calibrated 2D face image;

[0049] S104: Input the facial feature points into the regression network to obtain a 6D rotation matrix. The regression network is a regression model based on the 6D rotation representation.

[0050] S105: Convert the 6D rotation matrix into a 3×3 rotation matrix according to the orthogonal mapping;

[0051] S106: Convert the 3×3 rotation matrix into Euler angles, which are used to represent the rotation angles of the head pose.

[0052] In head pose estimation, choosing a suitable rotational representation is crucial. In some embodiments, rotational representations of human head pose include Euler angles, quaternions, and rotation matrix representations. Euler angles, compared to the other two, more directly represent human head pose, requiring only three values ​​(corresponding to rotation angles along the x, y, and z axes). However, Euler angles suffer from gimbal lock, leading to the loss of degrees of freedom. In this case, visually identical head poses may have multiple rotation parameters, thus reducing the probability of learning an accurate pose through a neural network. Quaternions avoid gimbal lock compared to Euler angles, but when learning the entire range of head poses, they still exhibit ambiguity due to their opposite symmetry, which degrades the performance of head pose estimation. A rotation matrix is ​​a continuous rotational representation, with each rotation having different rotation parameters. In three-dimensional space, the rotation matrix R is 3×3 in size, includes 9 parameters, and satisfies the orthogonality property (RR). T =E), where E is the identity matrix. R T Let R be the transpose of R. The 3D rotation group is denoted as SO(3), which is the set of all rotations of an object around the origin of Euclidean space. The rotation matrix is ​​a 3-dimensional special orthogonal matrix in the Lie group SO(3), and satisfies the following properties:

[0053]

[0054] Here, det[R] is the determinant of a matrix.

[0055] To represent head pose using a rotation matrix, some implementations employ a regression approach, obtaining nine parameters from the rotation matrix and then performing an orthogonalization transformation on these nine parameters. Rotation matrix representation avoids gimbal lock issues while also eliminating ambiguity caused by opposite symmetry. However, since predicting a complete rotation matrix requires regressing nine parameters, the accumulated loss of each parameter during training leads to a significant increase in the overall loss, making it difficult for the neural network to learn accurate pose.

[0056] Therefore, in order to obtain smaller errors and enable the neural network to learn accurate poses, facial feature points are input into the regression network to obtain a 6D rotation matrix. The nine parameters obtained from the regression are reduced to six parameters, and the 3×3 form of the rotation matrix is ​​transformed into a 3×2 form by discarding the last column vector to obtain smaller errors.

[0057] The head pose estimation method provided in the above embodiments can regard the facial feature points in the two-dimensional face image as the input sequence of the neural network, obtain the user's head pose estimation result through neural network regression, solve the fuzzy rotation label problem by using rotation matrix representation, and use the 6D rotation matrix representation as the intermediate state of the rotation matrix to achieve effective direct regression.

[0058] First, a two-dimensional face image to be identified is acquired. In some embodiments, this includes acquiring the two-dimensional face image to be identified using a continuous pose dataset, a discrete pose dataset, and a yaw angle label dataset. The continuous pose dataset can be the BIWI dataset, the discrete pose dataset can be the PRIMA dataset, and the yaw angle label dataset can be the Drive Face dataset. It is understood that other datasets can also be used, such as face detection datasets and benchmark FDDB (Face Detection Data Set and Benchmark) datasets, benchmark face detection datasets with annotated facial landmarks in the wild (AFLW) dataset, etc.

[0059] See Figure 9 , Figure 9 This diagram illustrates the BIWI, PRIMA, and Drive Face datasets. Specifically, the BIWI (Biwi Kinect) head pose dataset contains images acquired using the Kinect depth sensor. Each image includes not only RGB color information but also depth information. For example, the BIWI dataset was created by 20 participants using the Kinect depth sensor and comprises approximately 15,000 frames. Each image provides labels representing head pose in three categories: pitch, yaw, and roll. The head pose range is continuously represented, with pitch covering ±60°, yaw ±75°, and roll ±60°.

[0060] The PRIMA dataset is a dataset within the 2D color image dataset category. For example, the PRIMA dataset consists of facial images of 15 participants, comprising a total of 2790 images. Each image is labeled with two categories of head pose: pitch and yaw angles. Unlike the BIWI dataset, this dataset does not provide roll angle labels, and the head pose range is discretely represented. Yaw angles include nine discrete poses from -90° to 90° (Yaw∈{-90°,-60°,...,60°,90°}), and pitch angles include thirteen discrete poses from -90° to 90° (Pitch∈{-90°,-75°,...,75°,90°}).

[0061] The Drive Face dataset is a sequence of images captured in real-world scenarios. For example, it consists of images of four drivers (two men and two women) driving in real-world conditions. Facial features of these drivers include wearing glasses and having beards. This dataset contains 606 images, each labeled with a yaw angle representing head pose. Unlike the datasets mentioned above, this dataset only provides yaw angle labels, and the head pose range is discretely represented, with yaw angles ranging from -45° to 45° across seven discrete poses (Yaw∈{-45°,-30°,...,30°,45°}).

[0062] Using three different types of datasets can improve the diversity of the acquired 2D face images, indirectly making the obtained head pose estimation more accurate.

[0063] See Figure 5 , Figure 5 This diagram illustrates the comparison between results obtained with and without face calibration. Since the size and position of faces vary across different datasets, directly using 2D face images from the datasets would lead to discrepancies in the spatial distribution of extracted facial feature points, resulting in inaccurate results and reduced model generalization. Besides differences in the face images acquired from the datasets, factors such as the distance and position of different users relative to the camera, as well as camera performance, all affect the face images during image capture. Therefore, to minimize these differences, this embodiment first performs face calibration on the 2D face images and / or captured face images obtained from the three datasets mentioned above. In some embodiments, this includes: locating the head position of the 2D face image using the MTCNN face detection algorithm; cropping and / or expanding the 2D face image based on the head position to obtain the face region; and adjusting each face region to the same image size to obtain the calibrated face image.

[0064] It is understood that this embodiment uses the MTCNN face detection algorithm to locate the head position. Other algorithms can also be used for face detection and recognition to locate the head position, such as algorithms in the Dlib machine learning library (3D point cloud, SURF feature, Bayesian classification, SVM, deep learning, etc.), or the OpenCV cross-platform computer vision library, which consists of a series of C functions and a small number of C++ classes, and also provides a Python interface, implementing many general algorithms in image processing and computer vision.

[0065] After locating the head, the 2D face image is cropped and / or expanded. For example, each 2D face image is set to 400×400 pixels, which means expanding 2D face images smaller than 400×400 pixels or cropping 2D face images larger than 400×400 pixels. The 300×300 2D face image is filled with background, and its length and width are both expanded by 100 pixels to obtain a 400×400 2D face image. The 500×500 2D face image area is shrunk to ensure that the face is within the image, and then cropped. The length and width of the 500×500 2D face image are both cropped by 100 pixels to obtain a 400×400 2D face image.

[0066] The calibrated face image can be represented as:

[0067]

[0068]

[0069]

[0070]

[0071] w = x2 - x1

[0072] h = y2 - y1

[0073] Among them, X w1 X w2 y w1 y w2 The four vertices represent the face calibration region, x1, x2, y1, and y2 represent the four vertices for head localization, and α represents the expansion coefficient. In this embodiment, α = 0.4. w I represents the width of the input image. h The height of the input image is represented by w, and the length and width of the head region are represented by h.

[0074] The goal of face calibration is to ensure that faces in a 2D face image maintain a similar distribution in 2D space, placing faces in the center of the image and ensuring that facial regions are of similar size. To compare results, experiments were conducted using different datasets, DriveFace and PRIMA. See [link to relevant documentation]. Figure 9 The data in the second and third rows of the table were used for training on the PRIMA dataset (training set) and for testing on the Drive Face dataset (test set). The purpose of the experiment was to verify the generalization and robustness of the method proposed in this embodiment. As shown in Table 1, the method proposed in this embodiment can achieve good accuracy even when trained and tested on different datasets. Due to the significant differences between different datasets, there are obvious differences in face distribution and region size between the training and test sets.

[0075] See Figure 5 As can be seen, (a) shows an example of extracting facial feature points without face calibration, while (b) shows an example of extracting facial feature points with face calibration. The distribution areas of the facial features extracted from the three images in (a) are different, but the distribution areas of the facial features extracted from the three images in (b) are less different. This demonstrates that extracting facial feature points after face calibration can reduce the differences in distribution and size between facial images obtained from different datasets, resulting in stronger generalization ability. However, the method of directly extracting facial feature points without face calibration leads to differences in the distribution areas and sizes of the facial feature points, resulting in weak generalization ability.

[0076] Method Yaw Diaz-Chito et al.[7] 16.1±14.8 Drouard et al.[5] 19.5±8.2 Malek et al.[6] 7.49±1.13 Our* 14.31±3.99 Our** 6.21±0.37

[0077] Table 1

[0078] You can also use other datasets for experiments, such as the BIWI dataset or the PRIMA dataset.

[0079] The algorithm was evaluated using five-fold cross-validation experiments on the BIWI dataset. (See [link to relevant documentation]). Figure 9The data in the first row is shown. The dataset is randomly divided into 5 equal parts, with the first 4 parts used for training and the last part used for validation. The results are shown in Table 2, which shows the performance of different methods on the BIWI dataset. By introducing a column of indicators AVG, which represents the average value of yaw, pitch, and roll angles, the overall performance of the algorithm can be reflected. Compared with other algorithms, the method proposed in this embodiment not only has a significant improvement in the prediction effect of a single angle, but also has a better overall performance. This significant improvement can be explained in several ways. First, the rotation matrix representation contains rotation information of pitch, yaw, and roll angles. Therefore, the training process of 6DFLRNet based on rotation matrix representation requires a dataset containing these three types of real labels for training. The BIWI dataset already contains the real labels of these three types of angles, providing complete information for model training.

[0080] Method Yaw Pitch Roll AVG Liu et al.[1] 5.81 7.94 / 6.88 Yujia et al.[2] 4.76±4.33 5.48±3.23 4.29±3.30 4.84±3.61 Ruiz et al.[3] 3.29 3.39 3 3.22 Xia et al.[4] 2.39 4.92 3.09 3.47 Drouard et al.[5] 4.24 5.43 4.13 4.60 Our 2.66 3.25 2.46 2.79

[0081] Table 2

[0082] Experiments using the PRIMA dataset were conducted to evaluate the algorithm using double cross-validation. This involved randomly dividing the dataset into two equal parts, with 50% used for training and the other 50% for validation. (See [link to relevant documentation]). Figure 9 The data in the second row of the table. As shown in Table 3, the method provided in this embodiment performs best in estimating yaw angle compared to other methods, and performs slightly worse in estimating pitch angle [6]. Since the PRIMA dataset only contains two types of labels, pitch angle and yaw angle, the 6DFLRNet model lacks roll angle information during training. Therefore, during training, the roll angle rotation can only be assumed to be 0° to compensate for the loss of information in this dimension.

[0083] [1] By matching 2D and 3D facial feature points one by one, and then reconstructing the 2D face image into a 3D human head in 3D space through 3D reconstruction, head pose estimation is performed. Although this method can obtain very accurate results, it is highly dependent on the prediction results of the facial feature point positions. [6] Both the method proposed in this embodiment are based on the same facial feature extractor CNN (Recurrent Neural Network). The facial feature points obtained at some angles are not accurate enough, for example: images with yaw angle: -60°, pitch angle: +30°, images with yaw angle: +60°, pitch angle: +60°, images with yaw angle: +90°, pitch angle: +15°. [6] Head pose estimation is regarded as a classification problem. It divides a large range of angles into multiple categories. For example, angles in the range of [-90°, 90°] are divided into 12 categories with an interval of 15°. The PRIMA dataset and the Drive Face dataset are discrete datasets. They divide head poses into multiple categories with intervals of 30° and 15° respectively to collect data. [6] The partitioning interval is divisible by the partitioning interval of the two datasets (30° is twice 15°, and 15° is once 15°), which makes it easier to obtain a good metric on such discrete datasets. However, this has limitations because head movement is a continuous process. Treating head pose estimation as a regression problem to obtain continuous head pose estimation results, although the regression method is not very effective on these two discrete datasets, the results obtained are more granular and more in line with practical applications.

[0084] Method Yaw Pitch AVG Diaz-Chito et al.[7] 8.1±7.8 9.6±10.0 8.85±8.9 Drouard et al.[5] 13±9.0 22.6±12 17.8±10.5 Malek et al.[6] 7.36±0.40 8.32±0.31 7.84±0.36 Our 7.17±0.23 8.93±0.28 8.05±0.26

[0085] Table 3

[0086] To obtain facial feature points on the calibrated 2D face image, in some embodiments, a CNN-based facial feature extractor is used for extraction; see [link to relevant documentation]. Figure 6 , Figure 6 The diagram illustrates the process of facial feature point extraction, which includes: inputting a calibrated two-dimensional face image into a convolutional network to obtain a multidimensional matrix output by the convolutional network; converting the multidimensional matrix into a vector matrix; compressing the vector matrix and inputting it into a fully connected layer; and the fully connected layer outputting 136-dimensional vector features, wherein the vector features are facial feature points, and the facial feature points are two-dimensional coordinate points.

[0087] It is understood that the facial feature extractor used in this embodiment is CNN. In addition, facial feature point extraction can be any of the algorithms of other neural networks, such as algorithms in the Dlib machine learning library (3D point cloud, SURF feature, Bayesian classification, SVM, deep learning, etc.), or in the OpenCV cross-platform computer vision library, which consists of a series of C functions and a small number of C++ classes, and provides a Python interface to implement many general algorithms in image processing and computer vision.

[0088] See Figure 6 For example, a 2D face image input to a convolutional network is 128 pixels long and 128 pixels wide with 3 channels. After passing through a 3×3 convolutional layer, a neuron activation function, and batch normalization, the output is an image 126 pixels long and 126 pixels wide with 64 channels (a 64-dimensional 126×126 convolutional kernel). This 64-dimensional 126×126 convolutional kernel then passes through a max pooling layer, a 3×3 convolutional layer, and batch normalization, outputting an image 61 pixels long and 61 pixels wide with 64 channels (a 64-dimensional 61×61 convolutional kernel). This process iteratively passes through a 3×3 convolutional layer, a neuron activation function, batch normalization, and then through a max pooling layer... The process involves pooling layers, 3×3 convolutional layers, and batch normalization until an image with a length of 5 pixels, a width of 5 pixels, and 256 channels is output, resulting in a 256-dimensional 5×5 convolutional kernel, which is the aforementioned multidimensional matrix. Then, the 256-dimensional 5×5 convolutional kernel is flattened, which involves expanding the multidimensional matrix and representing it with vectors. This flattening operation yields a vector matrix, which is then compressed and input into the fully connected layer of the convolutional network. After passing through the fully connected layer, a 1×136-dimensional vector feature is output. This vector feature represents the facial feature points. In this embodiment, 68 facial feature points were detected, i.e., 68 two-dimensional coordinate points. (See [link to documentation]). Figure 7 , Figure 7 In the image, (a) shows a face image, and (b) shows 68 facial feature points detected in the face image corresponding to (a).

[0089] In the above embodiments, a nine-parameter rotation matrix can be directly predicted. However, since the cumulative loss of each parameter during training leads to an increase in the overall loss, this embodiment first obtains a 6D rotation matrix through a regression network, and then converts the 6D rotation matrix into a nine-parameter rotation matrix. In some embodiments, this includes: inputting a facial feature point sequence into a multilayer perceptron (MLP) to obtain a 6D vector, wherein the 6D vector is a feature vector with six nodes; and extracting the 6D rotation matrix from the 6D vector. Since this regression network obtains the 6D rotation matrix from the facial feature point sequence (Facial LandmarkFL) through regression, this regression network is called 6DFLRNet.

[0090] In some embodiments, the step of inputting a sequence of facial feature points into a Multilayer Perceptron (MLP) to obtain a 6D vector includes: sequentially inputting the sequence of facial feature points into the input layer, multiple fully connected hidden layers, and the output layer of the MLP to obtain a 6D vector. The Multilayer Perceptron (MLP) is a feedforward artificial neural network model that maps multiple input datasets to a single output dataset. An MLP can be viewed as a directed graph composed of multiple node layers, each fully connected to the next layer. The MLP adds one or more fully connected hidden layers between the output and input layers and transforms the output of the hidden layers through activation functions. In this embodiment, three fully connected layers are used to output the 6D vector. It is understood that the regression algorithm used in this embodiment is the MLP; however, the facial feature point regression algorithm can be any algorithm from other neural networks, such as attention mechanisms or recurrent neural networks (RNNs).

[0091] Specifically, by halving the feature vectors of the six nodes, we can obtain a 2×3 matrix, i.e., a 6D rotation matrix:

[0092]

[0093] Here, a1, a2, a3, b1, b2, b3 are six nodes, and α1 and α2 are the first and second column matrix vector groups.

[0094] The 6D rotation representation matrix is ​​then mapped to an orthogonal matrix:

[0095]

[0096] Among them, β1, β2, and β3 are orthogonal vector groups.

[0097] The last column, or the third column of the matrix vector group, can be calculated from the first and second column matrix vector groups. In some embodiments, Gram-Schmidt orthogonalization is used to orthogonalize the first and second columns of the 6D rotation matrix to obtain an orthogonal matrix that satisfies the constraints.

[0098]

[0099] γ2=α2-(β1·α2)β1

[0100] β3 = β1 × β2

[0101] It is understood that the orthogonal matrix satisfying the constraints contains six parameters, and in some embodiments, the orthogonal matrix satisfying the constraints is mapped to a 3×3 rotation matrix.

[0102] Finally, in some embodiments, the 3×3 rotation matrix is ​​converted into Euler angles. There is a definite correspondence between the 3×3 rotation matrix conversion and Euler angles, which can be performed using the following formula:

[0103]

[0104]

[0105]

[0106] Among them, R x (α) is the rotation matrix of the object around the X-axis, R y (β) is the rotation matrix of the object around the Y-axis, R z (γ) is the rotation matrix of the object around the Z-axis, α is the Euler angle of the object around the X-axis, β is the Euler angle of the object around the Y-axis, and γ is the Euler angle of the object around the Z-axis, representing the pitch angle, yaw angle, and roll angle, respectively.

[0107] Rotating the Euler angles along the X, Y, and Z axes also yields a 3×3 rotation matrix, with the formula:

[0108]

[0109] The 6DFLRNet model can also be trained using the total loss from head pose estimation. In this embodiment, the L2 norm is used to calculate the loss during training, and the formula is as follows:

[0110]

[0111] Where n represents the number of training samples, i represents the sample number, and y i For the true label of the sample, This is the result of the model estimating the head pose of the samples.

[0112] See Figure 8 , Figure 8 This diagram illustrates the training of a 6D rotation matrix. Training is performed by minimizing the error between the predicted and actual results of the 6DFLRNet model. During training, the Mean Absolute Error (MAE) is used as the performance evaluation metric, and its formula is as follows:

[0113]

[0114] Where n represents the number of test samples, i represents the sample number, and y i For the true label of the sample, This is the result of the model estimating the head pose of the samples.

[0115] Corresponding to the aforementioned embodiment of a head pose estimation method, this application also provides an embodiment of a head pose estimation system. The system includes: an acquisition unit, a calibration unit, and a training unit. The acquisition unit acquires a two-dimensional face image to be identified; the calibration unit performs face calibration on the two-dimensional face image to obtain a calibrated two-dimensional face image; the training unit acquires facial feature points on the calibrated two-dimensional face image, inputs the facial feature points into a regression network to obtain a 6D rotation matrix, converts the 6D rotation matrix into a 3×3 rotation matrix according to orthogonal mapping, and converts the 3×3 rotation matrix into Euler angles.

[0116] To ensure that faces in a 2D face image maintain a similar distribution in 2D space, with faces centered in the image and face regions of similar size, in some embodiments, the calibration unit further includes a convolutional network module. This module takes the calibrated 2D face image as input to the convolutional network to obtain a multidimensional matrix output by the network. The multidimensional matrix is ​​then converted into a vector matrix, compressed, and input to a fully connected layer. The fully connected layer outputs 136-dimensional vector features.

[0117] To achieve smaller errors and enable the neural network to learn accurate poses, in some embodiments, the training unit is also used to input the facial feature point sequence into a multilayer perceptron (MLP) to obtain a 6D vector, extract the 6D rotation matrix from the 6D vector, and then input the facial feature point sequence sequentially into the input layer, multiple fully connected hidden layers, and the output layer of the MLP to obtain a 6D vector. Gram-Schmidt orthogonalization is used to orthogonalize the first and second columns of the 6D rotation matrix to obtain an orthogonal matrix that satisfies the constraints. The orthogonal matrix that satisfies the constraints is then mapped to a 3×3 rotation matrix.

[0118] This application provides a head pose estimation method and system. The method includes: acquiring a two-dimensional face image to be identified; performing face calibration on the two-dimensional face image to obtain a calibrated two-dimensional face image; acquiring facial feature points on the calibrated two-dimensional face image; inputting the facial feature points into a regression network to obtain a 6D rotation matrix, wherein the regression network is a regression model based on the 6D rotation representation; converting the 6D rotation matrix into a 3×3 rotation matrix according to an orthogonal mapping; and converting the 3×3 rotation matrix into Euler angles, where Euler angles are used to represent the rotation angles of the head pose. This addresses the problem that model-based pose estimation methods and appearance-based pose estimation methods cannot simultaneously accurately represent both recognition accuracy and robustness.

[0119] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A head pose estimation method, characterized in that, include: Obtain the 2D face image to be identified; The two-dimensional face image is calibrated to obtain a calibrated two-dimensional face image; The calibrated two-dimensional face image is passed into a convolutional network to obtain a multidimensional matrix output by the convolutional network. Transform the multidimensional matrix into a vector matrix; The vector matrix is ​​compressed and then input into the fully connected layer; The fully connected layer outputs a 136-dimensional vector feature, which is a facial feature point, and the facial feature point is a two-dimensional coordinate point; The facial feature point sequence is input into a multilayer perceptron (MLP) to obtain a 6D vector, which is a feature vector with six nodes. A 6D rotation matrix is ​​extracted from the 6D vector. The facial feature point sequence is then sequentially input into the input layer, multiple fully connected hidden layers, and the output layer of a multilayer perceptron (MLP) to obtain the 6D vector. The 6D rotation matrix is ​​converted into a 3×3 rotation matrix according to the orthogonal mapping. The 3×3 rotation matrix is ​​converted into Euler angles, which are used to represent the rotation angles of the head posture.

2. The head pose estimation method according to claim 1, characterized in that, The step of performing face calibration on the two-dimensional face image to obtain a calibrated face image includes: The head position of the two-dimensional face image is located using the MTCNN face detection algorithm; The two-dimensional face image is cropped and / or expanded according to the head position to obtain the face region; Each of the facial regions is adjusted to the same image size to obtain a calibrated facial image.

3. The head pose estimation method according to claim 1, characterized in that, The step of converting the 6D rotation matrix into a 3×3 rotation matrix according to the orthogonal mapping includes: The first and second columns of the 6D rotation matrix are orthogonalized using Gram-Schmidt orthogonalization to obtain an orthogonal matrix that satisfies the constraints.

4. The head pose estimation method according to claim 3, characterized in that, After obtaining the orthogonal matrix that satisfies the constraints, the process also includes: The orthogonal matrix that satisfies the constraints is mapped to a 3×3 rotation matrix.

5. The head pose estimation method according to claim 1, characterized in that, The step of converting the 3×3 rotation matrix into Euler angles includes: Convert the 3×3 rotation matrix to Euler angles; The Euler angles are rotated along the X, Y, and Z axes to obtain the 3×3 rotation matrix.

6. The head pose estimation method according to claim 1, characterized in that, The steps for obtaining a 2D face image to be identified include: Two-dimensional face images to be identified are obtained from a continuous pose dataset, a discrete pose dataset, and a yaw angle label dataset. The continuous pose dataset is the BIWI dataset, the discrete pose dataset is the PRIMA dataset, and the yaw angle label dataset is the Drive Face dataset.

7. A head pose estimation system, characterized in that, include: Acquisition Unit: The acquisition unit is used to acquire a two-dimensional face image to be identified; Calibration unit: The calibration unit is used to perform face calibration on the two-dimensional face image to obtain a calibrated two-dimensional face image; Training unit: The training unit is used to input the calibrated two-dimensional face image into a convolutional network to obtain a multidimensional matrix output by the convolutional network; The multidimensional matrix is ​​converted into a vector matrix; the vector matrix is ​​compressed and input into a fully connected layer; the fully connected layer outputs a 136-dimensional vector feature, the vector feature being facial feature points, which are two-dimensional coordinate points; the sequence of facial feature points is input into a multilayer perceptron (MLP) to obtain a 6D vector, the 6D vector being a feature vector with six nodes; a 6D rotation matrix is ​​extracted from the 6D vector; the sequence of facial feature points is sequentially input into the input layer, multiple fully connected hidden layers, and the output layer of the MLP to obtain a 6D vector; the 6D rotation matrix is ​​converted into a 3×3 rotation matrix according to orthogonal mapping; the 3×3 rotation matrix is ​​then converted into Euler angles.

Images