A 3D head pose estimation algorithm based on facial feature information

Abstract

The present invention discloses a three-dimensional head posture estimation algorithm based on facial feature information, comprising the following steps: obtaining camera calibration parameters and three-dimensional coordinates of multiple model points selected in the facial area of ​​a head model; collecting a facial picture of a human face whose posture is to be estimated by a camera, extracting feature points corresponding to the model points, and obtaining two-dimensional pixel coordinates; calculating head posture estimation parameters according to the camera calibration parameters, the three-dimensional coordinates, and the two-dimensional pixel coordinates; reprojection verification of the head posture estimation parameters, if the reprojection error is greater than a set value, then correcting the head posture estimation parameters by an extended Kalman filter, so that the reprojection error of the corrected head posture parameters is less than or equal to the set value. In this way, by means of the extended Kalman filter correction, it is ensured that the reprojection error of the head posture estimation parameters is small, and it is avoided that the algorithm based on the two-dimensional image will misjudge the head posture estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition and posture analysis, and more specifically to a three-dimensional head posture estimation algorithm based on facial feature information. Background Art

[0002] Head pose estimation (HPE), as the intersection of face recognition and posture analysis, has a wide range of applications in real life. In road traffic accidents, accidents caused by driver inattention account for a large proportion of the total number of accidents. The head pose estimation algorithm can be used to judge the driver's attention status to reduce the probability of road safety accidents.

[0003] The estimation of head posture is completed through the position change of the face in three-dimensional space. Its estimation algorithm usually includes model-based algorithm and image appearance-based algorithm. Algorithms based on image appearance can be roughly divided into three categories: algorithms based on three-dimensional color images, algorithms based on image depth information, and algorithms based on two-dimensional images. Three-dimensional images contain more information, so the amount of data to be processed is large, and the equipment requirements are high; algorithms based on image depth information require the use of depth sensors to obtain image depth information, which increases the cost of equipment. Algorithms based on two-dimensional color images require low equipment costs, but when the head turns left or right or the upward angle is too large, facial feature points are easily lost, and the estimation of three-dimensional head posture is prone to misjudgment.

[0004] Therefore, how to solve the problem in the prior art that the equipment cost required for algorithms based on three-dimensional images and depth information is high, while algorithms based on two-dimensional images may cause misjudgment of head posture estimation, has become an important technical problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional head posture estimation algorithm based on facial feature information, which solves the problem that the equipment cost required for the algorithm based on three-dimensional images and depth information in the prior art is high, while the algorithm based on two-dimensional images will produce misjudgment of head posture estimation. The preferred technical solutions among the many technical solutions provided by the present invention can produce many technical effects as described below.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention provides a three-dimensional head posture estimation algorithm based on facial feature information, comprising the following steps:

[0008] Obtaining camera calibration parameters and three-dimensional coordinates of multiple model points selected in the facial area of ​​the head model;

[0009] Collect a facial image of a person whose posture is to be estimated by a camera, extract feature points corresponding to the model points in the facial image, and obtain the two-dimensional pixel coordinates of each feature point in a pixel coordinate system;

[0010] Calculate the head posture estimation parameters based on the camera calibration parameters, three-dimensional coordinates, and two-dimensional pixel coordinates;

[0011] The head posture estimation parameters are reprojected and checked to determine whether the reprojection error is greater than the set value. If the reprojection error is greater than the set value, the head posture estimation parameters are corrected through the extended Kalman filter so that the corrected head posture parameter reprojection error is less than or equal to the set value.

[0012] Preferably, the head posture estimation parameters include Euler angles and a translation vector, the Euler angles include a roll angle α, a pitch angle β and a yaw angle γ of the head, and the translation vector of the head is (tx, ty, tz).

[0013] Preferably, the system state equation corrected by extended Kalman filtering is:

[0014] θ k =f(θ k|k-1 )+S K

[0015] Z k =h(θ k )+V K (1)

[0016] Correction steps include:

[0017] Substitute the head posture prediction parameters into equation (2), and then substitute the result into equation (3). Repeat equation (2) and equation (3) to obtain the corrected head posture parameters, where:

[0018] x k =f(x _k-1 )

[0019]

[0020]

[0021] x _k =x k +K(z k -h(x k ))

[0022] p_k=(I-KH k ) k (3)

[0023] State variable x k=[α β γ tx ty tz]T,S k represents the state noise, V k represents the measurement noise. The matrix in equation (2) and equation (3) obtained based on equation (1) is expressed as:

[0024]

[0025]

[0026] Through the above correction steps, the corrected head posture parameters can be obtained based on the extended Kalman filter.

[0027] Preferably, a head coordinate system is constructed, and the face of the head model is scanned to obtain the three-dimensional coordinates of each model point in the head coordinate system.

[0028] Preferably, the origin of the head coordinate system is the nose tip of the facial area, the positive direction of the Y axis is from the chin point to the center of the eyebrows, the positive direction of the X axis is from the right eye to the left eye, and the positive direction of the Z axis is determined according to the right-hand rule.

[0029] Preferably, the multiple model points selected in the facial area of ​​the head model are P0, P1, ..., P n , the two-dimensional pixel coordinates of the feature points on the face image corresponding to each model point are Q0, Q1, ... Q n , according to P x and Q x , and camera calibration parameters, the PNP algorithm is used to solve the head posture estimation parameters, where x = 0, 1, ... n.

[0030] Preferably, the selected model points include the nose tip point, the chin point, the left eye outer canthus point, the right eye outer canthus point, the left corner of the mouth point and the right corner of the mouth point.

[0031] Preferably, feature points of the face image are extracted based on the DLIB library, and the two-dimensional pixel coordinates of the feature points are obtained.

[0032] Preferably, the camera calibration parameters are obtained by using a Matlab camera calibration toolbox according to a checkerboard calibration method, and the camera calibration parameters include camera intrinsic parameters and distortion coefficients.

[0033] In the technical solution provided by the present invention, the three-dimensional head posture estimation algorithm based on facial feature information first obtains the three-dimensional coordinates and camera calibration parameters of multiple model points selected in the facial area of ​​the head model, collects the face picture of the face to be estimated through the camera, obtains the two-dimensional pixel coordinates of the feature points corresponding to the model points in the face picture, calculates the posture change of the head posture restored from the face image relative to the reference posture (i.e., the front head posture) of the head model through the two-dimensional pixel coordinates, the three-dimensional coordinates and the camera calibration parameters, and obtains the head posture estimation parameters; the head posture estimation parameters are reprojected and verified, and when the reprojection error is greater than the set value, the head posture estimation parameters need to be corrected by the extended Kalman filter until the reprojection error of the corrected result is less than or equal to the set value, and the corrected result is output to obtain relatively accurate head posture parameters. In this way, the algorithm estimates the head posture based on the two-dimensional image information, and the required equipment cost is low. In the case of large reprojection error, the extended Kalman filter is used for correction, which solves the problem that the algorithm based on the two-dimensional image will misjudge the head posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 is a flow chart of a three-dimensional head posture estimation algorithm based on facial feature information in an embodiment of the present invention;

[0036] Figure 2 It is a flow chart of the extended Kalman filter correction in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0038] The purpose of this specific implementation is to provide a three-dimensional head posture estimation algorithm based on facial feature information, which solves the problem of misjudgment of head posture estimation while ensuring that the algorithm based on two-dimensional image information has a lower cost.

[0039] The following is an explanation of the embodiments with reference to the accompanying drawings. In addition, the embodiments shown below do not limit the invention described in the claims. In addition, the entire contents of the configurations shown in the following embodiments are not limited to the solutions required as the invention described in the claims.

[0040] See also Figure 1 and Figure 2 The three-dimensional head posture estimation algorithm based on facial feature information provided by the present invention comprises the following steps:

[0041] S01: Obtain camera calibration parameters and three-dimensional coordinates of multiple model points selected in the facial area of ​​the head model.

[0042] Among them, the camera calibration parameters are obtained through the Matlab camera calibration toolbox according to the checkerboard calibration method. The camera calibration parameters include camera intrinsic parameters and distortion coefficients. After completing the camera calibration, the camera intrinsic parameters are shown in Table 1, and the distortion coefficients are shown in Table 2.

[0043] Table 1 Camera internal parameters

[0044]

[0045] Table 2 Distortion coefficient

[0046]

[0047] Place the head model facing the scanner, select multiple model points in the facial area of ​​the head model, and determine the three-dimensional coordinates corresponding to each selected model point. The specific steps are as follows:

[0048] A head coordinate system is constructed, the origin of the head coordinate system is the nose tip point of the facial area of ​​the head model, the positive direction of the Y axis is from the chin point to the brow center point, the positive direction of the X axis is from the right eye to the left eye, and the positive direction of the Z axis is determined according to the right-hand rule, so that the head coordinate system is established; multiple model points are selected in the facial area of ​​the head model. In a specific embodiment, a total of six model points are selected in the facial area of ​​the head model, namely, the nose tip point Q0, the chin point Q1, the left eye outer canthus point Q2, the right eye outer canthus point Q3, the left corner of the mouth point Q4 and the right corner of the mouth point Q5. The point cloud data of the facial area of ​​the head model is obtained by scanning with an F6SR scanner, and the three-dimensional coordinates are collected by active triangulation through echo software. Stereo disparity is used to obtain the third dimension information, and the three-dimensional coordinates of each point are determined by the distance between the five points Q1, Q2, Q3, Q4 and Q5 and Q0 on the three axes. The specific data are shown in Table 3.

[0049] Table 3 Selected model points and their corresponding three-dimensional coordinates

[0050] Model Points 3D Coordinates <![CDATA[Q0]]> (0.0，0.0，0.0) <![CDATA[Q1]]> (0.0，-56.9，-14.6) <![CDATA[Q2]]> (43.5，21.3，-32.5) <![CDATA[Q3]]> (-43.5，21.3，-32.5) <![CDATA[Q4]]> (17.75，-30.1，-18.5) <![CDATA[Q5]]> (-17.75，-30.1，-18.5)

[0051] S02: Using a calibrated camera to collect a facial image of a person whose posture is to be estimated, extract feature points of the facial image, and obtain the two-dimensional pixel coordinates of each feature point in a pixel coordinate system. In a specific embodiment, six feature points are extracted, and they correspond one to one with the six model points in step S01.

[0052] The DLIB library is used to extract 68 feature points of the face from the head model, and the serial numbers of the feature points corresponding to Q0, Q1, ..., Q5 are obtained, as shown in Table 4.

[0053] Table 4 Sequence number table of feature points corresponding to selected model points

[0054] Model Points The sequence number of the feature point <![CDATA[Q0]]> 31 <![CDATA[Q1]]> 9 <![CDATA[Q2]]> 37 <![CDATA[Q3]]> 46 <![CDATA[Q4]]> 49 <![CDATA[Q5]]> 55

[0055] Based on the DLIB library, 68 facial feature points are extracted from the face image, and the two-dimensional pixel coordinates P0, P1, ... P5 of the feature points numbered 31, 9, 37, 46, 49, and 55 in the pixel coordinate system are obtained. The head posture of the face image collected by the camera is different, and the two-dimensional pixel coordinates of the feature points are different.

[0056] S03: Calculate head posture estimation parameters according to the camera calibration parameters and the three-dimensional coordinates and two-dimensional pixel coordinates of multiple model points selected in the facial area of ​​the head model. That is, according to the camera calibration parameters, the three-dimensional coordinates (P0, P1, ... P n ) and the two-dimensional pixel coordinates (Q0, Q1, ... Q n ), P x , Q x The camera calibration parameters are substituted into the PNP system framework, and the PNP algorithm is used to solve the head posture estimation parameters, where x = 0, 1, ... n.

[0057] Specifically, the head posture estimation parameters include Euler angles and translation vectors. The Euler angles include three angular parameters: roll angle α, pitch angle β, and yaw angle γ, which describe the direction of movement of the head in three-dimensional space; the translation vector of the head is (tx, ty, tz), which can be understood as the position of the origin of the object coordinate system in the camera coordinate system. In a specific embodiment, since the origin of the head coordinate system is the tip of the nose of the head model, the translation vector can be defined as the position of the tip of the nose in the camera coordinate system.

[0058] In a specific embodiment, the PNP algorithm uses the least squares iteration method (Least Square Iteration), performs a series of nonlinear operations on the input facial image, gives an initial value through iteration, continuously updates the current optimization variable, and reduces the objective function until the increment at a certain moment is very small and the function can no longer be reduced. At this time, the algorithm converges and completes the process of finding the minimum value.

[0059] S04: Perform reprojection verification on the head posture estimation parameters to determine whether the reprojection error is greater than a set value; if the reprojection error is not greater than the set value, execute step S06; if the reprojection error is greater than the set value, execute step S05.

[0060] In a specific embodiment, multiple groups of head posture estimation parameters are selected to verify the reprojection error. Combined with the reprojection results, the set value of the reprojection error is set to 1. It is concluded that when tz in the head translation vector satisfies the negative direction and the absolute value of tz is in the range of [300, 1000], the reprojection error is small and can meet the requirements. When tz does not meet the above conditions, the reprojection error is large and does not meet the requirements.

[0061] In the first embodiment of the reprojection verification, the head posture estimation parameters are: α=5.00213, β=10.0019, γ=4.99998, tx=-0.000220552, ty=0.00127817, tz=-499.987, tz satisfies the negative direction and the absolute value is in the range of [300, 1000]. The reprojection verification results show that the six projection points can all correspond to the six model points selected in the facial area of ​​the head model, and the reprojection error is very small.

[0062] In the second reprojection verification embodiment, the head posture estimation parameters are: α=-7.65338, β=-10.8251, γ=-174.688, tx=-0.568548, ty=0.0725333, tz=563.427, the absolute value of tz is in the range of [300, 1000], but the direction is not negative. The reprojection verification results show that the six projection points cannot coincide with the six model points selected in the facial area of ​​the head model, and the reprojection error increases.

[0063] In the third reprojection verification embodiment, the head posture estimation parameters are: α=10.4804, β=19.1756, γ=-11.2989, tx=2.26937e+07, ty=-1.56113e+07, tz=-1.39073e+09, tz satisfies the negative direction, but the absolute value of tz is not in the range of [300, 1000]. The reprojection verification results show that the six projection points are gathered in one place and cannot coincide with the six model points selected in the facial area of ​​the head model, and the reprojection error is large.

[0064] S05: Correct the head posture estimation parameters obtained in S03 by using the extended Kalman filter, obtain the corrected head posture parameters and then execute step S04. That is, after correction, the correction result needs to be reprojected and verified until the reprojection error of the corrected result is less than the set value, the correction process ends, and the corrected result is output.

[0065] In a specific embodiment, the system state equation corrected by the extended Kalman filter is:

[0066] θ k =f(θ k|k-1 )+S K

[0067] Z k =h(θ k )+V K (1)

[0068] The specific steps for correction are:

[0069] Substitute the head posture prediction parameters into equation (2), and then substitute the result into equation (3). Repeat equation (2) and equation (3) to obtain the corrected head posture parameters, where:

[0070] x k =f(x _k-1 )

[0071]

[0072]

[0073] x _k =x k +K(z k -h(x k ))

[0074] p_k=(I-KH k ) k (3)

[0075] State variable x k=[α β γ tx ty tz] T , S k represents the state noise, V k represents the measurement noise. The matrix in equation (2) and equation (3) obtained based on equation (1) is expressed as:

[0076]

[0077]

[0078] Through the above correction steps, the corrected head posture parameters can be obtained based on the extended Kalman filter.

[0079] In a specific embodiment, the head posture prediction parameter is the head posture estimation parameter obtained in step S03. Of course, in some other embodiments, the head posture estimation parameter may be input into the system or calculated by other relatively reasonable algorithms.

[0080] S06: Output the head posture estimation parameters obtained in step S03 and end.

[0081] The head posture is estimated by the above three-dimensional head posture estimation algorithm based on facial feature information. First, the camera calibration parameters and the three-dimensional coordinates of multiple model points selected in the facial area of ​​the head model are obtained, and then the two-dimensional pixel coordinates are extracted. The head posture estimation parameters are obtained by the PNP algorithm. After the head posture estimation parameters are obtained, reprojection verification is performed. When the reprojection error is too large, the head posture estimation parameters are corrected using EKF (Extended Kalman Filter) to finally obtain a more accurate result, avoiding the misjudgment of the head posture estimation by the algorithm based on the two-dimensional image. This algorithm can be used to judge the driver's attention. When it is judged that the driver is not paying attention, a prompt message can be issued to reduce the occurrence of road traffic accidents. Of course, the three-dimensional head posture estimation algorithm can also be applied to other processes that require face recognition and posture estimation.

[0082] It is understood that the same or similar parts in the above embodiments can refer to each other, and the content not described in detail in some embodiments can refer to the same or similar content in other embodiments. The multiple solutions provided by the present invention include the basic solution itself, which are independent of each other and do not restrict each other, but they can also be combined with each other without conflict to achieve multiple effects together.

[0083] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A three-dimensional head posture estimation algorithm based on facial feature information, characterized in that: The following steps are involved: S01: Obtain camera calibration parameters and three-dimensional coordinates of multiple model points selected in the facial area of ​​the head model; S02: collecting a facial image of a person whose posture is to be estimated through a camera, extracting feature points corresponding to the model points in the facial image of the person, and obtaining two-dimensional pixel coordinates of each feature point in a pixel coordinate system; S03: Calculate head posture estimation parameters according to camera calibration parameters, three-dimensional coordinates and two-dimensional pixel coordinates; S04: reprojection verification is performed on the head posture estimation parameters to determine whether the reprojection error is greater than a set value. If the reprojection error is greater than the set value, step S06 is executed; if the reprojection error is greater than the set value, step S05 is executed; S05: Correct the head posture estimation parameters obtained in S03 by using an extended Kalman filter, obtain the corrected head posture parameters and then execute step S04; S06: Output the head posture estimation parameters obtained in step S03 and end; The system state equation corrected by extended Kalman filtering is: i k =f(θ k|k-1 )+S K Z k =h(θ k )+V K (1), Correction steps include: Substitute the head posture prediction parameters into equation (2), and then substitute the result into equation (3). Repeat equation (2) and equation (3) to obtain the corrected head posture parameters, where: x k =f(x _k-1 ) x _k =x k +K(z k -h(x k )) p_k=(I-KH k )p k (3), State variable x k =[αβγtx ty tz] T , S k represents the state noise, V k represents the measurement noise. The matrix in equation (2) and equation (3) obtained based on equation (1) is expressed as: Through the above correction steps, the corrected head posture parameters can be obtained based on the extended Kalman filter.

2. The three-dimensional head posture estimation algorithm as claimed in claim 1, characterized in that: The head posture estimation parameters include Euler angles and translation vectors. The Euler angles include the roll angle α, pitch angle β and yaw angle γ of the head, and the translation vector of the head is (tx, ty, tz).

3. The three-dimensional head posture estimation algorithm as claimed in claim 1, characterized in that: Construct a head coordinate system, scan the face of the head model, and obtain the three-dimensional coordinates of each model point in the head coordinate system.

4. The three-dimensional head posture estimation algorithm as claimed in claim 3, characterized in that: The origin of the head coordinate system is the nose tip of the facial area. The positive direction of the Y axis is from the chin point to the center of the eyebrows. The positive direction of the X axis is from the right eye to the left eye. The positive direction of the Z axis is determined by the right-hand rule.

5. The three-dimensional head posture estimation algorithm as claimed in claim 1, characterized in that: The multiple model points selected in the facial area of ​​the head model are P0, P1, ... P n , the two-dimensional pixel coordinates of the feature points on the face image corresponding to each model point are Q0, Q1, ... Q n , according to P x and Q x , and camera calibration parameters, the PNP algorithm is used to solve the head posture estimation parameters, where x = 0, 1, ... n.

6. The three-dimensional head pose estimation algorithm as claimed in claim 1, characterized in that: The selected model points include the nose tip point, the chin point, the left eye outer canthus point, the right eye outer canthus point, the left mouth corner point and the right mouth corner point.

7. The three-dimensional head pose estimation algorithm according to claim 1, characterized in that: Based on the DLIB library, feature points of facial images are extracted and the two-dimensional pixel coordinates of the feature points are obtained.

8. The three-dimensional head posture estimation algorithm as claimed in claim 2, characterized in that: The camera calibration parameters are obtained by using the Matlab camera calibration toolbox according to the checkerboard calibration method. The camera calibration parameters include camera intrinsic parameters and distortion coefficients.

Images