A face pose estimation method and device based on structured light system

Abstract

This application provides a face pose estimation method and apparatus based on a structured light system. The method includes: acquiring a frontal zero-pose face image of the subject; using the 3D point cloud of the frontal zero-pose face image as a standard model; selecting 2D key points from the frontal zero-pose face image to obtain standard 2D key points; and acquiring standard 3D key points corresponding to the standard 2D key points using a structured light system; acquiring a 2D face image of the subject in real time; selecting 2D key points from the 2D face image to obtain real-time 2D key points; and acquiring real-time 3D key points corresponding to the real-time 2D key points using a structured light system; determining a 3D key point cloud based on the standard 3D key points and the real-time 3D key points; and determining the precise pose based on the 3D key point cloud. This scheme is fast, has strong real-time performance, high accuracy, and strong stability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a method and apparatus for estimating face pose based on a structured light system. Background Technology

[0002] With the continuous advancement and development of artificial intelligence (AI) technology, AI technologies, represented by computer vision, have been widely applied in various fields of production and daily life, further promoting automation, intelligence, and convenience in people's lives. Face pose estimation is an important subfield of computer vision, its main purpose being to obtain the orientation information of a face. This can be represented using rotation matrices, rotation vectors, quaternions, or Euler angles, and these representations can be converted to each other. Generally speaking, Euler angles offer stronger visibility and are more widely used. Euler angles include pitch, yaw, and roll, which, in layman's terms, correspond to tilting the head up, tilting the head down, and turning the head.

[0003] Facial pose estimation, as a popular and practical field, has been widely applied in numerous areas of production and daily life, including biometric recognition and human-computer interaction. Examples include facial recognition, driver status detection, human assistance systems, virtual reality, and various entertainment devices. In facial recognition, due to the complex and varied facial poses during actual recognition, facial recognition technology incorporating facial pose estimation can perform recognition after correction, significantly improving accuracy. In driver status detection, facial pose estimation can acquire the driver's facial pose, i.e., facial orientation, in real time. When the orientation exceeds a threshold, it can be considered as fatigue driving or an emergency, allowing for timely warnings. In virtual reality, facial pose estimation can provide relevant information for facial reconstruction, greatly enhancing the user experience in games, social media, and film.

[0004] Face pose estimation is quite challenging due to the diversity of human facial appearance, as well as issues such as changes in facial angles, facial expressions, facial texture, uneven lighting, and facial occlusion.

[0005] Existing methods have low accuracy in facial pose estimation. Summary of the Invention

[0006] The purpose of the embodiments in this specification is to provide a face pose estimation method and device based on a structured light system.

[0007] To solve the above-mentioned technical problems, the embodiments of this application are implemented in the following ways:

[0008] In a first aspect, this application provides a face pose estimation method based on a structured light system, the method comprising:

[0009] Acquire a frontal zero-pose face image of the subject under test, use the 3D point cloud of the frontal zero-pose face image as a standard model, select 2D key points on the frontal zero-pose face image to obtain standard 2D key points, and obtain the standard 3D key points corresponding to the standard 2D key points according to the structured light system.

[0010] The system acquires two-dimensional facial images of the subject in real time, selects 2D key points from the two-dimensional facial images to obtain real-time 2D key points, and obtains real-time 3D key points corresponding to the real-time 2D key points based on the structured light system.

[0011] Determine the 3D key point cloud based on standard 3D key points and real-time 3D key points;

[0012] Determine the precise pose based on the 3D key point cloud.

[0013] In one embodiment, 2D key point selection is performed on a frontal, zero-pose face image or a two-dimensional face image, including:

[0014] Feature point detection is performed on frontal zero-pose face images or two-dimensional face images to obtain facial feature points;

[0015] 2D key points are selected from facial feature points.

[0016] In one embodiment, determining a 3D keypoint cloud based on standard 3D keypoints and real-time 3D keypoints includes:

[0017] Determine the initial pose between the object under test and the standard model based on standard 3D key points and real-time 3D key points;

[0018] The 3D key point cloud is determined based on standard 3D key points, real-time 3D key points, and initial pose.

[0019] In one embodiment, determining a 3D keypoint cloud based on standard 3D keypoints, real-time 3D keypoints, and initial pose includes:

[0020] The complete 3D point cloud corresponding to the 2D face image of the object to be tested is used as the search object. The search is performed in a sphere with a preset radius, centered on the real-time 3D key points. The searched face 3D points are added to the candidate set, which includes the real-time key points and the searched face 3D points.

[0021] For each point in the candidate set, point cloud registration transformation is performed based on the initial pose. After the transformation, the distance from each point to the nearest point in the complete 3D point cloud of the standard model is calculated. When the distance is less than the threshold, the corresponding point in the candidate set is added to the first consensus set, and the corresponding nearest point in the complete 3D point cloud of the standard model is added to the second consensus set.

[0022] The first and second consistent sets constitute the 3D key point cloud.

[0023] In one embodiment, determining the precise pose based on the 3D key point cloud includes:

[0024] Using the initial pose as the initial value, the first and second consistent sets are finely registered to determine the accurate pose.

[0025] In one embodiment, the Trimmed ICP algorithm is used to perform fine registration on the first and second consistency sets.

[0026] In one embodiment, the structured light system includes an infrared camera, an infrared projector, and a terminal device, with the infrared camera and infrared projector respectively connected to the terminal device.

[0027] Secondly, this application provides a face pose estimation device based on a structured light system, the device comprising:

[0028] The first selection module is used to acquire the frontal zero-pose face image of the subject to be tested, use the three-dimensional point cloud of the frontal zero-pose face image as a standard model, select 2D key points on the frontal zero-pose face image to obtain standard 2D key points, and obtain the standard 3D key points corresponding to the standard 2D key points according to the structured light system.

[0029] The second selection module is used to acquire two-dimensional face images of the object under test in real time, select 2D key points from the two-dimensional face images to obtain real-time 2D key points, and obtain real-time 3D key points corresponding to the real-time 2D key points according to the structured light system.

[0030] The first determination module is used to determine the 3D key point cloud based on standard 3D key points and real-time 3D key points;

[0031] The second determination module is used to determine the precise pose based on the 3D key point cloud.

[0032] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the face pose estimation method based on a structured light system as described in the first aspect.

[0033] Fourthly, this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the face pose estimation method based on a structured light system as described in the first aspect.

[0034] As can be seen from the technical solutions provided in the embodiments of this specification above, this solution uses a structured light system for face pose estimation, which is fast, has strong real-time performance, high accuracy, and strong stability. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the structured light system;

[0037] Figure 2 This is a schematic diagram of the coded pattern projected by the projector in a structured light system.

[0038] Figure 3 This is a schematic diagram of the mathematical model of a structured light system;

[0039] Figure 4 A schematic diagram illustrating the use of a structured light system;

[0040] Figure 5 A flowchart illustrating the face pose estimation method based on a structured light system provided in this application;

[0041] Figure 6 This is a schematic diagram of a 68-feature point model of a face detected using Dlib.

[0042] Figure 7 A schematic diagram showing the nose area as a 2D key point;

[0043] Figure 8 A schematic diagram of the face pose estimation device based on a structured light system provided in this application;

[0044] Figure 9 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0046] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0047] Various modifications and variations can be made to the specific embodiments described in this application without departing from the scope or spirit of this application, as will be apparent to those skilled in the art. Other embodiments derived from this application will be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0048] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0049] Unless otherwise specified, "parts" in this application refers to parts by weight.

[0050] Existing methods for facial pose estimation generally include model-based methods, feature regression-based methods, classification-based methods, methods based on the geometric relationships of facial key points, and emerging methods such as subspace learning.

[0051] Among them, model-based methods extract 2D feature points from the face region in a 2D image and establish a correspondence with 3D feature points of a 3D face model to estimate the face pose. This method yields continuous face pose values ​​with high accuracy, making it a commonly used approach. However, the 3D average face model commonly used in this type of method often differs from the 2D face image, and it also exhibits lower accuracy and robustness in estimating poses for large-angle facial shifts and exaggerated expressions.

[0052] One approach, based on feature regression, uses machine learning to obtain a mapping relationship from image space to pose space. This involves building a mathematical model (mainly a neural network) to train a large number of face images with known poses to establish the mapping relationship and determine the face pose of the samples. However, in practice, this mapping relationship requires a large dataset for verification, and interpolation is often required during image processing, resulting in high computational cost. Furthermore, it is significantly affected by the face detection and localization results, and its robustness is insufficient.

[0053] One classification-based method involves dividing facial poses into different categories within a certain range and then classifying the sample to be determined. This type of method quantifies the head pose space into several discrete points and prepares several templates for each pose. The sample to be determined is then compared with each template, and the head pose corresponding to the template with the highest matching score is the classification result. Depending on the templates to be compared, specific methods can be categorized as shape-template based, detection-based, or local constraint-based methods. The physical quantities of the templates to be compared are image texture, pose detectors, and a set of sub-organ detectors arranged in a certain topology, respectively. The results obtained by this type of method are discrete values, resulting in high time complexity, low efficiency, large errors, and difficulty in guaranteeing real-time performance.

[0054] One method, based on the geometric relationships of facial keypoints, first determines the location of facial keypoints and then estimates the facial pose using the relative positions of these keypoints. This method determines the facial pose through certain geometric constraints after identifying the keypoints. This type of method is relatively simple and has low time complexity, but it is highly sensitive to occlusion and missing keypoints, resulting in poor stability.

[0055] One approach, based on subspace learning, views the face's pose space as a natural three-dimensional space, which can be considered as a three-dimensional pose manifold embedded in a high-dimensional image space. While relatively new, this approach currently suffers from high time complexity, low accuracy, and limited practicality.

[0056] To address the aforementioned shortcomings, this application proposes a face pose estimation method based on a structured light system, which can perform high-precision and robust estimation of face pose.

[0057] Structured light technology, as a mature active 3D information acquisition technology, has advantages such as non-contact operation, high precision, good real-time performance, low cost, large field of view, and strong anti-interference ability. 3D reconstruction technology based on structured light systems (including face pose estimation) is an active 3D reconstruction method.

[0058] Structured light systems replace a single camera in traditional binocular vision methods with a projector. For example... Figure 1 As shown, this application employs an infrared projector, an infrared camera (or simply an infrared camera or camcorder), and terminal equipment (such as...). Figure 1 A dynamic structured light system is built using a computer. The structured light system uses a template projected by a codec projector to solve the problem of matching corresponding points, which is difficult to solve in binocular vision. Then, using a calibrated camera and projector, the three-dimensional information of the object can be obtained using the principle of triangulation.

[0059] The structured light system of this application is based on the time-coding principle, specifically a stripe binary encoding method based on Gray code plus line shift encoding. The basic idea is to first create multiple consecutive coded patterns according to the encoding principle, then project this sequence of patterns onto the target surface using a projector, and finally decode the target image with the coded patterns. The point cloud calculated using this method not only has high spatial resolution but also high reconstruction accuracy.

[0060] The Gray code plus line shift encoding method used in this application achieves the encoding purpose by continuously projecting multiple encoded patterns onto the target to be tested. A total of 18 patterns are generated according to this method, such as... Figure 2 As shown, the first two images are all-white and all-black patterns used to extract valid pixels and obtain the current texture, the middle eight images are Gray code coded patterns, and the last eight images are line shift patterns based on Gray code patterns.

[0061] After projecting the target using an coded pattern, the coded value of each pixel needs to be found from the image sequence captured by the camera. Generally, the method for finding the coded value of each pixel is to first determine the position of the stripe edges in the image, then determine the stripe pattern based on the pixel values ​​on both sides of the stripe edge, and finally determine the code value corresponding to each pixel based on the stripe pattern. For example, zero-crossing detection can be used for image stripe edge detection, which can achieve sub-pixel level detection accuracy.

[0062] Let the three stripe edges detected in the coded image be L1, L2, and L3 from left to right, and let gray be the average pixel gray value of a row between edges L1 and L2. L The average grayscale value of pixels in a row between edges L2 and L3 is gray. R The difference in grayscale values ​​between the left and right sides of L2 is .

[0063] gray Δ =gray L -gray R (1)

[0064] If gray Δ If the value is less than 0, it means the pixel value of the stripe on the left side of edge L2 is less than that of the stripe on the right side. The left side of L2 corresponds to a black stripe, and the right side corresponds to a white stripe. If gray... Δ If the value is greater than 0, the stripes are reversed, thus determining the stripe pattern.

[0065] After determining the stripe pattern, the encoding value of the pixels corresponding to black stripes is set to 0, and the encoding value of the pixels corresponding to white stripes is set to 1. The Gray code and line shift encoded images are decoded separately. By further subdividing the Gray code decoding value based on each line shift decoding value, the unique absolute decoding value of each pixel position can be obtained. The process is as follows:

[0066]

[0067] Where P is the absolute decoding value, G is the Gray code decoding value, S is the line shift decoding value, n is the number of Gray code encoded patterns, and m is the number of line shifts. By encoding and decoding the 18 encoded patterns projected by the projector through the above operations, the matching points in the projector and the camera can be mapped one-to-one.

[0068] Based on the camera's imaging model, we can quickly establish the relationship between a spatial 3D point and its corresponding image plane coordinates. In structured light reconstruction technology, we generally consider the projector as a camera with a backlight path, so a camera model can also be used to establish the relevant relationship. The specific mathematical model is represented as follows: Figure 3 As shown, the left image is the projector imaging model, and the right image is the camera imaging model. M is a point on the object to be measured, and the corresponding point of M in the projector image is m. p At the corresponding point m in the camera image c From the camera imaging model, we can obtain:

[0069]

[0070]

[0071] In the formula, the parameters with the superscript 'c' correspond to the camera parameters, and the parameters with the superscript 'p' correspond to the projector parameters. c / p M represents the 2D image coordinates in the digital image coordinate system. c / p M represents the coordinates in the camera / projector coordinate system. w R represents the coordinates in the world coordinate system. c / p and T c / p Let be the rotation and translation matrix (extrinsic parameter) of the camera / projector coordinate system relative to the world coordinate system, s be the scale factor, and f be the translation matrix (extrinsic parameter). u f v u0, v0, and γ are the parameters (intrinsic parameters) of the intrinsic parameter matrix. Through calibration, we can obtain the intrinsic and extrinsic parameters of the camera and projector.

[0072] The pose relationship between the camera and the projector is shown in equation (5):

[0073]

[0074] Where R and T are the rotation and translation matrices between the projector and camera coordinate systems. The extrinsic parameter R is obtained through calibration. c R p T c and T p Then, R and T can be obtained from the coordinate transformation relationship.

[0075] Let the stripe edge points be extracted on the camera image plane. ( (representing the projection of M) and the projector pattern generation plane These are matching points for the same scene point M. For two-dimensional positions, we encode along the x-axis (horizontal direction), and determine u by matching the encoded values. c and u p The correspondence.

[0076] like Figure 3 As shown, O c O p It is an epipolar line. Using epipolar constraints, we can obtain:

[0077]

[0078] Where K c and K p This is the intrinsic parameter matrix of the camera and projector. According to Formula 6, v in the y-axis (vertical) can be determined. c and v p From the correspondence, we can determine the corresponding matching point. and The complete coordinates.

[0079] Finally, we can obtain the depth information z using the principle of triangulation. c :

[0080]

[0081] Using formula (3), the complete three-dimensional spatial point information M can be obtained. c (x c y c , z c This enables a one-to-one 2D-3D correspondence between two-dimensional pixels and three-dimensional spatial points.

[0082] like Figure 4 The diagram shows a specific schematic of the structured light system. In this application, a DLP4500 infrared projector continuously projects 18 coded patterns onto a person's face in real time, and an infrared camera captures the face data of the projected patterns in real time. By encoding and decoding the patterns, a high-precision real-time point cloud data sequence can be obtained, realizing a one-to-one correspondence between 2D and 3D face data (i.e., 2D face data and 3D point cloud data).

[0083] The structured light system used in this application is based on the time coding principle, specifically the Gray code plus line shift coding method. Using this structured light system, high-precision, high-frame-rate, and one-to-one correspondence between two-dimensional face pixels can be obtained for face pose estimation.

[0084] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0085] Reference Figure 5 It shows a flowchart of the face pose estimation method based on structured light system provided in the embodiments of this application.

[0086] like Figure 5 As shown, a face pose estimation method based on a structured light system may include:

[0087] S510. Obtain a frontal zero-pose face image of the subject to be tested. Use the 3D point cloud of the frontal zero-pose face image as a standard model. Select 2D key points on the frontal zero-pose face image to obtain standard 2D key points. Obtain the standard 3D key points corresponding to the standard 2D key points according to the structured light system.

[0088] Specifically, the object to be tested refers to the object whose facial pose is to be estimated, which can be a driver in a driving state, a player playing virtual reality, or a person performing facial recognition, etc.

[0089] A frontal zero-pose face image refers to an image of the face pose obtained when the subject's face is facing the camera.

[0090] Using the 3D point cloud of the frontal, zero-pose face image of the subject under test as the standard model (or standard face, standard face 3D model, or standard face model), compared with directly using the 3D average face as the standard model, this approach can overcome the diversity of face appearance and differences in face texture to some extent.

[0091] S520: Real-time acquisition of two-dimensional facial images of the object under test; selection of 2D key points for the two-dimensional facial images; and acquisition of 3D key points corresponding to the 2D key points based on the structured light system.

[0092] Specifically, a two-dimensional face image refers to an image captured by a camera in real time.

[0093] The infrared structured light system of this application can obtain one-to-one corresponding 2D face data and 3D point cloud data. For face pose estimation tasks, using the entire 3D point cloud (the complete point cloud corresponding to a 2D face photo is called the 3D point cloud) for calculation is computationally intensive and very time-consuming, and is also affected by factors such as changes in facial expressions, resulting in poor stability and robustness. Therefore, this application selects stable 2D keypoints and 3D keypoints for face pose estimation.

[0094] In one embodiment, 2D key point selection is performed on a frontal zero-pose face image or a two-dimensional face image, including:

[0095] Feature point detection is performed on frontal zero-pose face images or two-dimensional face images to obtain facial feature points;

[0096] 2D key points are selected from facial feature points.

[0097] Specifically, when performing keypoint detection and selection on frontal zero-pose face images or 2D face images, it should be noted that this application does not limit the specific algorithm for face feature point detection. Commonly used optimization-based methods (such as ASM and AAM), regression-based methods (cascaded pose regression and SDM), and deep learning-based methods can all be used in this invention to achieve face feature point detection. The following description uses Dlib, based on the cascaded regression method, to implement 68 face feature point detection as an example.

[0098] Dlib uses classic oriented histograms, gradient features, and linear classifiers for face detection, and employs ERT cascaded regression, a gradient-based improvement learning regression tree method, to detect 68 facial feature points. Figure 6 As shown, it includes parts such as eyebrows, eyes, nose, mouth, and facial contours.

[0099] Dlib can be used to obtain the coordinates of facial feature points from the 2D images acquired by the system. To avoid the influence of facial expression changes and improve robustness, we further select feature points. During different facial expressions, the eyebrows, eyes, mouth, and chin exhibit larger movements, so feature points from these areas are not selected. The nose is relatively stable and is chosen as the key point for subsequent estimation. Therefore, 9 points on the nose are selected as 2D key points, such as... Figure 7 As shown. Understandably, different stable regions can be selected as key points depending on the actual situation, such as using key point clouds of the nose, sides of the cheeks, and corners of the eyes simultaneously.

[0100] Based on the 2D keypoint coordinates already obtained and the 2D-3D geometric correspondence determined in the infrared structured light system, the 3D keypoint coordinates corresponding to the 2D keypoints can be obtained, specifically including the standard 3D keypoints corresponding to the standard 2D keypoints and the real-time 3D keypoints corresponding to the real-time 2D keypoints.

[0101] S530. Based on standard 3D key points and real-time 3D key points, determine the 3D key point cloud, which may include:

[0102] Determine the initial pose between the object under test and the standard model based on standard 3D key points and real-time 3D key points;

[0103] The 3D key point cloud is determined based on standard 3D key points, real-time 3D key points, and initial pose.

[0104] Specifically, based on the coordinates of the determined standard 3D key points and real-time 3D key points, the initial poses R0 and t0 between the object under test and the standard model can be calculated. These initial poses can be used as the initial values ​​for subsequent fine registration or for acquiring 3D key point clouds.

[0105] To prevent large-angle changes in the face and improve the stability and robustness of face pose estimation, this application extends the acquisition of 3D key point clouds based on the existing 3D key points.

[0106] In one embodiment, determining a 3D keypoint cloud based on standard 3D keypoints, real-time 3D keypoints, and initial pose may include:

[0107] The complete 3D point cloud corresponding to the 2D face image of the object to be tested is used as the search object. The search is performed in a sphere with a preset radius, centered on the real-time 3D key points. The searched face 3D points are added to the candidate set, which includes the real-time key points and the searched face 3D points.

[0108] For each point in the candidate set, point cloud registration transformation is performed based on the initial pose. After the transformation, the distance from each point to the nearest point in the complete 3D point cloud of the standard model is calculated. When the distance is less than the threshold, the corresponding point in the candidate set is added to the first consensus set, and the corresponding nearest point in the complete 3D point cloud of the standard model is added to the second consensus set.

[0109] The first and second consistent sets constitute the 3D key point cloud.

[0110] Specifically, candidate set C is constructed as follows: The complete 3D point cloud corresponding to the 2D face photo (i.e., the 2D face image of the subject to be tested) is used as the search object. Using the acquired real-time 3D keypoints as the center, a search is performed within a sphere of radius r (i.e., a preset radius, which can be set according to actual needs). The searched 3D face points are added to candidate set C. Following the above operations, the candidate set C of the face to be estimated can be obtained. Candidate set C includes 3D keypoints and newly searched 3D face points.

[0111] Construction of the Consistency Set S (i.e., 3D Keypoint Cloud): Based on the candidate set C, a consistency set S is constructed using an algorithm similar to RANSAC. Specifically, for each point p in the candidate set C, point cloud registration transformation is performed based on the initial pose R0 and T0. After the transformation, the distance from each point to the nearest point in the complete 3D point cloud of the standard face model is calculated. Points p whose distance is less than a threshold δ (which can be set according to actual needs) are added to the consistency set S0 (i.e., the first consistency set), and the nearest point q corresponding to point p is added to the consistency set S1 (i.e., the second consistency set). Consistency sets S0 and S1 constitute the acquired 3D keypoint cloud.

[0112] S540. Based on the 3D key point cloud, determine the precise pose, including:

[0113] Using the initial pose as the initial value, fine registration is performed on the first and second consistent sets to determine the accurate pose. Specifically, the Trimmed ICP algorithm is used for fine registration of the first and second consistent sets.

[0114] Specifically, this application uses the Trimmed ICP algorithm for fine registration, and uses the LTS (the leasttrimmed squares) method to sort the residuals obtained for each set of matching points in ascending order. Only the first proportion of corresponding points is selected to fit the error function, and R and T are solved by iteratively minimizing the error function.

[0115] The specific process is as follows:

[0116] a) For each point in the first consistent set S0 (also known as the source point cloud), find its matching point in the second consistent set S1 (also known as the target point cloud), and calculate the squared value of its residual d. i (R, T) 2 .

[0117] Wherein, let there be N points in the source point cloud S0. p One point, that is For each point p in S0 i The corresponding point after R and T transformations is p. i (R, T) = R·p i +T, we define each point p in the target point cloud S1. i (R, T) Find matching points, as shown in equation (8):

[0118]

[0119] After finding the nearest point as the matching point, calculate the residual d for each set of matching points. i (R, T):

[0120] d i (R, T) = |m cl (i, R, T)-p i (R, T)| (9)

[0121] b) Regarding d i (R, T) 2 Sort by appreciation value and select the top N. po The sum of the squared residuals corresponding to each point is S′. LTS Where N po The number of selected points is obtained according to equation (10):

[0122] Npo =ξN p (10)

[0123] Where ξ is the minimum overlap rate, which can be adaptively obtained by minimizing the objective function (11):

[0124] ψ(ξ)=e(ξ)ξ -(1+λ) (11)

[0125] In the formula, λ is a pre-defined non-negative number, and e is a trimmed MSE, as shown in formula (12):

[0126] e = S′ LTS / N po (12)

[0127] c) If the termination condition is met, exit the iteration; otherwise, start a new iteration by minimizing S′. LTS Calculate N po The optimal transformations R and T of the selected points are used to transform the corresponding points, and then a is returned.

[0128] The termination condition is: the maximum number of iterations is reached (this maximum number of iterations can be set according to actual needs), and the trimmed MSE e = S′. LTS / N po Either the relative transformation amount |ee′| / e is sufficiently small, or the trimmed MSE is sufficiently small.

[0129] The 3D keypoint clouds S0 and S1 are registered using the Trimmed ICP algorithm, and the optimal rotation matrix R is the final determined face pose matrix. Understandably, if better visualization is needed, R can be decomposed to obtain the Euler angle representation of the face pose, namely pitch, yaw, and roll.

[0130] It is understandable that, in this embodiment, other registration methods can also be used to perform fine registration operations on the first and second consistency sets.

[0131] The face pose estimation method based on structured light system provided in this application uses structured light system to estimate face pose. This method is fast, has strong real-time performance, high accuracy, and strong stability.

[0132] The face pose estimation method based on structured light systems provided in this application does not require many limitations. Furthermore, this invention selects stable regions for face pose estimation based on high-precision real-time face reconstruction data, achieving higher accuracy compared to most existing algorithms, and exhibiting better robustness to problems such as changes in facial expression and large angles.

[0133] Reference Figure 8 The diagram illustrates a structural schematic of a face pose estimation device based on a structured light system according to an embodiment of this application.

[0134] like Figure 8 As shown, the face pose estimation device 800 based on a structured light system may include:

[0135] The first selection module 810 is used to acquire a frontal zero-pose face image of the subject to be tested, use the three-dimensional point cloud of the frontal zero-pose face image as a standard model, select 2D key points on the frontal zero-pose face image to obtain standard 2D key points, and obtain standard 3D key points corresponding to the standard 2D key points according to the structured light system.

[0136] The second selection module 820 is used to acquire two-dimensional face images of the object under test in real time, select 2D key points from the two-dimensional face images to obtain real-time 2D key points, and obtain real-time 3D key points corresponding to the real-time 2D key points according to the structured light system.

[0137] The first determining module 830 is used to determine the 3D key point cloud based on standard 3D key points and real-time 3D key points;

[0138] The second determining module 840 is used to determine the precise pose based on the 3D key point cloud.

[0139] Optionally, the first selection module 810 or the second selection module 820 is further used for:

[0140] Feature point detection is performed on frontal zero-pose face images or two-dimensional face images to obtain facial feature points;

[0141] 2D key points are selected from facial feature points.

[0142] Optionally, the first determining module 830 is also used for:

[0143] Determine the initial pose between the object under test and the standard model based on standard 3D key points and real-time 3D key points;

[0144] The 3D key point cloud is determined based on standard 3D key points, real-time 3D key points, and initial pose.

[0145] Optionally, the first determining module 830 is also used for:

[0146] The complete 3D point cloud corresponding to the 2D face image of the object to be tested is used as the search object. The search is performed in a sphere with a preset radius, centered on the real-time 3D key points. The searched face 3D points are added to the candidate set, which includes the real-time key points and the searched face 3D points.

[0147] For each point in the candidate set, point cloud registration transformation is performed based on the initial pose. After the transformation, the distance from each point to the nearest point in the complete 3D point cloud of the standard model is calculated. When the distance is less than the threshold, the corresponding point in the candidate set is added to the first consensus set, and the corresponding nearest point in the complete 3D point cloud of the standard model is added to the second consensus set.

[0148] The first and second consistent sets constitute the 3D key point cloud.

[0149] Optionally, the second determining module 840 is also used for:

[0150] Using the initial pose as the initial value, the first and second consistent sets are finely registered to determine the accurate pose.

[0151] Optionally, the Trimmed ICP algorithm can be used to perform fine registration on the first and second consistent sets.

[0152] Optionally, the structured light system includes an infrared camera, an infrared projector, and a terminal device, with the infrared camera and infrared projector connected to the terminal device respectively.

[0153] This embodiment provides a face pose estimation device based on a structured light system, which can perform the above-described method. Its implementation principle and technical effect are similar, and will not be repeated here.

[0154] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 9 The diagram shows a structural schematic of an electronic device 900 suitable for implementing embodiments of this application.

[0155] like Figure 9 As shown, the electronic device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the device 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0156] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 906 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0157] In particular, according to embodiments of this disclosure, the above references Figure 1 The described process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the aforementioned face pose estimation method based on a structured light system. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911.

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0160] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a mobile phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0161] In another aspect, this application also provides a storage medium, which may be the storage medium included in the aforementioned apparatus in the above embodiments; or it may be a standalone storage medium not assembled into the device. The storage medium stores one or more programs, which are used by one or more processors to execute the face pose estimation method based on the structured light system described in this application.

[0162] Storage media, including both permanent and non-permanent, removable and non-removable media, can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0163] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0164] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for face pose estimation based on a structured light system, characterized in that, The method includes: A zero-pose frontal face image of the subject to be tested is acquired. The three-dimensional point cloud of the zero-pose frontal face image is used as a standard model. 2D key points are selected from the zero-pose frontal face image to obtain standard 2D key points. The standard 3D key points corresponding to the standard 2D key points are obtained according to the structured light system. The system acquires a two-dimensional face image of the object under test in real time, selects 2D key points from the two-dimensional face image to obtain real-time 2D key points, and obtains the real-time 3D key points corresponding to the real-time 2D key points according to the structured light system. Based on the standard 3D key points and the real-time 3D key points, the initial pose between the object under test and the standard model is determined; the complete 3D point cloud corresponding to the 2D face image of the object under test is used as the search object, and a search is performed within a sphere of a preset radius centered on the acquired real-time 3D key points. The searched face 3D points are added to a candidate set, which includes the real-time key points and the searched face 3D points; for each point in the candidate set, point cloud registration transformation is performed based on the initial pose, and the distance from each point to the nearest point in the complete 3D point cloud of the standard model is calculated after the transformation. When the distance is less than a threshold, the corresponding point in the candidate set is added to a first consistency set, and the corresponding nearest point in the complete 3D point cloud of the standard model is added to a second consistency set; the first consistency set and the second consistency set constitute a 3D key point cloud; Determining the precise pose based on the 3D key point cloud includes: using the initial pose as the initial value, performing fine registration on the first consistency set and the second consistency set to determine the precise pose.

2. The method of claim 1, wherein, Selecting 2D key points from the frontal zero-pose face image or the 2D face image includes: Feature point detection is performed on the frontal zero-pose face image or the two-dimensional face image to obtain face feature points; 2D key points are selected from facial feature points.

3. The method according to claim 1, characterized in that, The Trimmed ICP algorithm is used to perform fine registration on the first and second consensus sets.

4. The method according to any one of claims 1-3, characterized in that, The structured light system includes an infrared camera, an infrared projector, and a terminal device, wherein the infrared camera and the infrared projector are respectively connected to the terminal device.

5. A face pose estimation device based on a structured light system, characterized in that, The device includes: The first selection module is used to acquire a frontal zero-pose face image of the subject to be tested, use the three-dimensional point cloud of the frontal zero-pose face image as a standard model, select 2D key points on the frontal zero-pose face image to obtain standard 2D key points, and obtain standard 3D key points corresponding to the standard 2D key points according to the structured light system. The second selection module is used to acquire a two-dimensional face image of the object to be tested in real time, select 2D key points from the two-dimensional face image to obtain real-time 2D key points, and obtain the real-time 3D key points corresponding to the real-time 2D key points according to the structured light system. The first determining module is used to determine the initial pose between the object under test and the standard model based on the standard 3D key points and the real-time 3D key points; using the complete 3D point cloud corresponding to the 2D face image of the object under test as the search object, and searching within a sphere of a preset radius centered on the acquired real-time 3D key points, adding the searched face 3D points to a candidate set, the candidate set including the real-time key points and the searched face 3D points; for each point in the candidate set, performing point cloud registration transformation based on the initial pose, and calculating the distance from each point to the nearest point in the complete 3D point cloud of the standard model after the transformation; when the distance is less than a threshold, adding the corresponding point in the candidate set to a first consistency set, and adding the corresponding nearest point in the complete 3D point cloud of the standard model to a second consistency set; the first consistency set and the second consistency set constitute a 3D key point cloud; The second determining module is used to determine the precise pose based on the 3D key point cloud, including: using the initial pose as the initial value, performing fine registration on the first consistency set and the second consistency set to determine the precise pose.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the face pose estimation method based on a structured light system as described in any one of claims 1-4.

7. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the face pose estimation method based on the structured light system as described in any one of claims 1-4.

Images