Facial recognition method and apparatus based on depth information

Abstract

The present application relates to the technical field of fatigue detection. Disclosed are a facial recognition method and apparatus based on depth information, which can correspondingly obtain higher-precision 3D coordinates of facial landmarks when depth information is used for performing facial depth estimation, thereby improving the facial recognition effect. The method comprises: acquiring 2D coordinates of facial landmarks and 3D depth information of the facial landmarks; inputting the 2D coordinates and 3D depth information of the facial landmarks into a pre-constructed 3D facial model, and solving for parameters by means of iterative optimization to obtain facial pose parameters and weighting coefficients of the 3D facial model; on the basis of the facial pose parameters and the weighting coefficients of the 3D facial model, fusing facial geometric information contained in the 3D facial model to obtain 3D coordinates of the facial landmarks; and using the 3D coordinates of the facial landmarks to perform facial recognition.

Description

A face recognition method and device based on depth information

[0001] This application claims priority to Chinese Patent Application No. 202410489672.7, filed on April 23, 2024, entitled “Method and Apparatus for Face Recognition Based on Depth Information”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of face recognition technology, and in particular to a face recognition method and apparatus based on depth information. Background Technology

[0003] In the field of autonomous driving, monocular depth estimation can provide three-dimensional depth information of a face, which can help the system better understand the spatial structure of the face, thereby improving the accuracy of face recognition.

[0004] In facial recognition technology, monocular depth estimation obtains the depth information of objects in a facial image by analyzing the focal information of different regions. Generally, objects closer to the camera have higher sharpness, while objects farther away have lower sharpness. In related technologies, Time-of-Flight (TOF) cameras can directly acquire the 3D coordinates of a face, theoretically eliminating the need for depth estimation. However, in practice, TOF cameras are highly susceptible to environmental influences, generating invalid depth information and experiencing a significant drop in accuracy in distant scenes. Such depth information is unlikely to achieve the desired facial recognition results.

[0005] Application content

[0006] In view of this, this application provides a face recognition method and apparatus based on depth information, the main purpose of which is to solve the problem that the depth information provided by TOF in the prior art is difficult to achieve the expected face recognition effect.

[0007] According to a first aspect of this application, a face recognition method based on depth information is provided, the method comprising:

[0008] Obtain the 2D coordinates and 3D depth information of facial key points;

[0009] The 2D coordinates of the facial key points and the 3D depth information are input into a pre-constructed 3D face model. The parameters are solved through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model.

[0010] The facial geometric information contained in the 3D face model is fused based on the facial pose parameters and the weighting coefficients of the 3D face model to obtain the 3D coordinates of the facial key points.

[0011] Face recognition is performed using the 3D coordinates of the aforementioned facial key points.

[0012] Furthermore, the acquisition of the 2D coordinates and 3D depth information of the facial key points specifically includes:

[0013] The 2D coordinates of facial key points in the face image of the user to be detected are located using a pre-trained key point detection model;

[0014] Based on the position information of the 2D coordinates of the facial key points in the preset facial image, the 3D depth information of the facial key points at the corresponding positions is read. The preset facial image is a facial image captured by a camera device with a depth imaging mechanism.

[0015] Furthermore, after locating the 2D coordinates of facial key points in the face image of the user to be detected using a pre-trained key point detection model, the method further includes:

[0016] Intrinsic parameter calibration is performed on camera devices with depth imaging mechanisms, and distortion correction processing is performed on the 2D coordinates of the facial key points based on the calibrated intrinsic parameter information.

[0017] Furthermore, before inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model, and solving for the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model, the method further includes:

[0018] A 3D face model with fusion properties is generated using a parametric model, wherein the 3D face model includes a 3D face structure obtained by weighting different face features;

[0019] Based on the 3D structure of the face, extract the geometric information of the face contained in the 3D face model.

[0020] Furthermore, the step of inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model, and solving for the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model, specifically includes:

[0021] The 2D coordinates of the facial key points and the 3D depth information are input into a pre-constructed 3D face model to determine the 3D coordinates of the current facial key points in the initial state.

[0022] Based on the 3D coordinates of the current facial key points in the initial state, the facial pose parameters in the world coordinate system relative to the camera coordinate system are solved iteratively by the least squares method.

[0023] The loss function of the regularization term is defined based on the face pose parameters, and the weighting coefficients of the 3D face model are iteratively solved using the loss function of the regularization term.

[0024] Furthermore, after inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model to determine the 3D coordinates of the current facial key points in the initial state, the method further includes:

[0025] The system detects whether the 3D depth information of the facial key points is within the usable range using preset rules.

[0026] If so, the 3D coordinates of the current facial key points are added to the first input coordinate set, and the 3D depth information of the facial key points is added to the first output coordinate set;

[0027] Otherwise, add the 3D coordinates of the current facial key points to the second input coordinate set, and add the 2D coordinates of the facial key points to the second output coordinate set;

[0028] Accordingly, the step of iteratively solving the face pose parameters in the world coordinate system relative to the camera coordinate system using the least squares method based on the 3D coordinates of the current facial key points in the initial state specifically includes:

[0029] When the first output coordinate set is not empty and the second output coordinate set is empty, the loss function is defined using the first input coordinate set and the first output coordinate set;

[0030] When the second output coordinate set is not empty and the first output coordinate set is empty, the loss function is defined using the second input coordinate set and the second output coordinate set;

[0031] The face pose parameters in the world coordinate system relative to the camera coordinate system are solved iteratively by the least squares method. The face pose parameters are output when the value calculated by the loss function satisfies the iteration stopping condition.

[0032] Furthermore, the step of defining a loss function with regularization terms based on the face pose parameters, and iteratively solving for the weighting coefficients of the 3D face model using the loss function with regularization terms, specifically includes:

[0033] Based on the facial pose parameters, a weighting coefficient of the 3D face model is added to the loss function to define a regularization loss function, wherein the weighting coefficients include an identity coefficient and an expression coefficient.

[0034] The loss function of the regularization term is used to iteratively solve the identity coefficient and expression coefficient of the 3D face model. When the value calculated by the loss function meets the iteration stopping condition, the weighted coefficient of the 3D face model is output according to the current identity coefficient and expression coefficient.

[0035] According to a second aspect of this application, a face recognition device based on depth information is provided, the device comprising:

[0036] The acquisition unit is used to acquire the 2D coordinates of facial key points and the 3D depth information of facial key points;

[0037] The solving unit is used to input the 2D coordinates of the facial key points and the 3D depth information into the pre-constructed 3D face model, and solve the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model.

[0038] The fusion unit is used to fuse the facial geometric information contained in the 3D face model according to the facial pose parameters and the weighting coefficients of the 3D face model to obtain the 3D coordinates of the facial key points.

[0039] The recognition unit is used to perform face recognition using the 3D coordinates of the facial key points.

[0040] Furthermore, the acquisition unit is specifically used to locate the 2D coordinates of facial key points in the face image of the user to be detected using a pre-trained key point detection model; and to read the 3D depth information of the facial key points at the corresponding positions based on the position information of the 2D coordinates of the facial key points in a preset face image, wherein the preset face image is a face image captured by a camera device with a depth imaging mechanism.

[0041] Furthermore, the acquisition unit is specifically used to perform intrinsic parameter calibration on a camera device with a depth imaging mechanism after the 2D coordinates of the facial key points are located in the face image of the user to be detected using a pre-trained key point detection model, and to perform distortion correction processing on the 2D coordinates of the facial key points based on the calibrated intrinsic parameter information.

[0042] Furthermore, the device also includes:

[0043] The generation unit is used to generate a 3D face model with fusion properties using a parameterized model before inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model and solving the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model. The 3D face model includes a 3D face structure obtained by weighting different facial features.

[0044] The extraction unit is used to extract the facial geometric information contained in the 3D face model based on the 3D face structure.

[0045] Furthermore, the solution unit includes:

[0046] The determination module is used to input the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D facial model to determine the 3D coordinates of the current facial key points in the initial state;

[0047] The first solution module is used to solve the face pose parameters in the world coordinate system relative to the camera coordinate system by using the least squares method based on the 3D coordinates of the current face key points in the initial state.

[0048] The second solution module is used to define a loss function for the regularization term based on the face pose parameters, and to iteratively solve the weighting coefficients of the 3D face model using the loss function for the regularization term.

[0049] Furthermore, the solving unit also includes:

[0050] The detection module is used to detect whether the 3D depth information of the facial key points is within the usable range according to preset rules;

[0051] An addition module is used to add the 3D coordinates of the current facial key points to the first input coordinate set and add the 3D depth information of the facial key points to the first output coordinate set if the condition is met.

[0052] The adding module is also used to otherwise add the 3D coordinates of the current facial key points to the second input coordinate set and add the 2D coordinates of the facial key points to the second output coordinate set;

[0053] Accordingly, the first solution module is specifically used to define a loss function using the first input coordinate set and the first output coordinate set when the first output coordinate set is not empty and the second output coordinate set is empty; and to define a loss function using the second input coordinate set and the second output coordinate set when the second output coordinate set is not empty and the first output coordinate set is empty; to iteratively solve the face pose parameters in the world coordinate system relative to the camera coordinate system using the least squares method; and to output the face pose parameters when the value calculated by the loss function satisfies the iteration stopping condition.

[0054] Furthermore, the second solving module is specifically used to add weighting coefficients of the 3D face model to the loss function according to the face pose parameters, define a loss function with regularization terms, and the weighting coefficients include identity coefficients and expression coefficients; use the loss function with regularization terms to iteratively solve the identity coefficients and expression coefficients of the 3D face model respectively, and when the value calculated by the loss function satisfies the iteration stopping condition, output the weighting coefficients of the 3D face model according to the current identity coefficients and expression coefficients.

[0055] According to a third aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described face recognition method based on depth information.

[0056] According to a fourth aspect of this application, a depth-based face recognition device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the aforementioned depth-based face recognition method.

[0057] By employing the above technical solution, this application provides a face recognition method and apparatus based on depth information. Compared with the current method of using a TOF camera to estimate the face and obtain the 3D coordinates of facial key points for face recognition, this application obtains the 2D coordinates and 3D depth information of facial key points, inputs the 2D coordinates and 3D depth information of facial key points into a pre-constructed 3D face model, and solves the parameters through iterative optimization to obtain face pose parameters and weighting coefficients of the 3D face model. The weighting coefficients of the 3D face model are used to fuse the geometric information of the face contained in the 3D face model to obtain the 3D coordinates of the facial key points. Face recognition is then performed using the 3D coordinates of the facial key points. The entire process uses the 3D depth information of the facial key points to iteratively solve the face pose parameters and the weighting coefficients of the 3D face model in the pre-constructed 3D face model. This iterative solution can produce more stable depth information, and thus obtain higher precision 3D coordinates of the facial key points when using the depth information for face depth estimation, thereby improving the face recognition effect.

[0058] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0059] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0060] Figure 1 is a flowchart illustrating a face recognition method based on depth information in one embodiment of this application;

[0061] Figure 2 is a flowchart illustrating a specific implementation of step 101 in Figure 1;

[0062] Figure 3 is a flowchart illustrating a face recognition method based on depth information in another embodiment of this application;

[0063] Figure 4 is a flowchart illustrating a specific implementation of step 102 in Figure 1.

[0064] Figure 5 is a schematic diagram of the structure of a face recognition device based on depth information in one embodiment of this application;

[0065] Figure 6 is a schematic diagram of the device structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0066] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0067] In related technologies, TOF cameras can directly obtain the three-dimensional coordinates of a face, theoretically eliminating the need for depth estimation. However, in reality, TOF cameras are very susceptible to environmental influences, generating invalid depth information and experiencing a significant reduction in accuracy in distant scenes. Such depth information is unlikely to achieve the expected face recognition effect.

[0068] To address this issue, this embodiment provides a face recognition method based on depth information, as shown in Figure 2. This method can be applied to in-vehicle servers and includes the following steps:

[0069] 101. Obtain the 2D coordinates and 3D depth information of the facial key points.

[0070] Facial key points can include, but are not limited to, facial features and contours, such as eyebrows, eyes, nose, mouth, and facial outlines. The 2D coordinates of these facial key points correspond to the key points of a face image in two-dimensional space. Typically, in-vehicle facial image acquisition systems, such as video surveillance cameras with facial capture capabilities, can be used to acquire facial images and real-time video streams. The acquired facial images and / or real-time video streams are then transmitted to an in-vehicle server, which performs image processing to obtain the 2D coordinates of the facial key points. Specifically, in obtaining the 2D coordinates of facial key points, face detection algorithms can be used to identify the position and size of faces in the image. Deep learning algorithms can be used for face detection. Then, the detected faces are aligned to reduce the impact of pose and scale variations on key point extraction. Common alignment methods include feature-point-based alignment and geometric transformation-based alignment. Furthermore, on the aligned face image, a feature point localization algorithm is used to locate the 2D coordinates of key points. Common feature point localization methods include template matching, regression, and deep learning. Finally, the feature point localization results may contain some errors or inaccuracies. Smoothing, shape constraints, and other operations can be used to improve the accuracy of facial key points.

[0071] In practical applications, considering that the face captured by the device may not be a frontal view, it is necessary to correct the facial pose in the image. This is done by obtaining the coordinates of the key points of the face through facial landmark detection, adjusting the angle of the face based on the coordinates of the key points, and then using affine transformation to correct the face.

[0072] In this context, 3D depth information of facial landmarks refers to the distance information of facial landmarks in three-dimensional space. It can be used to represent the protrusions and depressions of the face, such as the nose, mouth, and eyes. Specifically, depth sensors or structured light technology can be used to acquire the depth information of facial landmarks. A depth sensor is a device that measures the distance between an object and the sensor. Common depth sensors include Time-of-Flight (TOF) sensors and infrared sensors. They calculate the distance between the object and the sensor by emitting light or infrared radiation and measuring the time or shape change of its return, thereby acquiring the 3D depth information of facial landmarks. Structured light technology, on the other hand, calculates the 3D depth information of facial landmarks by projecting special light patterns or light spot shape changes.

[0073] The execution subject of this application embodiment can be a face recognition device based on depth information. It can automatically detect and locate the 2D coordinates of some specific facial key points from a given face image, such as eyes, nose, mouth, etc. The position information of these key points contains information in the face image. The 3D depth information of the facial key points in the face image can be obtained through the TOF sensor. By combining the 2D coordinates and 3D depth information of the facial key points, it is very important for tasks such as face recognition, expression analysis, and pose estimation.

[0074] 102. Input the 2D coordinates of the facial key points and the 3D depth information into the pre-constructed 3D face model, and solve the parameters through iterative optimization to obtain the face pose parameters and the weighting coefficients of the 3D face model.

[0075] Among them, the pre-built 3D face model is a parametric model obtained by three-dimensional modeling of the face image. Specifically, a parametric 3DMM model can be built using facial key points. This model can generate different face shapes and textures according to the input parameters. The model usually consists of a basic shape and texture model and a set of principal components that vary the shape and texture.

[0076] Specifically, in the process of iterative optimization to solve for parameters, optimization algorithms can be used to iteratively match the facial features, which take the 2D coordinates and 3D depth information of the facial key points as input, with the 3DMM model. During the iterative matching process, the facial pose parameters and the weighting coefficients of the 3D facial model are solved. The facial pose parameters are equivalent to the head pose. The weighting coefficients of the 3D facial model include identity coefficients and expression coefficients, which can control the shape and texture changes of the facial model, thereby achieving different expression effects.

[0077] 103. Based on the facial pose parameters and the weighting coefficients of the 3D facial model, the facial geometric information contained in the 3D facial model is fused to obtain the 3D coordinates of the facial key points.

[0078] Understandably, 3D face models can reflect the fusion information of a face in three-dimensional space. The facial geometric information contained in a 3D face model contains the shape and texture variation features of a face image. By reconstructing the face pose parameters and the weighting coefficients of the 3D face model through the 3D face model, different facial poses and expressions can be fused with the facial geometric information contained in the 3D face model, making the local details of the face more refined, and thus obtaining more accurate 3D coordinates of facial key points.

[0079] Specifically, the average face, expression basis, and identity basis can be extracted from the facial geometric information contained in the 3D face model. The expression basis and identity basis are then weighted and summed with the weighting coefficients of the 3D face model to obtain the weighted face representation. Finally, the weighted face representation is fused into the average face to obtain the 3D coordinates of the facial key points.

[0080] 104. Perform face recognition using the 3D coordinates of the aforementioned facial key points.

[0081] Understandably, the 3D coordinates of facial key points in a three-dimensional coordinate system include not only positional information on the XY plane but also depth information on the Z axis, giving them a stronger descriptive ability and enabling them to more realistically reflect facial information.

[0082] Specifically, after obtaining the 3D coordinates of facial key points, a face recognition task can be established based on these coordinates. Face features can then be extracted based on the face recognition task to obtain features of facial region components, such as the eyes and mouth. These features can be obtained using various algorithms and models. The extracted features of facial region components are then matched with known facial features. Machine learning algorithms and deep learning models can be used for face matching to determine whether the individuals are the same person.

[0083] The face recognition method based on depth information provided in this application, compared with the current method of using a TOF camera to estimate the face and obtain the 3D coordinates of facial key points for face recognition, obtains the 2D coordinates and 3D depth information of facial key points. This information is then input into a pre-constructed 3D face model, and parameters are solved iteratively to obtain face pose parameters and weighting coefficients of the 3D face model. Based on these parameters, the geometric information of the face contained in the 3D face model is fused to obtain the 3D coordinates of the facial key points. Face recognition is then performed using these 3D coordinates. The entire process uses the 3D depth information of the facial key points to iteratively solve for the face pose parameters and weighting coefficients of the pre-constructed 3D face model. This iterative solution produces more stable depth information, resulting in higher precision 3D coordinates of facial key points when using depth information for face depth estimation, thus improving the face recognition effect.

[0084] In practical applications, the accuracy and efficiency of facial landmark detection are crucial for many applications, such as face recognition, expression analysis, and face tracking. Specifically, in the above embodiment, as shown in Figure 2, step 101 includes the following steps:

[0085] 201. Use a pre-trained keypoint detection model to locate the 2D coordinates of facial keypoints in the face image of the user to be detected.

[0086] 202. Based on the position information of the 2D coordinates of the facial key points in the preset facial image, read the 3D depth information of the facial key points at the corresponding positions.

[0087] The pre-trained keypoint detection model can identify the 2D coordinates of facial keypoints in the face image of the user to be detected. This keypoint detection model can be trained using a regression model or a deep learning model. In practical applications, for the image of the user to be detected output by the camera device, the pre-trained face recognition model can be used to find image regions that may contain faces, and then keypoint detection can be performed on the image regions containing faces.

[0088] Since the pre-trained keypoint detection model learns the mapping relationship from face image to keypoint coordinates, after obtaining the face image of the user to be detected, the 2D coordinates of the face keypoints in the face image of the user to be detected can be predicted by the pre-trained keypoint detection model, and the face keypoints can be drawn on the face image according to the predicted 2D coordinates of the face keypoints to visualize the feature points of the face.

[0089] Here, the preset face image is a face image captured by a camera device with a depth imaging mechanism. The camera device can be a time-of-flight (TOF) based depth camera, which can emit pulse light towards the target and then determine the distance between the object and itself based on the flight time of the light beam between transmission and return. It can capture the depth information of pixels at different positions in the entire image while emitting pulse light, and then read the 3D depth information of the face key points at the corresponding positions based on the position information of the 2D coordinates of the face key points in the preset face image.

[0090] Furthermore, to reduce random errors that may occur during actual shooting, after locating the 2D coordinates of facial key points in the face image of the user being detected using a pre-trained key point detection model, intrinsic parameter calibration can be performed on the camera device with depth imaging mechanism. Based on the calibrated intrinsic parameter information, distortion correction processing can be applied to the 2D coordinates of the facial key points. Considering that camera shooting is a 3D to 2D process, which is described using a mathematical model, intrinsic parameter calibration involves calculating the parameters in the mathematical model. Through the calibrated intrinsic parameter information, a 3D world can be reconstructed from a 2D image.

[0091] Furthermore, in the above embodiment, as shown in FIG3, before step 102, the method further includes the following steps:

[0092] 301. Use a parametric model to generate a 3D face model with fusion properties.

[0093] 302. Based on the 3D structure of the face, extract the geometric information of the face contained in the 3D face model.

[0094] Among them, the 3D face model with fusion properties includes a 3D face structure obtained by weighting different face features. This 3D face model can be modeled using the 3DMM method. During the modeling process, the 3DMM method can parameterize the 3D structure of the face to obtain face geometric information including shape parameters and texture parameters. Using the face combination information, possible deformations and texture changes of the face can be simulated.

[0095] Specifically, in the process of 3D facial modeling, a large amount of 3D scan data or 2D image data of faces can be collected in advance. This data includes variations in different races, genders, ages, etc. The collected data is preprocessed, including image alignment, noise removal, and key point marking. For 3D scan data, point cloud registration and denoising may be required. Then, facial features are extracted from the preprocessed data. Common features include facial shape features (such as key point coordinates) and texture features (such as color information). Using the extracted feature data, a statistical model is established through methods such as principal component analysis. This model can describe the variation patterns of facial shape and texture. For new faces, the parameters in the statistical model are estimated through optimization algorithms. The estimated parameters are then used to reconstruct and deform the facial model through linear interpolation or nonlinear deformation methods, thereby constructing a 3D facial model.

[0096] Specifically, in the above embodiment, as shown in FIG4, step 102 includes the following steps:

[0097] 401. Input the 2D coordinates of the facial key points and the 3D depth information into the pre-constructed 3D facial model to determine the 3D coordinates of the current facial key points in the initial state.

[0098] 402. Based on the 3D coordinates of the current facial key points in the initial state, the facial pose parameters in the world coordinate system relative to the camera coordinate system are solved iteratively by the least squares method.

[0099] 403. Define a loss function for the regularization term based on the face pose parameters, and use the loss function for the regularization term to iteratively solve the weighting coefficients of the 3D face model.

[0100] Specifically, after inputting the 2D coordinates and 3D depth information of facial landmarks into a pre-built 3D face model, the 2D coordinates of the facial landmarks can be obtained using P... 2d It is stated that the 3D depth information of facial landmarks can be used with ToF. 3dThis indicates that the 3D face model contains the following facial geometric information: average face M, expression basis E, identity basis S. The initial 3D coordinates of the current facial keypoints are: P. 3dmm = Ss + Ee + M (e and s are updated during each iteration)

[0101] Specifically, a preset rule can be used to detect whether the 3D depth information of facial key points is within the available range. If so, the 3D coordinates of the current facial key points are added to the first input coordinate set, and the 3D depth information of the facial key points is added to the first output coordinate set. Otherwise, the 3D coordinates of the current facial key points are added to the second input coordinate set, and the 2D coordinates of the facial key points are added to the second output coordinate set. Correspondingly, in the process of iteratively solving the facial pose parameters in the world coordinate system relative to the camera coordinate system using the least squares method, when the first output coordinate set is not empty and the second output coordinate set is empty, the first input coordinate set and the first output coordinate set are used to define a loss function. When the second output coordinate set is not empty and the first output coordinate set is empty, the second input coordinate set and the second output coordinate set are used to define a loss function. The facial pose parameters in the world coordinate system relative to the camera coordinate system are further solved iteratively using the least squares method. When the value obtained by the loss function meets the iteration stopping condition, the facial pose parameters are output.

[0102] For example, based on the 3D coordinates of the current facial key points in the initial state, the facial pose parameters, i.e., the head pose R and T, are solved, and certain rules are set to check the ToF (Take-of-Face) value. 3d Availability, such as ToF 3d If the coordinates z are not within the range [z_min, z_max], then it is unusable; conversely, if it is usable, then P... 3dmm Add to X1, Tof 3d Add to Y1; if unavailable, then P. 3dmm Add to X2, P 2d Add to Y2.

[0103] Further, the pose R and T of the world coordinate system relative to the camera coordinate system are solved using the least squares method:

[0104] When Y1 is not empty and Y2 is empty, define the loss function as follows:

[0105] When Y2 is not empty and Y1 is empty, define the loss function as follows:

[0106] When Y1 is not empty and Y2 is empty, the above loss function is jointly optimized.

[0107] It should be noted that Proj is the calculation that projects the 3D coordinates of facial key points onto the phase plane. After distortion correction, Proj(X)3d )=(X x / X z ,X y / X z ).

[0108] Specifically, in the process of iteratively solving the weighted coefficients of the 3D face model using the loss function of the regularization term, the weighted coefficients of the 3D face model can be added to the loss function according to the face pose parameters. Define the loss function of the regularization term, and the weighted coefficients include the identity coefficient and the expression coefficient. Then, use the loss function of the regularization term to iteratively solve the identity coefficient and expression coefficient of the 3D face model respectively. When the value calculated by the loss function meets the iteration stopping condition, the weighted coefficients of the 3D face model are output according to the current identity coefficient and expression coefficient.

[0109] For example, after obtaining the head pose R and T, the identity coefficient s and expression coefficient e are further optimized by defining a loss function with regularization terms for e and s:

[0110] To solve for the expression coefficient e, we can fix s and let M″=R(Ss+M)+TE″=RE

[0111] After substituting into the loss function formula,

[0112] Convert to matrix form:

[0113] Furthermore, the analytical solution can be directly calculated using ridge regression:

[0114] To solve for the identity coefficient s, we can fix the expression coefficient e calculated in the previous step. The specific process is similar to that above. Let: m″=R(Ee+M)+TS″=RS

[0115] The analytical solution obtained after substituting the loss function is:

[0116] It should be noted that in practical applications, the identity coefficient is relatively fixed or changes little compared to the expression coefficient. In order to avoid instability in the final result, the historical values ​​of the identity coefficient can be cached and the average of the historical values ​​can be taken as the result of the current frame. When the number of frames stored in the cache exceeds the set value, there is no need to update the identity coefficient.

[0117] Furthermore, as a specific implementation of the method in Figures 1-4, this application embodiment provides a face recognition device based on depth information, as shown in Figure 5. The device includes: an acquisition unit 51, a solution unit 52, a fusion unit 53, and a recognition unit 54.

[0118] The acquisition unit 51 is used to acquire the 2D coordinates of the facial key points and the 3D depth information of the facial key points;

[0119] The solving unit 52 is used to input the 2D coordinates of the facial key points and the 3D depth information into the pre-constructed 3D face model, and solve the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model.

[0120] The fusion unit 53 is used to fuse the facial geometric information contained in the 3D face model according to the facial pose parameters and the weighting coefficients of the 3D face model to obtain the 3D coordinates of the facial key points.

[0121] The recognition unit 54 is used to perform face recognition using the 3D coordinates of the facial key points.

[0122] The face recognition device based on depth information provided in this application, compared with the current method of using a TOF camera to estimate the face and obtain the 3D coordinates of facial key points for face recognition, obtains the 2D coordinates and 3D depth information of facial key points, inputs the 2D coordinates and 3D depth information of facial key points into a pre-constructed 3D face model, solves the parameters through iterative optimization, obtains the face pose parameters and the weighting coefficients of the 3D face model, fuses the face geometric information contained in the 3D face model according to the face pose parameters and the weighting coefficients of the 3D face model, obtains the 3D coordinates of facial key points, and uses the 3D coordinates of facial key points for face recognition. The entire process uses the 3D depth information of facial key points to iteratively solve the face pose parameters and the weighting coefficients of the 3D face model in the pre-constructed 3D face model, which can produce more stable depth information during the iterative solution process. Therefore, when using depth information to estimate the face depth, higher precision 3D coordinates of facial key points are obtained, thus improving the face recognition effect.

[0123] In a specific application scenario, the acquisition unit 51 is specifically used to locate the 2D coordinates of facial key points in the face image of the user to be detected using a pre-trained key point detection model; and to read the 3D depth information of the facial key points at the corresponding positions based on the position information of the 2D coordinates of the facial key points in a preset face image, wherein the preset face image is a face image captured by a camera device with a depth imaging mechanism.

[0124] In specific application scenarios, the acquisition unit 51 is further used to perform intrinsic parameter calibration on a camera device with a depth imaging mechanism after the 2D coordinates of the facial key points are located in the face image of the user to be detected using a pre-trained key point detection model, and to perform distortion correction processing on the 2D coordinates of the facial key points based on the calibrated intrinsic parameter information.

[0125] In specific application scenarios, the device further includes:

[0126] The generation unit is used to generate a 3D face model with fusion properties using a parameterized model before inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model and solving the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model. The 3D face model includes a 3D face structure obtained by weighting different facial features.

[0127] The extraction unit is used to extract the facial geometric information contained in the 3D face model based on the 3D face structure.

[0128] In specific application scenarios, the solving unit 52 includes:

[0129] The determination module is used to input the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D facial model to determine the 3D coordinates of the current facial key points in the initial state;

[0130] The first solution module is used to solve the face pose parameters in the world coordinate system relative to the camera coordinate system by using the least squares method based on the 3D coordinates of the current face key points in the initial state.

[0131] The second solution module is used to define a loss function for the regularization term based on the face pose parameters, and to iteratively solve the weighting coefficients of the 3D face model using the loss function for the regularization term.

[0132] In specific application scenarios, the solving unit 52 further includes:

[0133] The detection module is used to detect whether the 3D depth information of the facial key points is within the usable range according to preset rules;

[0134] An addition module is used to add the 3D coordinates of the current facial key points to the first input coordinate set and add the 3D depth information of the facial key points to the first output coordinate set if the condition is met.

[0135] [According to Rule 91, Correction 26.12.2024] The adding module is also used to otherwise add the 3D coordinates of the current facial key points to the second input coordinate set and add the 2D coordinates of the facial key points to the second output coordinate set;

[0136] [Corrected according to Rule 91, 26.12.2024] Accordingly, the first solving module is specifically used to define a loss function using the first input coordinate set and the first output coordinate set when the first output coordinate set is not empty and the second output coordinate set is empty; and to define a loss function using the second input coordinate set and the second output coordinate set when the second output coordinate set is not empty and the first output coordinate set is empty; to solve the face pose parameters in the world coordinate system relative to the camera coordinate system by least squares iterative solution, and to output the face pose parameters when the value calculated by the loss function satisfies the iteration stopping condition.

[0137] [Corrected according to Rule 91, 26.12.2024] In a specific application scenario, the second solving module is specifically used to add the weighting coefficients of the 3D face model to the loss function according to the face pose parameters, define the loss function of the regularization term, and the weighting coefficients include the identity coefficient and the expression coefficient; use the loss function of the regularization term to iteratively solve the identity coefficient and expression coefficient of the 3D face model respectively, and when the value calculated by the loss function satisfies the iteration stopping condition, output the weighting coefficient of the 3D face model according to the current identity coefficient and expression coefficient.

[0138] [Correction 26.12.2024 according to Rule 91] Based on the methods shown in Figures 1-4 above, the present application also provides a storage medium storing a computer program that, when executed by a processor, implements the face recognition method based on depth information shown in Figures 1-4 above.

[0139] [Revised according to Rule 91, 26.12.2024] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods described in various implementation scenarios of this application.

[0140] [Correction 26.12.2024 according to Rule 91] Based on the methods shown in Figures 1-4 and the virtual device embodiment shown in Figure 5, in order to achieve the above objectives, this application embodiment also provides a physical device for face recognition based on depth information, which can be a computer, smartphone, tablet computer, smartwatch, server, or network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the face recognition method based on depth information shown in Figures 1-4.

[0141] [As amended to Rule 91, December 26, 2024] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optionally, a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface), etc.

[0142] [Correction 26.12.2024 based on Rule 91] In an exemplary embodiment, referring to FIG6, the above-described physical device includes a communication bus, a processor, a memory, and a communication interface, and may further include an input / output interface and a display device, wherein the various functional units can communicate with each other through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory to perform the depth information-based face recognition method in the above embodiment.

[0143] Those skilled in the art will understand that the physical device structure for face recognition based on depth information provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0144] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned depth-based facial recognition device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing device.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, compared with the existing methods, this application uses the 3D depth information of facial key points to iteratively solve the facial pose parameters and the weighting coefficients of the 3D facial model in a pre-constructed 3D facial model. This can produce more stable depth information during the iterative solution process, and thus obtain higher-precision 3D coordinates of facial key points when using the depth information for facial depth estimation, thereby improving the face recognition effect.

[0146] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0147] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A face recognition method based on depth information, wherein, include: Obtain the 2D coordinates and 3D depth information of facial key points; The 2D coordinates of the facial key points and the 3D depth information are input into a pre-constructed 3D face model. The parameters are solved through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model. The facial geometric information contained in the 3D face model is fused based on the facial pose parameters and the weighting coefficients of the 3D face model to obtain the 3D coordinates of the facial key points. Face recognition is performed using the 3D coordinates of the aforementioned facial key points.

2. The method according to claim 1, wherein, The acquisition of the 2D coordinates and 3D depth information of facial key points specifically includes: The 2D coordinates of facial key points in the face image of the user to be detected are located using a pre-trained key point detection model; Based on the position information of the 2D coordinates of the facial key points in the preset facial image, the 3D depth information of the facial key points at the corresponding positions is read. The preset facial image is a facial image captured by a camera device with a depth imaging mechanism.

3. The method according to claim 2, wherein, After locating the 2D coordinates of facial key points in the face image of the user to be detected using a pre-trained key point detection model, the method further includes: Intrinsic parameter calibration is performed on camera devices with depth imaging mechanisms, and distortion correction processing is performed on the 2D coordinates of the facial key points based on the calibrated intrinsic parameter information.

4. The method according to claim 1, wherein, Before inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model, and solving for the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model, the method further includes: A 3D face model with fusion properties is generated using a parametric model, wherein the 3D face model includes a 3D face structure obtained by weighting different face features; Based on the 3D structure of the face, extract the geometric information of the face contained in the 3D face model.

5. The method according to any one of claims 1-4, wherein, The process involves inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model, and solving for the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model. Specifically, this includes: The 2D coordinates of the facial key points and the 3D depth information are input into a pre-constructed 3D face model to determine the 3D coordinates of the current facial key points in the initial state. Based on the 3D coordinates of the current facial key points in the initial state, the facial pose parameters in the world coordinate system relative to the camera coordinate system are solved iteratively by the least squares method. The loss function of the regularization term is defined based on the face pose parameters, and the weighting coefficients of the 3D face model are iteratively solved using the loss function of the regularization term.

6. The method according to claim 5, wherein, After inputting the 2D coordinates of the facial key points and the 3D depth information into a pre-constructed 3D face model to determine the 3D coordinates of the current facial key points in the initial state, the method further includes: The system detects whether the 3D depth information of the facial key points is within the usable range using preset rules. If so, the 3D coordinates of the current facial key points are added to the first input coordinate set, and the 3D depth information of the facial key points is added to the first output coordinate set; Otherwise, add the 3D coordinates of the current facial key points to the second input coordinate set, and add the 2D coordinates of the facial key points to the second output coordinate set; Accordingly, the step of iteratively solving the face pose parameters in the world coordinate system relative to the camera coordinate system using the least squares method based on the 3D coordinates of the current facial key points in the initial state specifically includes: When the first output coordinate set is not empty and the second output coordinate set is empty, the loss function is defined using the first input coordinate set and the first output coordinate set; When the second output coordinate set is not empty and the first output coordinate set is empty, the loss function is defined using the second input coordinate set and the second output coordinate set; The face pose parameters in the world coordinate system relative to the camera coordinate system are solved iteratively using the least squares method. The face pose parameters are output when the value calculated by the loss function satisfies the iteration stopping condition.

7. The method according to claim 5, wherein, The step of defining a loss function with regularization terms based on the face pose parameters, and iteratively solving for the weighting coefficients of the 3D face model using the loss function with regularization terms, specifically includes: Based on the facial pose parameters, a weighting coefficient of the 3D face model is added to the loss function to define a regularization loss function, wherein the weighting coefficients include an identity coefficient and an expression coefficient. The loss function of the regularization term is used to iteratively solve the identity coefficient and expression coefficient of the 3D face model. When the value calculated by the loss function meets the iteration stopping condition, the weighted coefficient of the 3D face model is output according to the current identity coefficient and expression coefficient.

8. A face recognition device based on depth information, wherein, include: The acquisition unit is used to acquire the 2D coordinates of facial key points and the 3D depth information of facial key points; The solving unit is used to input the 2D coordinates of the facial key points and the 3D depth information into the pre-constructed 3D face model, and solve the parameters through iterative optimization to obtain the facial pose parameters and the weighting coefficients of the 3D face model. The fusion unit is used to fuse the facial geometric information contained in the 3D face model according to the facial pose parameters and the weighting coefficients of the 3D face model to obtain the 3D coordinates of the facial key points. The recognition unit is used to perform face recognition using the 3D coordinates of the facial key points.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images