Three-dimensional face model reconstruction method

Abstract

Embodiments of the present disclosure disclose a three-dimensional face model reconstruction method. A specific implementation of the method includes: generating each initial face key point, each initial face segmentation region, and an initial face normal; obtaining a rough fitting three-dimensional face model and shape parameters, pose parameters, expression parameters, and a model vertex coordinate matrix; obtaining depth feature information and pose feature information; obtaining fused feature information; generating a vertex offset matrix; obtaining a reconstructed three-dimensional face model, and generating each face key point, each face segmentation region, a face normal, and face depth data; generating a face model loss value; determining the reconstructed three-dimensional face model as a three-dimensional face model; and using the reconstructed three-dimensional face model as an initial three-dimensional face model. The implementation can perform nonlinear deformation on the three-dimensional model, thereby obtaining a three-dimensional face model that is more similar to the original picture.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, in particular to a three-dimensional face model reconstruction method. BACKGROUND

[0002] A face contains a person's main identity features and attributes, and is of great significance for the recognition of a person's identity. Therefore, there are many related researches on reconstructing a digital face image in the field of computer graphics and computer vision. Reconstructing a real and detailed face image can be used for facial biometric recognition, speech expression driving, face interaction, etc. At the same time, in many scenes such as movies, animations, games, virtual reality (VR) and augmented reality (AR), digital faces play a very important role. In addition, in the Virtual Standard Patient (VSP) project of the Virtual Reality Technology and System State Key Laboratory in cooperation with hospitals, there is an urgent need for accurate details and rich expressions of digital faces. Three-dimensional face reconstruction can provide a high-precision face model for a standard virtual patient. Such a model can be used to simulate the facial features of a real patient, including skin texture and wrinkles; three-dimensional face reconstruction can provide a more realistic appearance for a standard virtual patient; three-dimensional face reconstruction can provide more personalized and customized medical solutions for a standard virtual patient. Therefore, digital three-dimensional face reconstruction is of great significance.

[0003] Although two-dimensional faces can meet the needs in identity recognition, face detection, etc., three-dimensional face models are still needed in aspects such as face avatar, expression transfer, speech driving, etc. Traditional methods usually use large optical acquisition systems or large digital scanning devices based on multiple cameras. Although the acquired face models are relatively realistic, these devices are usually expensive, and it takes a lot of manpower and resources to build a large-scale acquisition system, which is difficult to popularize to ordinary consumers.

[0004] Therefore, with the continuous development of computer technology and neural networks, more and more works based on parametric models have achieved the reconstruction of complete three-dimensional face models from color videos or single-frame RGB pictures. However, due to the limited expressive ability of parametric models, it is difficult to well reconstruct fine poses and expressions. In recent years, with the advent of RGB-D cameras such as Kinect and RealSense, depth information has been provided as an additional geometric information for color videos. The present invention is committed to further combining RGB and depth data, using RGB-D monocular video to reconstruct a complete face head model, using depth information to optimize model details, using a deep neural network to deform the model, and finally obtaining a personalized face model with fine shape, pose and expression information that can be used for animation and face avatar. SUMMARY

[0005] The summary of the disclosure is partly to introduce the concepts in a brief form, which will be described in detail in the specific embodiments part. The summary of the disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] The technical problem solved by the present application is to propose a novel three-dimensional face reconstruction method based on RGBD data, which inputs depth data as an additional channel into the reconstruction process; and designs a depth encoder to encode the depth data, uses different fusion methods to integrate the depth feature vector into the network as the offset of the vertex of the conditional auxiliary model; finally, a depth-related loss function is designed to constrain the model vertex offset, and the entire training process is completed.

[0007] Some embodiments of the present disclosure propose a three-dimensional face model reconstruction method to solve one or more of the technical problems mentioned in the background section. Step (1): based on a pre-set face image, generate each initial face key point, each initial face segmentation region, and an initial face normal; Step (2): based on the above face image, perform fitting processing on the initial three-dimensional face model to obtain a rough fitting three-dimensional face model and the shape parameters, pose parameters, expression parameters and model vertex coordinate matrix corresponding to the rough fitting three-dimensional face model; Step (3): encode the pre-acquired initial depth data and the pose parameters corresponding to the rough fitting three-dimensional face model to obtain depth feature information and pose feature information; Step (4): fuse the above depth feature information and the above pose feature information to obtain fused feature information; Step (5): based on the above model vertex coordinate matrix and the above fused feature information, generate a vertex offset matrix corresponding to the above model vertex coordinate matrix; Step (6): superimpose the above vertex offset matrix to the rough fitting three-dimensional face model to obtain a reconstructed three-dimensional face model, and based on the reconstructed three-dimensional face model, generate each face key point, each face segmentation region, face normal and face depth data; Step (7): based on the above each initial face key point, the above each initial face segmentation region, the above initial face normal, the above initial depth data, the above each face key point, the above each face segmentation region, the above face normal and the above face depth data, generate a face model loss value; Step (8): in response to determining that the face model loss value meets a pre-set loss value condition, determining the reconstructed three-dimensional face model as a three-dimensional face model; Step (9): in response to determining that the face model loss value does not meet the loss value condition, using the reconstructed three-dimensional face model as an initial three-dimensional face model, and executing the above steps (2) to (9) again.

[0008] The above various embodiments of the present disclosure have the following beneficial effects: The beneficial positive effects of the present application are:

[0009] (1) The present application proposes a complete process scheme for obtaining a three-dimensional face model from a two-dimensional picture to solve the problem of low accuracy of existing three-dimensional face reconstruction methods. The idea of nonlinear deformation is used to obtain a three-dimensional face model with more geometric details. Compared with other methods, the hair, ears, nose, mouth and other fine geometric regions can be accurately represented.

[0010] (2) Some current reconstruction methods require a large amount of data set or accurate three-dimensional model, and take a lot of training time. The present application is committed to providing a simple and feasible scheme, which only uses RGBD data and sparse FLAME model as input, takes a shorter training time, and obtains a three-dimensional face model with accurate geometric details. It is proved that the depth data is effective in three-dimensional face reconstruction, and the balance between accuracy and performance is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, same or similar reference numerals can represent same or similar elements. It should be understood that the drawings are schematic, and elements and elements are not necessarily drawn to scale.

[0012] Figure 1 is a flowchart of some embodiments of the three-dimensional face model reconstruction method according to the present disclosure;

[0013] Figure 2 is a structure diagram of a pose encoder, a depth encoder and a vertex offset vector generation network;

[0014] Figure 3 is a schematic diagram of a rough fitting three-dimensional face model generated by a face image according to the three-dimensional face model reconstruction method of the present disclosure. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0016] In addition, it should be noted that only the parts related to the present application are shown in the drawings for ease of description. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0017] It should be noted that the terms "first", "second", and the like in the present disclosure are merely used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0018] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0019] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0020] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] Figure 1 Flow 100 of some embodiments of the three-dimensional face model reconstruction method according to the present disclosure is shown. The three-dimensional face model reconstruction method includes the following steps:

[0022] Step 101, based on a preset face image, generating each initial face key point, each initial face segmentation region and an initial face normal.

[0023] In some embodiments, the initial face key points, the initial face segmentation regions, and the initial face normal can be generated based on a preset face image. The face image can be an RGBD image corresponding to the face. The initial face key points can be labeled points obtained by labeling key positions in the face image. The key positions can include, but are not limited to, eyes, eyebrows, nose, ears, and mouth. The initial face segmentation regions can be image regions corresponding to the key positions in the face image. The initial face normal can be a normal map corresponding to the face image. In practice, the face key point detection model can be used to generate the initial face key points of the face image. The face key point detection model can be a neural network model taking the face image as input and outputting the initial face key points corresponding to the face image. For example, the face key point detection model can be a FAN (Face Alignment Network). In practice, first, a semantic segmentation method can be used to perform semantic segmentation on the face image to obtain a semantic segmentation face image as an initial semantic segmentation face image. The initial semantic segmentation face image contains image regions of different colors, and each pixel point in an image region of the same color corresponds to the same class. Then, each image region of different colors in the initial semantic segmentation face image can be determined as an initial face segmentation region. In practice, an image processing software can be used to generate the initial face normal corresponding to the face image. For example, the image processing software can be Blender.

[0024] At step 102, the initial three-dimensional face model is fitted based on the face image to obtain a rough fitting three-dimensional face model and shape parameters, pose parameters, expression parameters, and a model vertex coordinate matrix corresponding to the rough fitting three-dimensional face model.

[0025] In some embodiments, the initial three-dimensional face model can be fitted based on the face image to obtain a rough fitting three-dimensional face model and shape parameters, pose parameters, expression parameters and a model vertex coordinate matrix corresponding to the rough fitting three-dimensional face model. The initial three-dimensional face model can be a preset face model. For example, the initial three-dimensional face model can be a FLAME model with shape parameters, pose parameters and expression parameters all being 0. The rough fitting three-dimensional face model can be a face model obtained by rough fitting the initial three-dimensional face model based on the face image. The rough fitting three-dimensional face model can include various vertices. The shape parameters can be parameters for representing the basic shape of the face. The basic shape of the face can include, but is not limited to, any one of the following: face width, face length. The pose parameters can be parameters for representing the position and orientation of the face. For example, the pose parameters can be CN001, representing that the face is oriented 15 degrees to the left. The expression parameters can be parameters for representing the facial expression of the face. For example, the expression parameters can be CS0 10, representing that the corners of the mouth are raised by 10 degrees. The model vertex coordinate matrix can be a matrix in which each row corresponds to a vertex, each column corresponds to the three-dimensional coordinates of the vertex, and each element is a model vertex coordinate. The model vertex coordinates can be the three-dimensional coordinates corresponding to the vertices in the rough fitting three-dimensional face model. In practice, first, the shape parameters, pose parameters and expression parameters corresponding to the face image can be generated by a library function. Second, the parameters of the initial three-dimensional face model can be adjusted according to the shape parameters, pose parameters and expression parameters to obtain a rough fitting three-dimensional face model and a model vertex coordinate matrix corresponding to the rough fitting three-dimensional face model. The library function can be FaceTracker library.

[0026] In step 103, the initial depth data obtained in advance and the pose parameters corresponding to the rough fitting three-dimensional face model are encoded to obtain depth feature information and pose feature information.

[0027] In some embodiments, the initial depth data obtained in advance and the pose parameters corresponding to the rough fitting three-dimensional face model can be encoded to obtain depth feature information and pose feature information. The initial depth data can be a color depth map corresponding to the face image. The color depth map can be a depth map processed by an image processing function. The image processing function can be a color map function in OpenCV library. The depth feature information can be a feature vector obtained by encoding the initial depth data. The pose feature information can be a feature vector obtained by encoding the pose parameters.

[0028] In some optional implementations of some embodiments, the pre-acquired initial depth data and the posture parameters corresponding to the above-mentioned roughly fitted three-dimensional face model can be encoded to obtain depth feature information and posture feature information through the following steps:

[0029] The pre-acquired initial depth data and the posture parameters corresponding to the above-mentioned roughly fitted three-dimensional face model are encoded by the respective multi-layer perceptron to obtain the depth feature information and the posture feature information. The multi-layer perceptron includes:

[0030] P(θ 9*1 )→p 1536*1 ,

[0031] D(γ 512*512*1 )→d 1536*1 ;

[0032] P can be a multi-layer perceptron for encoding the above-mentioned posture parameters. P can be a multi-layer perceptron composed of respective linear layers and respective LeakyReLU functions, taking the posture parameters as input and outputting a feature vector corresponding to the posture parameters. θ 9*1 in θ can be the above-mentioned posture parameters, 9*1 can be the dimension of the above-mentioned posture parameters, and θ 9*1 can be posture parameters with a dimension of 9*1. P(θ 9*1 ) can represent the process of encoding the above-mentioned posture parameters θ 9*1 by the above-mentioned multi-layer perceptron P. p 1536*1 in p can be the above-mentioned posture feature information, 1536*1 can be the dimension of the above-mentioned posture feature information, and p 1536*1 can be posture feature information with a dimension of 1536*1. D can be a multi-layer perceptron for encoding the above-mentioned initial depth data. D can be a multi-layer perceptron composed of respective linear layers and respective LeakyReLU functions, taking the initial depth data as input and outputting a feature vector corresponding to the initial depth data. γ 512*512*1 in γ can be the above-mentioned initial depth data, 512*512*1 can be the dimension of the above-mentioned initial depth data, and γ 512*512*1 can be initial depth data with a dimension of 512*512*1. D(γ 512*512*1 ) can represent the process of encoding the above-mentioned initial depth data γ 512*512*1 by the above-mentioned multi-layer perceptron D. d 1536*1 in d can be the above-mentioned depth feature information, 1536*1 can be the dimension of the above-mentioned depth feature information, and d 1536 *1 can be depth feature information with a dimension of 1536*1.

[0033] In step 104, the depth feature information and the pose feature information are fused to obtain fused feature information.

[0034] In some embodiments, the depth feature information and the pose feature information can be fused to obtain fused feature information. The fused feature information can be a feature vector obtained by fusing the depth feature information and the pose feature information.

[0035] In some optional implementations of some embodiments, the depth feature information and the pose feature information can be fused by the following steps to obtain fused feature information:

[0036] The depth feature information and the pose feature information can be fused by the following formula to obtain fused feature information:

[0037]

[0038] In the formula, f can be the fused feature information, 1536*1 can be the dimension of the fused feature information, and f can be fused feature information with a dimension of 1536*1. 1536*1 In the formula, f can be the fused feature information, 1536*1 can be the dimension of the fused feature information, and f can be fused feature information with a dimension of 1536*1. 1536*1 In the formula, f can be the fused feature information, 1536*1 can be the dimension of the fused feature information, and f can be fused feature information with a dimension of 1536*1. The fusion manner of the depth feature information and the pose feature information can be represented by the formula. The fusion manner of the depth feature information and the pose feature information can be represented by the formula.

[0039] In step 105, a vertex offset matrix corresponding to the model vertex coordinate matrix is generated based on the model vertex coordinate matrix and the fused feature information.

[0040] In some embodiments, a vertex offset matrix corresponding to the model vertex coordinate matrix can be generated based on the model vertex coordinate matrix and the fused feature information. The vertex offset matrix can correspond to one vertex per row, offset of each vertex per column, and column matrix of vertex offset per element. The vertex offset can be the offset between each model vertex coordinate in the model vertex coordinate matrix and the fused feature information.

[0041] In some optional implementations of some embodiments, a vertex offset matrix corresponding to the model vertex coordinate matrix can be generated based on the model vertex coordinate matrix and the fused feature information by the following steps:

[0042] In a first step, the model vertex coordinate matrix is filled to obtain vertex embedding information corresponding to the model vertex coordinate matrix. The vertex embedding information can be a matrix obtained by filling each model vertex coordinate in the model vertex coordinate matrix. In practice, each model vertex coordinate in the model vertex coordinate matrix can be filled with a preset fill value to a preset number of digits to expand the dimension of the model vertex coordinate matrix to obtain the vertex embedding information. The preset fill value can be a preset value. The specific setting of the preset fill value is not limited. The preset number of digits can be a preset value. The specific setting of the preset number of digits is not limited. For example, when a model vertex coordinate in the model vertex coordinate matrix is (1, 0, 2), the preset fill value is 0, and the preset number of digits is 5, the element corresponding to the model vertex coordinate in the vertex embedding information obtained by filling the model vertex coordinate is (1, 0, 2, 0, 0).

[0043] In a second step, a vertex offset matrix corresponding to the model vertex coordinate matrix is generated based on the vertex embedding information and the fused feature information according to the following formula:

[0044] V o =T(G(f 1536*1 ,v 15135*35 ));

[0045] wherein V o may be the vertex offset matrix. T can be an activation function. For example, T can be a hyperbolic tangent function. G can be a vertex offset vector generation network. G can be a linear layer composed of a plurality of Feature-wise Linear Modulation (FiLM) layers, which takes the vertex embedding information and the fused feature information as input and outputs an offset matrix. The offset matrix can be a column matrix representing the offset between each model vertex coordinate in the model vertex coordinate matrix and the corresponding vertex of each vertex in the initial three-dimensional face model. The vertex offset matrix can be a column matrix obtained by inputting the offset matrix into the activation function T. v 15135*35 in v 15135*35 may be the vertex embedding information corresponding to the model vertex coordinate matrix, and the dimension of the vertex embedding information can be 15135*35.

[0046] In step 106, the vertex offset matrix is superimposed on the coarse fitting three-dimensional face model to obtain a reconstructed three-dimensional face model, and each face key point, each face segmentation region, face normal and face depth data are generated based on the reconstructed three-dimensional face model.

[0047] In some embodiments, the vertex offset matrix can be superimposed to the coarse fitting three-dimensional face model to obtain a reconstructed three-dimensional face model, and based on the reconstructed three-dimensional face model, each face key point, each face segmentation region, face normal and face depth data are generated. Wherein, the reconstructed three-dimensional face model can be a three-dimensional face model superimposed with the vertex offset matrix to the coarse fitting three-dimensional face model. The reconstructed three-dimensional face model can include each vertex. The face key point in each face key point can be a vertex in the reconstructed three-dimensional face model corresponding to each initial face key point.

[0048] The face segmentation region in each of the face segmentation regions can be a region in the reconstructed three-dimensional face model corresponding to the initial face segmentation region. The face normal can be a normal map corresponding to the front view of the reconstructed three-dimensional face model. The face normal can be determined by determining a face normal corresponding to each pixel in the face image region, and then combining the face normals corresponding to the pixels in the face image region into the face normal. The face depth data can be a color depth map corresponding to the front view of the reconstructed three-dimensional face model. The color depth map can be a depth map processed by an image processing function. The image processing function can be a color map function in an OpenCV library. In practice, each face key point of the reconstructed three-dimensional face model can be generated by the face key point detection model. The face key point detection model can be a neural network model that takes the front view of the reconstructed three-dimensional face model as input and outputs each face key point corresponding to the front view. For example, the face key point detection model can be a FAN (Face Alignment Network). In practice, the front view of the reconstructed three-dimensional face model can be processed by a semantic segmentation method to obtain a semantic segmentation front view. The semantic segmentation front view includes image regions of different colors, and each pixel in an image region of the same color corresponds to the same class. Each face segmentation region can be determined by each image region of a different color in the semantic segmentation front view. In practice, the face normal of the reconstructed three-dimensional face model can be generated by an image processing software. For example, the image processing software can be Blender. In practice, for each triangle region formed by every three vertices in the reconstructed three-dimensional face model, the image region corresponding to the triangle region in the face image can be determined as a face image region. Then, the face image region can be referenced to fill each pixel in the triangle region and generate a depth value for each pixel by a rasterization algorithm, so that the position and number of each pixel in the filled triangle region are the same as those in the face image region, and each pixel in the filled triangle region corresponds to a unique depth value. Finally, the face depth data can be obtained by combining the determined depth values according to the position of each pixel.

[0049] In some optional implementations of some embodiments, the vertex offset matrix can be superimposed on the rough fitting three-dimensional face model to obtain the reconstructed three-dimensional face model by the following steps:

[0050] The vertex offset matrix can be superimposed on the rough fitting three-dimensional face model to obtain the reconstructed three-dimensional face model by the following formula:

[0051] M = M FLAME(β, θ, φ) + V o ;

[0052] wherein M can be the reconstructed three-dimensional face model. M FLAME (β, θ, φ) can be a coarsely fitted three-dimensional face model with a shape parameter β, a pose parameter θ, and an expression parameter φ. β can be a shape parameter corresponding to the coarsely fitted three-dimensional face model. θ can be a pose parameter corresponding to the coarsely fitted three-dimensional face model. φ can be an expression parameter corresponding to the coarsely fitted three-dimensional face model. In practice, for each vertex offset in the vertex offset matrix, a sum of the vertex offset and a model vertex coordinate corresponding to the vertex offset in the model vertex coordinate matrix can be determined as a reconstructed model vertex coordinate. Then, positions of vertices in the coarsely fitted three-dimensional face model can be adjusted according to the determined reconstructed model vertex coordinates, to obtain the reconstructed three-dimensional face model.

[0053] At step 107, a face model loss value can be generated based on the initial face key points, the initial face segmentation regions, the initial face normal, the initial depth data, the face key points, the face segmentation regions, the face normal, and the face depth data.

[0054] In some embodiments, a face model loss value can be generated based on the initial face key points, the initial face segmentation regions, the initial face normal, the initial depth data, the face key points, the face segmentation regions, the face normal, and the face depth data. The face model loss value can be a numerical value representing a difference between the reconstructed three-dimensional face model and the face image.

[0055] In some optional implementations of some embodiments, a face model loss value can be generated based on the initial face key points, the initial face segmentation regions, the initial face normal, the initial depth data, the face key points, the face segmentation regions, the face normal, and the face depth data by:

[0056] In the first step, the face key points are projected to a preset two-dimensional plane to obtain face key point coordinates. The face key point coordinates can be the coordinates of the face key points in the two-dimensional plane. In practice, for each face key point, a preset coordinate value is added to the three-dimensional coordinates of the face key point to obtain four-dimensional coordinates. The preset coordinate value can be a preset numerical value. For example, the preset coordinate value can be 1. The camera extrinsic parameters in the pre-acquired camera parameters are multiplied by the four-dimensional coordinates to obtain camera coordinates of the face key point. The camera intrinsic parameters in the pre-acquired camera parameters are multiplied by the camera coordinates to obtain a three-dimensional coordinate. The three-dimensional coordinate includes a horizontal coordinate, a vertical coordinate, and a homogeneous coordinate. The ratio of the horizontal coordinate to the homogeneous coordinate is the horizontal coordinate of the face key point. The ratio of the vertical coordinate to the homogeneous coordinate is the vertical coordinate of the face key point. The horizontal coordinate and the vertical coordinate of the face key point are combined to obtain the face key point coordinates. The camera parameters can be the parameters of the camera used to capture the face image. The camera parameters can include camera intrinsic parameters and camera extrinsic parameters.

[0057] In the second step, based on the initial face key points and the face image, initial face key point coordinates are generated. The initial face key point coordinates can be two-dimensional coordinates of the initial face key points in the face image. In practice, the upper left corner of the first pixel in the face image is taken as the origin, the horizontal right direction is taken as the horizontal axis, and the vertical downward direction is taken as the vertical axis to construct a coordinate system as a face image coordinate system. The coordinates of the initial face key points in the face image coordinate system are determined as the initial face key point coordinates.

[0058] In the third step, based on the face key point coordinates and the initial face key point coordinates, a key point loss value is generated by the following formula:

[0059]

[0060] wherein, l lmk The key point loss value can be the loss value between the face key point coordinates and the initial face key point coordinates. i can be a serial number. The initial face key point coordinates can be the initial face key point coordinates, The i-th initial face key point coordinate can be the i-th initial face key point coordinate in the initial face key point coordinates. y can be a face key point coordinate, yi This can be the coordinates of the i-th facial landmark among the aforementioned facial landmark coordinates. With y i They correspond to each other. For example: when When the initial facial keypoint coordinates corresponding to the tip of the nose in the above face image are given, y i These are the coordinates of the facial landmarks corresponding to the tip of the nose in the reconstructed 3D face model described above. The 1 in the text can be a marker, representing the above. This is the formula used to generate the L1 loss function.

[0061] Fourth, using the following formula, generate the normal loss value based on the initial face normal and the face normal mentioned above:

[0062]

[0063] Among them, l normal The above-mentioned normal loss value can be the loss value between the initial face normal and the face normal. λ can be a preset normal processing function. For example, λ can be the Laplacian function. The initial face normal can be defined as described above. N can be any of the face normals described above. λ(N) can be the initial face normal result, that is, the initial face normal after processing by the above normal processing function. The 1 in the text can be a marker, representing the above. This is the formula used to generate the L1 loss function.

[0064] Fifth, based on the initial face segmentation regions and the aforementioned face segmentation regions, generate the segmentation region loss value using the following formula:

[0065]

[0066] Among them, l semantic The loss value can be the loss value of the segmented region mentioned above. This loss value can be the loss value between each initial face segmentation region and between each of the aforementioned face segmentation regions. k represents the sequence number. It can represent the initial face segmentation region. This can be the k-th initial face segmentation region among the aforementioned initial face segmentation regions. S can represent the face segmentation region. k It can be the k-th face segmentation region among the above face segmentation regions. The XOR can be performed. As an example, when the k is the neck region, the eye region, the ear region, and the hair region from 1 to 4, respectively, and when k is 1, is the neck region in each of the initial face segmentation regions. S k is the neck region in each of the face segmentation regions.

[0067] In the sixth step, a depth data loss value is generated based on the initial depth data and the face depth data by the following formula: wherein the initial depth data includes each initial depth value corresponding to each pixel point, and the face depth data includes each depth value corresponding to each pixel point.

[0068]

[0069] wherein l depth The depth data loss value can be the loss value between the initial depth data and the face depth data. n can be each pixel point. i can be the index of each pixel point. The depth value can be the depth value of the i-th pixel point. The depth value of the i-th pixel point can be the initial depth value. z i The initial depth value of the i-th pixel point can be the initial depth value. The The 1 in the above formula can be a label, which can represent the is a formula for generating an L1 loss function.

[0070] In the seventh step, a face model loss value is generated based on the key point loss value, the normal loss value, the segmentation region loss value, and the depth data loss value by the following formula:

[0071]

[0072] wherein l geo The face model loss value can be the index of the key point loss value, the normal loss value, the segmentation region loss value, and the depth data loss value. ω can be a weight coefficient. ω can be a value preset by a technician. Here, the specific setting of ω is not limited. l can represent the key point loss value, the normal loss value, the segmentation region loss value, and the depth data loss value, and ω i The weight coefficient corresponding to the loss value with index i can be ω. i The loss value with index i in the key point loss value, the normal loss value, the segmentation region loss value, and the depth data loss value can be l.

[0073] In step 108, in response to determining that the face model loss value meets the preset loss value condition, the reconstructed three-dimensional face model is determined as the three-dimensional face model.

[0074] In some embodiments, in response to determining that the face model loss value meets the preset loss value condition, the reconstructed three-dimensional face model can be determined as the three-dimensional face model. The loss value condition can be that the face model loss value is less than a preset loss threshold. The loss threshold can be a preset numerical value. The specific setting of the loss threshold is not limited.

[0075] In step 109, in response to determining that the face model loss value does not meet the loss value condition, the reconstructed three-dimensional face model is used as the initial three-dimensional face model, and steps 102-109 are executed again.

[0076] In some embodiments, in response to determining that the face model loss value does not meet the loss value condition, the reconstructed three-dimensional face model can be used as the initial three-dimensional face model, and steps 102-109 can be executed again.

[0077] In the process of using the technical solutions to solve the above technical problems, the following problems often occur:

[0078] When a large optical acquisition system or a large digital scanning device is used to generate a three-dimensional face model for face recognition of a user, the generated face model is more realistic, but the cost of generating the three-dimensional face model is high. When a user's face image is used for face recognition, the recognition may be incorrect due to factors such as angle, resulting in waste of computing resources during recognition.

[0079] In the face of the above technical problems, the following solutions are adopted:

[0080] Optionally, after step 109, the following steps can be performed:

[0081] First, the three-dimensional face model is stored in a preset database. The preset database can be a database that is preset to store three-dimensional face models.

[0082] In the second step, in response to receiving the user face image sent by the user terminal, key point recognition processing is performed on the user face image to obtain each face key point information. The user terminal can be a terminal device corresponding to the user. The user face image can be a face image of the user. The user face image corresponds to an image size. The image size can include an image width and an image height. The face key point information in each face key point information can be the coordinate position of the key point in the user face image in the user face image. In practice, first, the execution subject can input the user face image into a pre-trained key point generation model to obtain each face key point information. The key point generation model can be a machine learning model that takes a user face image as input and outputs each face key point information corresponding to the user face image. The key point generation model can be a pre-trained shape_predictor model. The pre-training can be training the shape_predictor model using a set of labeled face images and a cross-entropy loss function. The face images in the set of face images can be images of face parts.

[0083] In a third step, face pose information is generated based on the face key point information. The face pose information can be Euler angles corresponding to the face image of the user. The Euler angles include a pitch angle, a yaw angle, and a roll angle. In practice, first, the face key point information corresponding to the left eye, the right eye, the nose, the left corner of the mouth, and the right corner of the mouth is selected from the face key point information as the left eye key point coordinate, the right eye key point coordinate, the nose key point coordinate, the left corner of the mouth key point coordinate, and the right corner of the mouth key point coordinate. Second, the average of the vertical coordinates in the left eye key point coordinate and the vertical coordinates in the right eye key point coordinate is determined as an eye vertical coordinate. Then, the average of the vertical coordinates in the left corner of the mouth key point coordinate and the vertical coordinates in the right corner of the mouth key point coordinate is determined as a mouth corner vertical coordinate. Then, the absolute value of the difference between the vertical coordinates in the nose key point coordinate and the eye vertical coordinate is determined as an eye-nose distance. The absolute value of the difference between the vertical coordinates in the nose key point coordinate and the mouth corner vertical coordinate is determined as a nose-mouth distance. Then, the ratio of the image height included in the image size to a preset height value is determined as a height factor. The preset height value can be a non-zero numerical value. Here, the specific setting of the preset height value is not limited. Then, the ratio of a preset angle value to a reserved five decimal places of pi is determined as an angle factor. The preset angle value can be a non-zero numerical value. Here, the specific setting of the preset angle value is not limited. Then, the quotient of the difference between the eye-nose distance and the nose-mouth distance and the height factor is determined as a pitch factor. Then, the pitch factor is input into an inverse tangent function to obtain pitch data. The product of the pitch data and the angle factor is determined as a pitch angle. Then, the ratio of the image width included in the image size to a preset width value is determined as a width factor. The preset width value can be a non-zero numerical value. Here, the specific setting of the preset width value is not limited. Then, the difference between the horizontal coordinates in the nose key point coordinate and the width factor is determined as a nose offset data. The difference between the nose offset data and 1 is determined as a nose offset factor. Then, the nose offset factor is input into an inverse tangent function to obtain yaw data. The product of the yaw data and the angle factor is determined as a yaw angle. Then, the difference between the horizontal coordinates in the left eye key point coordinate and the horizontal coordinates in the right eye key point coordinate is determined as an eye horizontal coordinate data. The difference between the vertical coordinates in the left eye key point coordinate and the vertical coordinates in the right eye key point coordinate is determined as an eye vertical coordinate data. Then, the ratio of the eye horizontal coordinate data and the eye vertical coordinate data is input into an inverse tangent function to obtain roll data. The product of the roll data and the angle factor is determined as a roll angle. Finally, the pitch angle, the yaw angle, and the roll angle are combined as Euler angles as the face pose information.

[0084] In the fourth step, each historical three-dimensional face model is obtained. Each historical three-dimensional face model in the historical three-dimensional face models can be a pre-stored three-dimensional face model. Each historical three-dimensional face model in the historical three-dimensional face models corresponds to a plurality of vertices. Each vertex in the plurality of vertices corresponds to a three-dimensional coordinate. In practice, the execution subject can obtain each historical three-dimensional face model from the preset database.

[0085] In the fifth step, for each historical three-dimensional face model in the historical three-dimensional face models, the following steps are performed:

[0086] In the first sub-step, based on the face posture information, the historical three-dimensional face model is adjusted in posture to obtain a historical three-dimensional face model adjusted in posture as a three-dimensional face model to be processed. In practice, the execution subject can input the Euler angle and the preset rotation information into a rotation function to obtain a plurality of rotation matrices. The rotation function can be eul2rotm. The rotation information can be information for representing a rotation order. For example, the rotation information can be "Z-Y-X", which represents that the rotation function first generates a rotation matrix of a yaw angle, then generates a rotation matrix of a pitch angle, and finally generates a rotation matrix of a roll angle. Then, the plurality of rotation matrices can be multiplied in the order corresponding to the rotation information to obtain a transformation matrix. Then, the historical three-dimensional face model can be uploaded to a three-dimensional model software. The three-dimensional model software can be software capable of adjusting the vertices of a three-dimensional model. For example, the three-dimensional model software can be MATLAB. Then, for each vertex of the vertices corresponding to the historical three-dimensional face model, the three-dimensional coordinate corresponding to the vertex and the product of the transformation matrix are determined as an adjustment coordinate. Then, the adjustment coordinate can be input into a modification function to modify the position of the vertex. In this way, until the vertices in the historical three-dimensional face model are adjusted, the historical three-dimensional face model adjusted can be determined as the three-dimensional face model to be processed. The modification function can be a function capable of adjusting the position of a vertex in a model. For example, the modification function can be a patch function.

[0087] In the second sub-step, based on the three-dimensional face model to be processed, a two-dimensional image frame corresponding to the three-dimensional face model to be processed is generated as a model image. In practice, first, the execution subject can display the three-dimensional face model to be processed. Then, the execution subject can capture the display of the three-dimensional face model to be processed as a two-dimensional image frame by a capture function to obtain a two-dimensional image frame corresponding to the three-dimensional face model to be processed as a model image. The capture function can be a getframe function.

[0088] A third sub-step is to compare the similarity of the model image and the user face image to obtain a similarity comparison result. The similarity comparison result can be a numerical value representing the similarity between the model image and the user face image. In practice, the execution subject can compare the similarity of the model image and the user face image by using an image comparison algorithm to obtain the similarity comparison result. The image comparison algorithm can be a mean hash algorithm.

[0089] A sixth step is to control the corresponding driving device to perform a preset door opening task in response to determining that the obtained similarity comparison results meet a preset similarity comparison result condition. The preset similarity comparison result condition can be that one of the similarity comparison results is less than a preset similarity comparison result threshold. The similarity comparison result threshold can be a preset numerical value, and the specific setting of the similarity comparison result threshold is not limited. The driving device can be a device installed on a door shaft that can drive the door body to open or close upon receiving a corresponding instruction. For example, the driving device can be an automatic door controller. In practice, the execution subject can send a preset door opening instruction to the driving device to perform the door opening task. The door opening instruction can be an instruction for requesting the target terminal to perform a door opening operation.

[0090] The technical scheme and related content thereof combined with steps 101 to 109 are an inventive point of an embodiment of the present disclosure, which solves the problem of waste of computing resources. The factors that cause high cost consumption when generating a three-dimensional face model when using a large optical acquisition system or a large digital scanning device to generate a three-dimensional face model for face recognition of a user, although the acquired face model is more realistic, but the cost consumed when generating a three-dimensional face model is high. When a user's face image is used for face recognition, it is easy to cause recognition errors due to factors such as angle, resulting in waste of computing resources when recognizing. If the above factors are solved, the waste of computing resources can be reduced. In order to achieve this effect, the present disclosure first stores the above three-dimensional face model to a preset database. Secondly, in response to receiving a user face image sent by a user terminal, the user face image is subjected to key point recognition processing to obtain individual face key point information. Thus, the key point information on the user face image can be identified. Then, based on the individual face key point information, face posture information is generated. Thus, the user's face posture information can be generated through the individual face key point information of the user. Then, each historical three-dimensional face model is obtained. Thus, each three-dimensional face model corresponding to each user pre-stored can be obtained. Then, for each historical three-dimensional face model in the above each historical three-dimensional face model, the following steps are performed: first, based on the face posture information, the historical three-dimensional face model is adjusted in posture to obtain a posture-adjusted historical three-dimensional face model as a three-dimensional face model to be processed. Thus, the three-dimensional face model corresponding to the user can be adjusted to the posture corresponding to the user face image. Secondly, based on the three-dimensional face model to be processed, a two-dimensional image frame corresponding to the three-dimensional face model to be processed is generated as a model image. Thus, an image corresponding to the three-dimensional face model to be processed can be obtained. Then, the model image and the user face image are subjected to image similarity comparison to obtain a similarity comparison result. Thus, the image corresponding to the three-dimensional face model to be processed can be subjected to similarity comparison with the user face image to obtain a similarity comparison result. Finally, in response to determining that each similarity comparison result obtained satisfies a preset similarity comparison result condition, a corresponding driving device is controlled to perform a preset door opening task. Thus, whether the face recognition is successful can be determined through each similarity comparison result obtained, and after successful recognition, the driving device is controlled to perform the door opening task. Because the three-dimensional face model corresponding to the user can be generated through steps 101 to 109 without using a large optical acquisition system or a large digital scanning device, the cost of constructing and maintaining a large optical acquisition system or a large digital scanning device can be reduced.In addition, the three-dimensional face model can be adjusted according to the angle identified by the user, and then the three-dimensional face model is used for face recognition, so that the probability of recognition error caused by the angle factor can be reduced, and the waste of computing resources during recognition can be reduced.

[0091] Figure 2 is a structural diagram of a pose encoder, a depth encoder and a vertex offset vector generation network. The pose parameter is the pose parameter described above. The pose encoder is a multi-layer perceptron for encoding the pose parameter. The "LeakyReLU" in the pose encoder is a LeakyReLU function. The depth data is the initial depth data described above. The depth encoder is a multi-layer perceptron for encoding the initial depth data described above. The "LeakyReLU" in the depth encoder is a LeakyReLU function. represents a process of fusing the pose feature information output by the pose encoder and the depth feature information output by the depth encoder to obtain the fusion feature information. The model vertex vector is the vertex embedding information described above. Figure 2 Each "linear FILM layer" in is the vertex offset vector generation network described above. "TanH" is a TanH activation function. The vertex offset is the vertex offset matrix described above.

[0092] Figure 3 is a schematic diagram of a rough fitting three-dimensional face model generated by a face image according to the three-dimensional face model reconstruction method of the present disclosure. Figure 3 In, each image in the odd column corresponds to each face image. Each image in the even column can represent a rough fitting three-dimensional face model corresponding to the face image on the left of each image. The collection, storage, use, etc. of user personal information (such as face images and images of rough fitting three-dimensional face models) involved in the present disclosure shall be performed by the relevant organization or individual to fulfill the obligations including conducting personal information security impact assessment, fulfilling the notification obligations to the personal information subject, and obtaining the prior authorization consent of the personal information subject, etc. before performing the corresponding operation.

[0093] The above description is only some of the preferred embodiments of the present disclosure and the explanation of the technical principles applied. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features and the technical features disclosed in the embodiments of the present disclosure (but not limited to) with similar functions are replaced with each other to form a technical solution.

Claims

1. A three-dimensional face model reconstruction method, comprising: Step (1): based on a preset face image, generating each initial face key point, each initial face segmentation region and an initial face normal; Step (2): based on the face image, performing fitting processing on an initial three-dimensional face model to obtain a rough fitting three-dimensional face model and shape parameters, pose parameters, expression parameters and a model vertex coordinate matrix corresponding to the rough fitting three-dimensional face model; Step (3): performing encoding processing on the pre-acquired initial depth data and the pose parameters corresponding to the rough fitting three-dimensional face model to obtain depth feature information and pose feature information; Step (4): performing fusion processing on the depth feature information and the pose feature information to obtain fusion feature information; Step (5): based on the model vertex coordinate matrix and the fusion feature information, generating a vertex offset matrix corresponding to the model vertex coordinate matrix; Step (6): superimposing the vertex offset matrix to the rough fitting three-dimensional face model to obtain a reconstructed three-dimensional face model, and based on the reconstructed three-dimensional face model, generating each face key point, each face segmentation region, a face normal and face depth data; Step (7): based on the each initial face key point, the each initial face segmentation region, the initial face normal, the initial depth data, the each face key point, the each face segmentation region, the face normal and the face depth data, generating a face model loss value; Step (8): in response to determining that the face model loss value meets a preset loss value condition, determining the reconstructed three-dimensional face model as a three-dimensional face model; Step (9): in response to determining that the face model loss value does not meet the loss value condition, using the reconstructed three-dimensional face model as an initial three-dimensional face model, and executing the steps (2) to (9) again.

2. The method of claim 1, wherein, The encoding processing on the pre-acquired initial depth data and the pose parameters corresponding to the rough fitting three-dimensional face model to obtain the depth feature information and the pose feature information comprises: encoding processing on the pre-acquired initial depth data and the pose parameters corresponding to the rough fitting three-dimensional face model by each multi-layer perception to obtain the depth feature information and the pose feature information, wherein the multi-layer perception comprises: P(θ 9*1 )→p 1536*1 D(γ 512*512*1 )→d 1536*1 ; Wherein, P is a multi-layer perception machine for encoding the posture parameter, θ 9*1 in the formula is the posture parameter, 9*1 is the dimension of the posture parameter, θ 9*1 is a posture parameter with a dimension of 9*1, P(θ 9*1 ) represents a process of encoding the posture parameter θ 9*1 by the multi-layer perception machine P, p 1536*1 in the formula is the posture feature information, 1536*1 is the dimension of the posture feature information, p 1536*1 is posture feature information with a dimension of 1536*1, D is a multi-layer perception machine for encoding the initial depth data, γ 512*512*1 in the formula is the initial depth data, 512*512*1 is the dimension of the initial depth data, γ 512*512*1 is initial depth data with a dimension of 512*512*1, D(γ 512*512*1 ) represents a process of encoding the initial depth data γ 512*512*1 by the multi-layer perception machine D, d 1536*1 in the formula is the depth feature information, 1536*1 is the dimension of the depth feature information, d 1536*1 is depth feature information with a dimension of 1536*1.

3. The method of claim 2, wherein, The fusion processing on the depth feature information and the pose feature information to obtain the fusion feature information comprises: fusion processing on the depth feature information and the pose feature information by the following formula to obtain the fusion feature information: wherein f 1536*1 f is the fusion feature information, 1536*1 is the dimension of the fusion feature information, f 1536*1 is the fusion feature information with the dimension of 1536*1, represents the fusion manner of the deep feature information and the pose feature information.

4. The method of claim 1, wherein, The generation of the vertex offset matrix corresponding to the model vertex coordinate matrix based on the model vertex coordinate matrix and the fusion feature information comprises: performing padding processing on the model vertex coordinate matrix to obtain vertex embedding information corresponding to the model vertex coordinate matrix; generating the vertex offset matrix corresponding to the model vertex coordinate matrix based on the vertex embedding information and the fusion feature information by the following formula: V o = T(G(f 1536*1 , v 15135*35 )); wherein V o is the vertex offset matrix, T is an activation function, G is a vertex offset vector generation network, v 15135*35 V in the above equation is vertex embedding information corresponding to the model vertex coordinate matrix, 15135*35 is the dimension of the vertex embedding information, v 15135*35 is vertex embedding information with a dimension of 15135*35.

5. The method of claim 4, wherein, The superimposition of the vertex offset matrix to the rough fitting three-dimensional face model to obtain the reconstructed three-dimensional face model comprises: The vertex offset matrix is superimposed to the coarse fitting three-dimensional face model by the following formula to obtain a reconstructed three-dimensional face model: M = M FLAME (β, θ, φ) + V o ; wherein M is the reconstructed three-dimensional face model, M FLAME (β, θ, φ) is the coarse-fitted three-dimensional face model, β is a shape parameter corresponding to the coarse-fitted three-dimensional face model, θ is a pose parameter corresponding to the coarse-fitted three-dimensional face model, and φ is an expression parameter corresponding to the coarse-fitted three-dimensional face model.

6. The method of claim 1, wherein, The face model loss value is generated based on the initial face key points, the initial face segmentation regions, the initial face normal, the initial depth data, the face key points, the face segmentation regions, the face normal and the face depth data, and includes: The face key points are projected to a preset two-dimensional plane to obtain face key point coordinates; Initial face key point coordinates are generated based on the initial face key points and the face image: A key point loss value is generated based on the face key point coordinates and the initial face key point coordinates by the following formula: wherein, l lmk is the key point loss value, i is a serial number, is an initial face key point coordinate, is the i-th initial face key point coordinate in the plurality of initial face key point coordinates, y is a face key point coordinate, and y i is the i-th face key point coordinate in the plurality of face key point coordinates, and the 1 in the above formula is a mark, representing the is a formula for generating an L1 loss function. A normal loss value is generated based on the initial face normal and the face normal by the following formula: wherein, l normal is the normal loss value, λ is a preset normal processing function, is the initial face normal, N is the face normal, is an initial face normal result obtained by processing the initial face normal through the normal processing function, λ(N) is a face normal result obtained by processing the face normal through the normal processing function, and the 1 in the above formula is a mark, indicating that the is a formula for generating an L1 loss function; A segmentation region loss value is generated based on the initial face segmentation regions and the face segmentation regions by the following formula: wherein, l semantic is the segmentation area loss value, k is a serial number, is an initial face segmentation area, is the kth initial face segmentation area in the initial face segmentation areas, S represents a face segmentation area, S k is the kth face segmentation area in the face segmentation areas, is XOR; A depth data loss value is generated based on the initial depth data and the face depth data by the following formula, wherein the initial depth data includes initial depth values corresponding to each pixel point, and the face depth data includes depth values corresponding to each pixel point: wherein, l depth is the depth data loss value, n is each pixel point, i is the index of the each pixel point, is a depth value, is a depth value corresponding to the i-th pixel point, z represents an initial depth value, z i is an initial depth value corresponding to the i-th pixel point, the 1 in the above formula is a mark, representing the is a formula for generating an L1 loss function; A face model loss value is generated based on the key point loss value, the normal loss value, the segmentation region loss value and the depth data loss value by the following formula: wherein, l geo is the face model loss value, i is an index of the key point loss value, the normal loss value, the segmentation region loss value and the depth data loss value, ω is a weight coefficient, l represents the key point loss value, the normal loss value, the segmentation region loss value and the depth data loss value, and ω i is a weight coefficient corresponding to the loss value with index i, l i is the loss value with index i in the key point loss value, the normal loss value, the segmentation region loss value and the depth data loss value.

Images