A method and system for stylized face 3D shape modeling, device, storage medium

Abstract

The application discloses a style face three-dimensional shape modeling method and system, equipment and storage medium, and belongs to the visual technical field, and comprises the following steps: training a 3DMM fitting model by using real face images; inputting a style face image into the trained 3DMM fitting model to obtain an initial triangular mesh matched with a face shape; and performing Laplace deformation on the triangular mesh by taking face key points in the style face image as targets to obtain a final triangular mesh matched with the style face shape. The 3DMM of the real face is subjected to the Laplace deformation, and the face three-dimensional shape can be automatically and efficiently reconstructed from the style face image.

Description

Technical Field

[0001] This invention belongs to the field of visual technology, and specifically relates to a stylized three-dimensional face shape modeling method, system, device, and storage medium. Background Technology

[0002] 3D facial models are fundamental to creating much digital content, such as movies and animations, and modeling the 3D shape of the face is a crucial step in this process. Furthermore, many real-world applications require stylized 3D facial models; for example, creating cartoonish, large-eyed Disney characters requires correspondingly exaggerated 3D facial models. However, current 3D modeling of such stylized faces primarily relies on manual work by professional modelers, lacking an automated and efficient method. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides a stylized 3D face shape modeling method, system, device, and storage medium, the method comprising:

[0004] The 3DMM fitting model was trained using real face images;

[0005] The stylized face image is input into the trained 3DMM fitting model to obtain an initial triangular mesh that matches the face shape;

[0006] Using the facial key points in the stylized face image as targets, perform Laplacian deformation on the initial triangular mesh to obtain a final triangular mesh that accurately matches the face shape.

[0007] Using facial key points in the stylized face image as targets, perform Laplacian deformation on the initial triangular mesh to obtain a final triangular mesh that accurately matches the face shape, including:

[0008] The objective function formula for the final triangular mesh is:

[0009] ;

[0010] In the formula: This represents the deviation between the Laplace coordinates of the vertices of the triangular mesh before and after deformation; α is the deviation between the projected position of the key point on the deformed triangular mesh on the stylized face image and the corresponding key point position detected on the stylized face image; α is a parameter.

[0011] Preferably, The formula is:

[0012] , ;

[0013] The The linear function expressed as P is :

[0014] ;

[0015] ;

[0016] In the formula: l represents the number of key points, The two-dimensional image coordinates representing key points detected on a stylized face image. P represents the 3D spatial coordinates of the key points of the associated triangular mesh; P is the set of 3D spatial coordinates of the vertices of the deformed triangular mesh; Π κ This is a perspective projection operation based on the camera intrinsic parameter matrix κ.

[0017] Preferably, The formula is:

[0018] ;

[0019] ;

[0020] In the formula: For the vertices v of the deformed triangular mesh i Laplace coordinates, For the vertex v of the triangle mesh before deformation i The Laplace coordinates of ; E is the set of edges of the triangular mesh.

[0021] Preferably, the triangular mesh include:

[0022] ;

[0023] V = {v1, v2, ..., v} n};

[0024] E = {e1, e2, ..., e} k}, e i ∈V×V;

[0025] F = {f1, f2, ..., f} m}, t i ∈V×V×V;

[0026] ;

[0027] In the formula: v i ∈V、e i ∈E and f i∈F represent the vertices, edges, and triangles in the triangle mesh, respectively. The indices are given by n, k, and m, where n, k, and m are the number of vertices, edges, and triangles within the triangle mesh, respectively. For the vertex v of the triangle mesh i The three-dimensional spatial coordinates.

[0028] Preferably, the stylized face image is input into the trained 3DMM fitting model to obtain an initial triangular mesh, as well as the camera intrinsic and extrinsic parameter matrices.

[0029] Preferably, the initial triangular mesh is optimized by Laplace deformation, and the optimization steps include:

[0030] Detect the coordinates of facial key points in the stylized face image;

[0031] The intrinsic parameter matrix κ is optimized using the following formula: ;

[0032] The extrinsic parameter matrix [R, t] is optimized using the following formula: ;

[0033] The initial triangular mesh vertex coordinate set is optimized using the following formula: ;

[0034] Repeat the above optimization steps until the number of iterations reaches the set limit.

[0035] This invention provides a system for stylized 3D face shape modeling, comprising:

[0036] The training module is used to train the 3DMM fitting model using real human face images;

[0037] The output module is used to input the stylized face image into the trained 3DMM fitting model to obtain an initial triangular mesh that matches the face shape.

[0038] The deformation module is used to perform Laplacian deformation on the initial triangular mesh with the facial key points in the stylized face image as the target, so as to obtain the final triangular mesh that accurately matches the face shape.

[0039] An embodiment of the present invention provides an apparatus comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit performs the method described above.

[0040] An embodiment of the present invention provides a computer-readable storage medium storing a computer program executable by a device, which, when run on the device, causes the device to perform the method described above.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] This invention applies Laplacian deformation to a 3DMM of a real human face, enabling automatic and efficient reconstruction of the three-dimensional shape of a face from a stylized face image. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the stylized three-dimensional face shape modeling method provided in the embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0047] Reference Figure 1 A stylized method, system, device, and storage medium for 3D face shape modeling, the method comprising:

[0048] The 3DMM fitting model was trained using real face images;

[0049] Specifically, training the 3DMM fitting model is a current technique, which can be found in the paper "Accurate 3DFaceReconstruction with Weakly-Supervised Learning: From Single Image to ImageSet". After training, when a new real face image is input, we can obtain a triangular mesh that matches the shape of the face in the image. Camera intrinsic parameter matrix and extrinsic parameter matrix ( Represents the rotation matrix. (Represents the translation vector).

[0050] Stylized face images are input into a trained 3DMM fitting model to obtain an initial triangular mesh that matches the face shape;

[0051] Specifically, the stylized face image is input into the trained 3DMM fitting model to predict the initial triangular mesh. Camera intrinsic parameter matrix κ 0 and extrinsic parameter matrix [R 0 t 0 (The superscript of the variable in the formula indicates the corresponding state, where "0" is the initial state). Since the 3DMM fitting model is trained on real face image data, its prediction results on stylized face images are not accurate. However, experimental verification shows that the triangular mesh predicted by the 3DMM fitting model achieves a good degree of matching with the overall face shape in the stylized face image. Therefore, the prediction results of the 3DMM fitting model are used as the initial stylized face 3D shape model, and further optimization is carried out based on this.

[0052] The initial triangular mesh is subjected to Laplacian deformation based on the facial key points in the stylized face image to obtain the final triangular mesh that accurately matches the face shape.

[0053] Specifically, the initial triangular mesh While the overall face shape matches well with the stylized face image, there are still some discrepancies in terms of facial feature shapes. Therefore, this paper proposes a Laplacian deformation of a triangular mesh to fit a sparse set of facial keypoints in the stylized face image. This deformed mesh retains local detail features while better matching the face shape in the stylized face image. This is a typical optimization problem, where the variable is the set of 3D spatial coordinates P of the triangular mesh vertices, and the objective function includes... and Two energy terms.

[0054] The deviation between the projected positions of keypoints on the triangular mesh onto the stylized face image and the corresponding keypoint positions detected on the stylized face image is measured by the following formula:

[0055] ,

[0056] Where l represents the number of key points, The two-dimensional image coordinates representing key points detected on a stylized face image. Π represents the three-dimensional spatial coordinates of the associated triangular mesh key points. κ This represents the perspective projection operation based on the camera intrinsic parameter matrix κ. It is a nonlinear function of the set of vertex coordinates P of the triangular mesh, which is relatively difficult to optimize. Therefore, we will... A further approximation is a linear function of P. :

[0057]

[0058]

[0059] The deviation between the Laplacian coordinates of the vertices of the triangular mesh before and after deformation was calculated to measure the degree of preservation of local details of the triangular mesh after deformation. The formula is as follows:

[0060]

[0061]

[0062] in, For the vertices v of the deformed triangular mesh i Laplace coordinates, For the vertex v of the triangle mesh before deformation i The Laplace coordinates.

[0063] The objective function for the final triangular mesh is:

[0064]

[0065] in, Used to adjust and The two energy terms in the objective function The weights in the equation.

[0066] Furthermore, triangular mesh Represents the three-dimensional shape of a human face. Triangular mesh. A triangular mesh is formed by connecting a set of triangles in three-dimensional space through shared edges or vertices. Mathematically, it is a triple representing the mesh topology—(V, E, F)—consisting of a set of vertices V, a set of edges E, and a set of triangles F, and a set of three-dimensional coordinates P of the vertices. include:

[0067]

[0068] V = {v1, v2, ..., v} n};

[0069] E = {e1, e2, ..., e} k}, e i ∈V×V;

[0070] F = {f1, f2, ..., f} m}, t i ∈V×V×V;

[0071] ;

[0072] In the formula: v i ∈V、e i ∈E and f i ∈F represent the vertices, edges, and triangles in the triangle mesh, respectively. The indices are given by n, k, and m, where n, k, and m are the number of vertices, edges, and triangles within the triangle mesh, respectively. For the vertex v of the triangle mesh i The three-dimensional spatial coordinates.

[0073] Furthermore, due to the initial camera intrinsic parameter matrix κ 0 and extrinsic parameter matrix [R 0 t 0 The initial triangular mesh is not accurate enough, therefore these two matrices need to be optimized before optimizing the initial triangular mesh. The steps for optimizing the initial triangular mesh include:

[0074] The coordinates of facial landmarks in stylized face images are detected. The keypoint detection technique can be found in the paper "RobustFace Alignment via Deep Progressive Reinitialization and Adaptive Error-Driven Learning".

[0075] The optimized intrinsic parameter matrix κ is given by the following formula: ;

[0076] The optimization formula for the extrinsic parameter matrix [R, t] is as follows: For techniques on optimizing intrinsic and extrinsic parameter matrices, please refer to the paper "Adam: A Method for Stochastic Optimization";

[0077] The optimization formula for the vertex coordinate set of the initial triangular mesh is as follows: It was solved using the Cholesky decomposition method;

[0078] Repeat the above optimization steps until the number of iterations reaches the set limit.

[0079] This invention provides a system for stylized 3D face shape modeling, comprising:

[0080] The training module is used to train the 3DMM fitting model using real human face images;

[0081] The output module is used to input stylized face images into a trained 3DMM fitting model to obtain an initial triangular mesh that matches the face shape.

[0082] The deformation module is used to perform Laplacian deformation on the initial triangular mesh based on the facial key points in the stylized face image, so as to obtain the final triangular mesh that accurately matches the face shape.

[0083] An embodiment of the present invention provides an apparatus including at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit performs the above-described method.

[0084] The present invention provides a computer-readable storage medium storing a computer program executable by a device, which, when run on the device, causes the device to perform the above-described method.

[0085] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A stylized method for modeling the 3D shape of a human face, characterized in that, include: The 3DMM fitting model was trained using real face images; The stylized face image is input into the trained 3DMM fitting model to obtain an initial triangular mesh that matches the face shape; Using facial key points in the stylized face image as targets, perform Laplacian deformation on the initial triangular mesh to obtain a final triangular mesh that accurately matches the face shape; specifically including: The objective function formula for the final triangular mesh is: ； In the formula: This represents the deviation between the Laplace coordinates of the vertices of the triangular mesh before and after deformation; The deviation between the projection position of the key point on the deformed triangular mesh onto the stylized face image and the corresponding key point position detected on the stylized face image; For parameters; The formula is: ， ； The Expressed as The linear function is : ； ； In the formula: Indicates the number of key points. The two-dimensional image coordinates representing key points detected on a stylized face image. Represents the three-dimensional spatial coordinates of the associated triangular mesh key points; This is the set of three-dimensional spatial coordinates of the vertices of the deformed triangular mesh; For camera intrinsic parameter matrix Perspective projection operation, This is the extrinsic parameter matrix; The formula is: ； ； In the formula: Vertices of the deformed triangular mesh Laplace coordinates, Vertices of the triangular mesh before deformation Laplace coordinates; For the set of edges of a triangular mesh, For the vertices of the triangular mesh The three-dimensional spatial coordinates.

2. The stylized 3D face shape modeling method as described in claim 1, characterized in that, The triangular mesh include: ； ； ； ； ； In the formula: , and Vertices, edges, and triangles in a triangle mesh, respectively. In-line index, , and These represent the number of vertices, edges, and triangles within the triangular mesh.

3. The stylized 3D face shape modeling method as described in claim 2, characterized in that, The stylized face image is input into the trained 3DMM fitting model to obtain an initial triangular mesh, as well as the camera intrinsic and extrinsic parameter matrices.

4. The stylized 3D face shape modeling method as described in claim 3, characterized in that, The initial triangular mesh is optimized by performing Laplace deformation. The optimization steps include: Detect the coordinates of facial key points in the stylized face image; Optimize the intrinsic parameter matrix The optimized formula is: ; Optimize the extrinsic parameter matrix The optimized formula is: ; The initial triangular mesh vertex coordinate set is optimized using the following formula: ; Repeat the above optimization steps until the number of iterations reaches the set limit.

5. A system for stylized 3D face shape modeling, used to implement the stylized 3D face shape modeling method according to any one of claims 1 to 4, characterized in that, include: The training module is used to train the 3DMM fitting model using real human face images; The output module is used to input the stylized face image into the trained 3DMM fitting model to obtain an initial triangular mesh that matches the face shape. The deformation module is used to perform Laplacian deformation on the initial triangular mesh with the facial key points in the stylized face image as the target, so as to obtain the final triangular mesh that accurately matches the face shape.

6. A device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the stylized three-dimensional face shape modeling method according to any one of claims 1 to 4.

7. A storage medium, characterized in that, It stores a computer program executable by the device, which, when run on the device, causes the device to perform the stylized three-dimensional face shape modeling method according to any one of claims 1 to 4.

Images