Three-dimensional head mesh generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for generating a three-dimensional head mesh, the method comprising: determining key points of a face in a two-dimensional image; inputting the first coordinates of the key points of the face and the second coordinates of the vertices in a preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model to output the third coordinates of the vertices in the three-dimensional head mesh corresponding to the face in the two-dimensional image; generating the three-dimensional head mesh according to the plurality of third coordinates. According to the technical solution of the present disclosure, a three-dimensional head mesh can be automatically constructed based on a single two-dimensional image so as to be subsequently used to generate a specific texture image, without the need to pre-build a deformable three-dimensional face model or to manually draw, thus saving manpower and time.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional head mesh generation, and in particular to a three-dimensional head mesh generation method, device, electronic device and storage medium. Background Art

[0002] With the popularity of selfie and short video functions, users' demand for facial beautification and related facial effects has gradually increased. Many special effects rely on 3D face models, which include 3D real face models and 3D cartoon (also known as two-dimensional) face models.

[0003] In the related technology, a deformable 3D face model (such as a 3D Morphable Model) can be used to construct a 3D real face model for a single real face image. Since the deformable 3D face model is pre-constructed based on scan data of a large number of real face images, only a 3D real face model can be constructed based on it.

[0004] In order to build a 3D cartoon face model, designers often need to draw it manually, which takes a lot of time and energy. Summary of the invention

[0005] The present disclosure provides a method, device, electronic device and storage medium for generating a three-dimensional head mesh, so as to at least solve the technical problems in the related art. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a method for generating a three-dimensional head mesh is provided, comprising:

[0007] Determine facial key point information in a two-dimensional image;

[0008] Inputting the facial key point information and the information of vertices in a preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model to output a target three-dimensional head mesh corresponding to the face in the two-dimensional image;

[0009] In which, the three-dimensional head reconstruction model is obtained based on loss function training, and the loss function includes at least a first constraint item and a second constraint item. The first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point, and the second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh.

[0010] Optionally, the first constraint item is calculated in the following way:

[0011] Determine the difference between the projection coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point

[0012] Optionally, the second constraint item is calculated in the following way:

[0013] Determine the difference between the Laplace coordinates of each vertex in the target three-dimensional face mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional face mesh.

[0014] Optionally, the loss function of the model further includes a third constraint term, wherein the third constraint term is calculated in the following manner:

[0015] Determine the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

[0016] Optionally, the loss function of the model further includes a fourth constraint term, wherein the fourth constraint term is calculated in the following manner:

[0017] The difference between the length of the edge in the target three-dimensional head mesh and the length of the corresponding edge in the preset standard three-dimensional head mesh is determined.

[0018] Optionally, the method further comprises:

[0019] Determining a facial skin mask in the two-dimensional image;

[0020] Wherein, after outputting the target three-dimensional head mesh corresponding to the face in the two-dimensional image, the method further comprises:

[0021] Determine a facial skin area in the two-dimensional image according to the facial skin mask;

[0022] Projecting the human face skin region onto the target three-dimensional head mesh to determine the first coordinate of each pixel point in the human face skin region and the corresponding second coordinate in the target three-dimensional head mesh, and determining the texture information of the vertex corresponding to the corresponding second coordinate according to the color information of the pixel point corresponding to the first coordinate and the depth information of the vertex corresponding to the corresponding second coordinate in the target three-dimensional head mesh;

[0023] The target three-dimensional head mesh is projected onto a preset texture space to determine a second coordinate of each vertex in the target three-dimensional head mesh and a corresponding third coordinate in the preset texture space, and texture information of a pixel point corresponding to the third coordinate is determined according to texture information of the vertex corresponding to the second coordinate.

[0024] Optionally, after determining the corresponding texture information of the third coordinate according to the texture information of the second coordinate, the method further includes:

[0025] Projecting the human face skin mask onto the target three-dimensional head mesh to determine a three-dimensional human face skin area in the target three-dimensional head mesh;

[0026] Projecting the three-dimensional human face skin area to the preset texture space to determine a target human face skin mask in the preset texture space;

[0027] According to the target facial skin mask, the texture information of the third coordinate in the texture space located in the target facial skin mask, and a preset texture completion model, the texture information of the third coordinate in the texture space located outside the target facial skin mask is completed.

[0028] According to a second aspect of an embodiment of the present disclosure, a three-dimensional head mesh generation device is provided, comprising:

[0029] A key point determination module is configured to determine the key point information of the face in the two-dimensional image;

[0030] A coordinate determination module is configured to input the facial key point information and the information of vertices in a preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model to output a target three-dimensional head mesh corresponding to the face in the two-dimensional image;

[0031] In which, the three-dimensional head reconstruction model is obtained based on loss function training, and the loss function includes at least a first constraint item and a second constraint item. The first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point, and the second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh.

[0032] Optionally, the first constraint item is calculated in the following way:

[0033] Determine the difference between the projection coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point.

[0034] Optionally, the second constraint item is calculated in the following way:

[0035] Determine the difference between the Laplace coordinates of each vertex in the target three-dimensional face mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional face mesh.

[0036] Optionally, the loss function of the model further includes a third constraint term, wherein the third constraint term is calculated in the following manner:

[0037] Determine the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

[0038] Optionally, the loss function of the model further includes a fourth constraint term, wherein the fourth constraint term is calculated in the following manner:

[0039] The difference between the length of the edge in the target three-dimensional head mesh and the length of the corresponding edge in the preset standard three-dimensional head mesh is determined.

[0040] Optionally, the device further comprises:

[0041] A mask determination module is configured to determine a face skin mask in the two-dimensional image;

[0042] a skin determination module, configured to determine a human face skin area in the two-dimensional image according to the human face skin mask;

[0043] The first projection module is configured to perform projection of the human face skin area onto the target three-dimensional head mesh to determine the first coordinate of each pixel point in the human face skin area, the corresponding second coordinate in the target three-dimensional head mesh, and determine the texture information of the vertex corresponding to the corresponding second coordinate according to the color information of the pixel point corresponding to the first coordinate and the depth information of the vertex corresponding to the corresponding second coordinate in the target three-dimensional head mesh; and project the target three-dimensional head mesh onto a preset texture space to determine the second coordinate of each vertex in the target three-dimensional head mesh, the corresponding third coordinate in the preset texture space, and determine the texture information of the pixel point corresponding to the corresponding third coordinate according to the texture information of the vertex corresponding to the second coordinate.

[0044] Optionally, the device further comprises:

[0045] A second projection module is configured to project the face skin mask onto the three-dimensional head mesh to determine a three-dimensional face skin area in the three-dimensional head mesh; and project the three-dimensional face skin area onto the preset texture space to determine a target face skin mask in the preset texture space;

[0046] The texture completion module is configured to complete the texture information of the third coordinate in the texture space outside the target face skin mask based on the target face skin mask, the texture information of the third coordinate in the texture space located in the target face skin mask, and a preset texture completion model.

[0047] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including:

[0048] processor;

[0049] a memory for storing instructions executable by the processor;

[0050] The processor is configured to execute the instructions to implement the three-dimensional head mesh generation method as described in any of the above embodiments.

[0051] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is proposed. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the three-dimensional head mesh generation method described in any of the above embodiments.

[0052] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product is configured to execute the three-dimensional head mesh generation method described in any of the above embodiments.

[0053] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0054] According to an embodiment of the present disclosure, the model outputs a three-dimensional head mesh corresponding to the face in the two-dimensional image based on the loss function. On the one hand, through the first constraint item, each vertex in the target three-dimensional head mesh can be made as close as possible to the position of the corresponding face key point, so that the three-dimensional head mesh is as close as possible to the face in the two-dimensional image; on the other hand, based on the second constraint item, the relationship between the vertices in the three-dimensional head mesh corresponding to the face in the two-dimensional image can be made as close as possible to the relationship between each vertex in the preset standard three-dimensional head mesh, thereby ensuring that the relationship between the vertices in the three-dimensional head mesh can conform to the positional relationship in the three-dimensional space, which is conducive to ensuring the accuracy of the generated three-dimensional head mesh.

[0055] A three-dimensional head mesh can be automatically constructed based on a single two-dimensional image for subsequent use in generating specific texture images. There is no need to pre-build a deformable three-dimensional face model or manually draw it, saving manpower and time.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0058] Figure 1 It is a schematic flow chart of a method for generating a three-dimensional head mesh according to an embodiment of the present disclosure.

[0059] Figure 2 is a schematic diagram of a two-dimensional image according to an embodiment of the present disclosure.

[0060] Figure 3 It is a schematic diagram of key points of a human face according to an embodiment of the present disclosure.

[0061] Figure 4 It is a schematic flow chart of another method for generating a three-dimensional head mesh according to an embodiment of the present disclosure.

[0062] Figure 5 It is a schematic diagram of a facial skin mask according to an embodiment of the present disclosure.

[0063] Figure 6 It is a partial schematic diagram of a three-dimensional face image determined based on texture information of a third coordinate according to an embodiment of the present disclosure.

[0064] Figure 7 is a schematic flow chart of another method for generating a three-dimensional head mesh according to an embodiment of the present disclosure.

[0065] Figure 8 is a schematic diagram of a target face skin mask according to an embodiment of the present disclosure.

[0066] Fig. 9 It is a schematic diagram of a completed three-dimensional face image according to an embodiment of the present disclosure.

[0067] Fig.10 It is a schematic diagram of a rendered three-dimensional face image according to an embodiment of the present disclosure.

[0068] Fig.11 It is a schematic diagram of another rendered three-dimensional face image according to an embodiment of the present disclosure.

[0069] Fig.12 It is a schematic block diagram of a three-dimensional head mesh generating device according to an embodiment of the present disclosure.

[0070] Fig.13 It is a schematic block diagram of another three-dimensional head mesh generating device according to an embodiment of the present disclosure.

[0071] Fig.14 It is a schematic block diagram of another three-dimensional head mesh generating device according to an embodiment of the present disclosure.

[0072] Fig.15 It is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] Figure 1 The present invention is a schematic flow chart of a method for generating a three-dimensional (also referred to as 3D) head mesh according to an embodiment of the present invention. The method shown in this embodiment can be applied to electronic devices such as terminals and servers, and the terminals include but are not limited to mobile phones, tablet computers, wearable devices, personal computers, etc.

[0076] like Figure 1 As shown, the method may include the following steps:

[0077] In step S101, determining facial key point information in a two-dimensional image;

[0078] In step S102, the facial key point information and the second coordinates of the vertices in the preset standard three-dimensional head mesh are input into a pre-trained three-dimensional head reconstruction model to output a three-dimensional head mesh corresponding to the face in the two-dimensional image;

[0079] In which, the three-dimensional head reconstruction model is obtained based on loss function training, and the loss function includes at least a first constraint item and a second constraint item. The first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point, and the second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh.

[0080] In one embodiment, a user may select a two-dimensional image to generate a three-dimensional head mesh, and for the two-dimensional model, the key point information of the face may be determined. The two-dimensional image may be a two-dimensional image containing a real person, or a two-dimensional image containing a cartoon character. The following embodiments are mainly exemplified for two-dimensional images containing cartoon characters.

[0081] Figure 2 is a schematic diagram of a two-dimensional image according to an embodiment of the present disclosure. Figure 3 It is a schematic diagram of key points of a human face according to an embodiment of the present disclosure.

[0082] In one embodiment, a face detection model and a key point detection model can be obtained in advance through machine learning training. Then, the face detection model is used to determine Figure 2 The face in the two-dimensional cartoon image (cartoon face) shown in FIG. 1 and the key points of the face are determined in the face by the key point detection model, and the following is obtained: Figure 3 The key points shown.

[0083] In one embodiment, a model may be obtained in advance through machine learning training (eg supervised training) for a large number (eg 5,000 to 50,000) of two-dimensional cartoon face images, and the input of the model includes two aspects.

[0084] On the one hand, the first coordinates of the determined facial key points are included. For example, if 60 facial key points are determined, then 60 first coordinates are included.

[0085] On the other hand, the second coordinates of the vertices in the preset standard three-dimensional head mesh are provided, wherein the preset standard three-dimensional head mesh may be pre-constructed, including a plurality of vertices, the vertices are connected by edges, forming a plurality of triangles, and then a mesh is formed by the plurality of triangles. For example, if the preset standard three-dimensional head mesh includes 1000 vertices, then it includes 1000 second coordinates.

[0086] In addition, for the vertices in the preset standard three-dimensional head mesh, the correspondence between them and the key points detected by the key point detection model can be pre-set. For example, 60 facial key points are determined by the key point detection model, then in the preset standard three-dimensional head mesh, there are vertices corresponding to these 60 facial key points.

[0087] Since a two-dimensional image contains less three-dimensional information, it is difficult to accurately construct a three-dimensional face model based only on the key points of the face in the two-dimensional image. To this end, this embodiment pre-constructs a preset standard three-dimensional head mesh, which is three-dimensional and contains sufficient three-dimensional information, so that the two-dimensional image and the preset standard three-dimensional head mesh can be combined to relatively accurately construct a three-dimensional face model.

[0088] By inputting facial key point information and information of vertices in a preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model, the three-dimensional head reconstruction model can output a three-dimensional head mesh corresponding to the face in the two-dimensional image.

[0089] For example, the facial key point information is the first coordinate of the facial key point, and the information of the vertices in the preset standard three-dimensional head mesh is the second coordinate of the vertices in the preset standard three-dimensional head mesh. Then, the first coordinate of each facial key point and the second coordinate of each vertex in the preset standard three-dimensional head mesh are input into the three-dimensional head reconstruction model, and the third coordinate of each vertex in the three-dimensional head mesh can be output.

[0090] In one embodiment, in order to enable the model to reflect both the relationship between the two-dimensional face image and the three-dimensional head mesh and the relationship between the preset standard three-dimensional head mesh and the three-dimensional head mesh, when constructing the loss function, the loss function can be set to include at least two constraints.

[0091] Among them, the first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point. For example, the first constraint can be obtained by determining the difference between the projection coordinates of the coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point. Specifically, the first constraint item is the sum of the differences between the projection coordinates of each third coordinate projected onto the plane where the two-dimensional image is located and each corresponding first coordinate.

[0092] For example, if there are n facial key points in a two-dimensional image, the first coordinate of the i-th facial key point can be expressed as P i , 1≤i≤n, among the vertices of the three-dimensional head mesh, the vertex corresponding to the i-th facial key point, the projection coordinates of the projection to the plane where the two-dimensional image is located are expressed as LM i(M), the 3D head mesh corresponding to the 2D face image is denoted as M.

[0093] Then the first constraint can be expressed as The superscript 2 indicates square calculation, and the subscript 2 indicates calculation according to the L2 norm.

[0094] Among them, the second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh. For example, the second constraint can be obtained by determining the difference between the Laplace coordinates of each vertex in the target three-dimensional face mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional face mesh. Specifically, the second constraint item is the sum of the differences between the Laplace coordinates of each vertex in the three-dimensional head mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional head mesh.

[0095] The preset standard 3D head mesh is represented as M 0 , the Laplace coordinates of each vertex are calculated as Lap(M 0 ), the Laplace coordinates of each vertex in the three-dimensional head mesh corresponding to the two-dimensional face image are calculated as Lap(M), then the second constraint can be expressed as

[0096] Then the loss function can be expressed as:

[0097] where λ 1 and λ 2 is the weight, which can be set as needed. When the loss function is small (for example, less than the preset value), the model can output the third coordinate corresponding to each vertex.

[0098] The model outputs a three-dimensional head mesh corresponding to the face in the two-dimensional image based on the loss function. On the one hand, through the first constraint item, each vertex in the target three-dimensional head mesh can be made as close as possible to the position of the corresponding facial key point, so that the three-dimensional head mesh is as close as possible to the face in the two-dimensional image; on the other hand, based on the second constraint item, the relationship between the vertices in the three-dimensional head mesh corresponding to the face in the two-dimensional image can be made as close as possible to the relationship between each vertex in the preset standard three-dimensional head mesh, thereby ensuring that the relationship between the vertices in the three-dimensional head mesh can conform to the positional relationship in the three-dimensional space, which is conducive to ensuring the accuracy of the generated three-dimensional head mesh.

[0099] According to the embodiments of the present disclosure, a three-dimensional head mesh can be automatically constructed based on a single two-dimensional image so as to be used to subsequently generate a specific texture image. There is no need to pre-build a deformable three-dimensional face model or manual drawing, thus saving manpower and time.

[0100] Optionally, the first constraint item is calculated in the following way:

[0101] Determine the difference between the projection coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point.

[0102] Optionally, the second constraint item is calculated in the following way:

[0103] Determine the difference between the Laplace coordinates of each vertex in the target three-dimensional face mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional face mesh.

[0104] Optionally, the loss function of the model further includes a third constraint term, wherein the third constraint term is calculated in the following manner:

[0105] Determine the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

[0106] In one embodiment, in addition to the first constraint term and the second constraint term, the loss function of the model may also include other constraints, such as a third constraint term. The third constraint term can be obtained by determining the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh. Specifically, the third constraint term can be the sum of the differences between the normal vector of each vertex in the three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

[0107] The normal vector calculated for each vertex in the preset standard 3D head mesh is expressed as Norm(M 0 ), the normal vector calculated for each vertex in the three-dimensional head mesh corresponding to the two-dimensional face image is expressed as Norm(M), then the third constraint can be expressed as

[0108] Then, based on the first constraint and the second constraint, the loss function can be expressed as:

[0109] where λ 3 is the weight, which can be set as needed.

[0110] Based on the third constraint item, the normal vectors of the vertices in the three-dimensional head mesh corresponding to the face in the two-dimensional image can be made as close as possible to the normal vectors between each vertex in the preset standard three-dimensional head mesh, thereby further ensuring that the relationship between the vertices in the three-dimensional head mesh can conform to the positional relationship in the three-dimensional space, which is conducive to ensuring the accuracy of the three-dimensional head mesh generated according to the third coordinate.

[0111] Optionally, the loss function of the model also includes a fourth constraint term, wherein the fourth constraint term can be obtained by determining the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh. Specifically, the fourth constraint term is the sum of the differences between the length of the edge in the three-dimensional head mesh and the length of the corresponding edge in the preset standard three-dimensional head mesh.

[0112] In one embodiment, the loss function of the model may include other constraints in addition to the first constraint term and the second constraint term, such as a fourth constraint term, where the fourth constraint term is the sum of the differences between the lengths of the edges in the three-dimensional head mesh and the lengths of the corresponding edges in the preset standard three-dimensional head mesh.

[0113] The length of each edge in the preset standard four-dimensional head mesh is expressed as EL(M 0 ), the length of each edge in the four-dimensional head grid corresponding to the two-dimensional face image is expressed as EL(M), then the fourth constraint can be expressed as

[0114] Then, based on the first constraint and the second constraint, the loss function can be expressed as:

[0115] where λ 4 is the weight, which can be set as needed.

[0116] Based on the fourth constraint item, the length of the edge in the three-dimensional head mesh corresponding to the face in the two-dimensional image can be made as close as possible to the length of the corresponding edge in the preset standard three-dimensional head mesh, thereby further ensuring that the relationship between the vertices in the three-dimensional head mesh can conform to the positional relationship in the three-dimensional space, which is conducive to ensuring the accuracy of the three-dimensional head mesh generated according to the third coordinate.

[0117] It should be noted that the above embodiments of the third constraint item and the fourth constraint item can be combined, that is, the loss function can include four constraint items, then the loss function is:

[0118]

[0119] Figure 4FIG. 1 is a schematic flow chart of another method for generating a three-dimensional head mesh according to an embodiment of the present disclosure. Figure 4 As shown, the method also includes:

[0120] In step S103, a human face skin mask in the two-dimensional image is determined; this step may be performed after step S102, or the execution order may be adjusted as needed, for example, performed before step S102;

[0121] Wherein, after outputting the target three-dimensional head mesh corresponding to the face in the two-dimensional image, the method further comprises:

[0122] In step S104, the facial skin area in the two-dimensional image is determined according to the facial skin mask;

[0123] In step S105, the human face skin area is projected onto the target three-dimensional head mesh to determine the first coordinate of each pixel point in the human face skin area and the corresponding second coordinate in the target three-dimensional head mesh, and the texture information of the vertex corresponding to the corresponding second coordinate is determined according to the color information of the pixel point corresponding to the first coordinate and the depth information of the vertex corresponding to the corresponding second coordinate in the target three-dimensional head mesh;

[0124] In step S106, the target three-dimensional head mesh is projected onto a preset texture space to determine the second coordinate of each vertex in the target three-dimensional head mesh and the corresponding third coordinate in the preset texture space, and the texture information of the pixel corresponding to the third coordinate is determined according to the texture information of the vertex corresponding to the second coordinate.

[0125] Figure 5 It is a schematic diagram of a facial skin mask according to an embodiment of the present disclosure. Figure 6 It is a partial schematic diagram of a three-dimensional face image determined based on texture information of a third coordinate according to an embodiment of the present disclosure.

[0126] In one embodiment, a face segmentation model can be obtained in advance through machine learning training, and then based on the face segmentation model, Figure 2 The face in the two-dimensional image in the embodiment shown is segmented to obtain the following Figure 5 The face template in the two-dimensional face image shown.

[0127] The facial skin area in the two-dimensional image can be determined based on the facial skin mask, and then the facial skin area can be projected onto the three-dimensional head mesh to determine the second coordinate corresponding to each first coordinate in the facial skin area in the three-dimensional head mesh, and the texture information of the corresponding second coordinate can be determined based on the color information of the first coordinate. Since the second coordinate is located in the three-dimensional head mesh, it is a three-dimensional coordinate that can express information such as depth, and further combined with color information, it can be used as texture information.

[0128] The three-dimensional head mesh can then be projected onto a preset texture space to determine a third coordinate corresponding to each second coordinate in the three-dimensional head mesh in the preset texture space, and the texture information corresponding to the third coordinate can be determined based on the texture information of the second coordinate.

[0129] In one embodiment, a texture space, i.e., a preset texture space, can be pre-constructed. The preset texture space can be a two-dimensional plane. By projecting the second coordinate into the texture space, the corresponding third coordinate can be determined. Then, the texture information of the corresponding third coordinate can be determined based on the texture information of the second coordinate. Based on this, the texture information of the corresponding coordinate in the texture space can be determined for each point in the three-dimensional head mesh (including not only the vertices, but also the points inside the triangle formed by the vertices). Then, based on the texture information of the third coordinate, a partial three-dimensional face image can be determined, for example Figure 6 shown.

[0130] Figure 7 FIG. 1 is a schematic flow chart of another method for generating a three-dimensional head mesh according to an embodiment of the present disclosure. Figure 7 As shown, after determining the corresponding texture information of the third coordinate according to the texture information of the second coordinate, the method further includes:

[0131] In step S107, the face skin mask is projected onto the three-dimensional head mesh to determine a three-dimensional face skin area in the three-dimensional head mesh;

[0132] In step S108, the three-dimensional human face skin area is projected to the preset texture space to determine a target human face skin mask in the preset texture space;

[0133] In step S109, based on the target facial skin mask, the texture information of the third coordinate in the texture space located in the target facial skin mask, and a preset texture completion model, the texture information of the third coordinate in the texture space located outside the target facial skin mask is completed.

[0134] Figure 8 is a schematic diagram of a target face skin mask according to an embodiment of the present disclosure. Fig. 9 It is a schematic diagram of a completed three-dimensional face image according to an embodiment of the present disclosure. Fig.10 It is a schematic diagram of a rendered three-dimensional face image according to an embodiment of the present disclosure. Fig.11 It is a schematic diagram of another rendered three-dimensional face image according to an embodiment of the present disclosure.

[0135] In one embodiment, since the face skin mask determined in the two-dimensional image and the face skin mask in the texture space may be different, a relationship between the two can be established based on the three-dimensional head mesh. First, the face skin mask is projected onto the three-dimensional head mesh to determine the three-dimensional face skin area in the three-dimensional head mesh, and then the three-dimensional face skin area is projected onto the preset texture space to determine the target face skin mask in the preset texture space, for example Figure 8 As shown, it is Figure 6 The partial three-dimensional face images shown are corresponding.

[0136] Then, according to the target face skin mask, the texture information of the third coordinate in the texture space located in the target face skin mask, and the preset texture completion model, the texture information of the third coordinate outside the target face skin mask in the texture space can be completed. Based on the texture information of the third coordinate inside and outside the target face skin mask, a three-dimensional face image with completed texture can be determined, for example Fig. 9 shown.

[0137] Based on Fig. 9 The three-dimensional face image shown in FIG. 1 can be rendered to obtain a specific three-dimensional face image. For example, when the two-dimensional face image is a two-dimensional cartoon face image as shown in the above embodiment, the three-dimensional face image shown in FIG. 1 can be obtained by using the phong rendering method. Fig.10 The three-dimensional face image shown in the figure can be obtained through cartoon rendering. Fig.11 The three-dimensional face image shown.

[0138] Corresponding to the above-mentioned embodiment of the three-dimensional head mesh generation method, the present disclosure also proposes an embodiment of a three-dimensional head mesh generation device.

[0139] Fig.12 FIG. 1 is a schematic block diagram of a three-dimensional head mesh generation device according to an embodiment of the present disclosure. Fig.12 As shown, the three-dimensional head mesh generating device may include:

[0140] A key point determination module 101 is configured to determine the key point information of a face in a two-dimensional image;

[0141] The coordinate determination module 102 is configured to input the facial key point information and the vertex information in the preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model to output a target three-dimensional head mesh corresponding to the face in the two-dimensional image;

[0142] In which, the three-dimensional head reconstruction model is obtained based on loss function training, and the loss function includes at least a first constraint item and a second constraint item. The first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point, and the second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh.

[0143] Optionally, the first constraint item is calculated in the following way:

[0144] Determine the difference between the projection coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point.

[0145] Optionally, the second constraint item is calculated in the following way:

[0146] Determine the difference between the Laplace coordinates of each vertex in the target three-dimensional face mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional face mesh.

[0147] Optionally, the loss function of the model further includes a third constraint term, wherein the third constraint term is calculated in the following manner:

[0148] Determine the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

[0149] Optionally, the loss function of the model further includes a fourth constraint term, wherein the fourth constraint term is calculated in the following manner:

[0150] The difference between the length of the edge in the target three-dimensional head mesh and the length of the corresponding edge in the preset standard three-dimensional head mesh is determined.

[0151] Fig.13 FIG. 1 is a schematic block diagram of another apparatus for generating a three-dimensional head mesh according to an embodiment of the present disclosure. Fig.13 As shown, the device also includes:

[0152] The mask determination module 103 is configured to determine the face skin mask in the two-dimensional image;

[0153] The skin determination module 104 is configured to determine the human face skin area in the two-dimensional image according to the human face skin mask;

[0154] The first projection module 105 is configured to perform projection of the human face skin area onto the target three-dimensional head mesh to determine the first coordinate of each pixel point in the human face skin area, the corresponding second coordinate in the target three-dimensional head mesh, and determine the texture information of the vertex corresponding to the corresponding second coordinate according to the color information of the pixel point corresponding to the first coordinate and the depth information of the vertex corresponding to the corresponding second coordinate in the target three-dimensional head mesh; and project the target three-dimensional head mesh onto a preset texture space to determine the second coordinate of each vertex in the target three-dimensional head mesh, the corresponding third coordinate in the preset texture space, and determine the texture information of the pixel point corresponding to the third coordinate according to the texture information of the vertex corresponding to the second coordinate.

[0155] Fig.14 FIG. 1 is a schematic block diagram of another apparatus for generating a three-dimensional head mesh according to an embodiment of the present disclosure. Fig.14 As shown, the device also includes:

[0156] The second projection module 106 is configured to project the face skin mask onto the three-dimensional head mesh to determine the three-dimensional face skin area in the three-dimensional head mesh; and project the three-dimensional face skin area onto the preset texture space to determine the target face skin mask in the preset texture space;

[0157] The texture completion module 107 is configured to complete the texture information of the third coordinate in the texture space outside the target face skin mask based on the target face skin mask, the texture information of the third coordinate in the texture space located in the target face skin mask, and a preset texture completion model.

[0158] Regarding the device in the above embodiment, the specific manner in which each module / unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0159] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0160] The present disclosure also provides an electronic device, including:

[0161] processor;

[0162] a memory for storing instructions executable by the processor;

[0163] The processor is configured to execute the instructions to implement the three-dimensional head mesh generation method as described in any of the above embodiments.

[0164] An embodiment of the present disclosure further provides a storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the three-dimensional head mesh generation method described in any of the above embodiments.

[0165] An embodiment of the present disclosure further provides a computer program product, which is configured to execute the three-dimensional head mesh generation method described in any of the above embodiments.

[0166] Fig.15 1 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 1500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0167] Reference Fig.15 , the electronic device 1500 may include one or more of the following components: a processing component 1502 , a memory 1504 , a power component 1506 , a multimedia component 1508 , an audio component 1510 , an input / output (I / O) interface 1512 , a sensor component 1514 , and a communication component 1516 .

[0168] The processing component 1502 generally controls the overall operation of the electronic device 1500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1502 may include one or more processors 1520 to execute instructions to complete all or part of the steps of the above-mentioned three-dimensional head mesh generation method. In addition, the processing component 1502 may include one or more modules to facilitate the interaction between the processing component 1502 and other components. For example, the processing component 1502 may include a multimedia module to facilitate the interaction between the multimedia component 1508 and the processing component 1502.

[0169] The memory 1504 is configured to store various types of data to support operations on the electronic device 1500. Examples of such data include instructions for any application or method operating on the electronic device 1500, contact data, phone book data, messages, pictures, videos, etc. The memory 1504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0170] The power supply component 1506 provides power to the various components of the electronic device 1500. The power supply component 1506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1500.

[0171] The multimedia component 1508 includes a screen that provides an output interface between the electronic device 1500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1508 includes a front camera and / or a rear camera. When the electronic device 1500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0172] The audio component 1510 is configured to output and / or input audio signals. For example, the audio component 1510 includes a microphone (MIC), and when the electronic device 1500 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1504 or sent via the communication component 1516. In some embodiments, the audio component 1510 also includes a speaker for outputting audio signals.

[0173] I / O interface 1512 provides an interface between processing component 1502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0174] The sensor assembly 1514 includes one or more sensors for providing various aspects of status assessment for the electronic device 1500. For example, the sensor assembly 1514 can detect the open / closed state of the electronic device 1500, the relative positioning of components, such as the display and keypad of the electronic device 1500, and the sensor assembly 1514 can also detect the position change of the electronic device 1500 or a component of the electronic device 1500, the presence or absence of user contact with the electronic device 1500, the orientation or acceleration / deceleration of the electronic device 1500, and the temperature change of the electronic device 1500. The sensor assembly 1514 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0175] The communication component 1516 is configured to facilitate wired or wireless communication between the electronic device 1500 and other devices. The electronic device 1500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0176] In one embodiment of the present disclosure, the electronic device 1500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned three-dimensional head mesh generation method.

[0177] In an embodiment of the present disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1504 including instructions, and the instructions can be executed by a processor 1520 of an electronic device 1500 to complete the above-mentioned three-dimensional head mesh generation method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0180] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0181] The method and device provided in the embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A three-dimensional head mesh generation method, characterized in that: include: Determine facial key point information in a two-dimensional image; Inputting the facial key point information and the information of vertices in a preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model to output a target three-dimensional head mesh corresponding to the face in the two-dimensional image; The three-dimensional head reconstruction model is obtained by training based on a loss function, and the loss function includes at least a first constraint term and a second constraint term. The first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point, and the first constraint item is calculated in the following manner: determining the difference between the projection coordinates of the coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point; The second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh; The second constraint item is calculated by determining the difference between the Laplace coordinates of each vertex in the target three-dimensional head mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional head mesh.

2. The method according to claim 1, characterized in that: The loss function of the model also includes a third constraint term, wherein the third constraint term is calculated in the following manner: Determine the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

3. The method according to any one of claims 1 to 2, characterized in that The loss function of the model also includes a fourth constraint term, wherein the fourth constraint term is calculated in the following manner: The difference between the length of the edge in the target three-dimensional head mesh and the length of the corresponding edge in the preset standard three-dimensional head mesh is determined.

4. The method according to any one of claims 1 to 2, characterized in that The method further comprises: Determining a facial skin mask in the two-dimensional image; Wherein, after outputting the target three-dimensional head mesh corresponding to the face in the two-dimensional image, the method further comprises: Determine a facial skin area in the two-dimensional image according to the facial skin mask; Projecting the human face skin region onto the target three-dimensional head mesh to determine the first coordinate of each pixel point in the human face skin region and the corresponding second coordinate in the target three-dimensional head mesh, and determining the texture information of the vertex corresponding to the corresponding second coordinate according to the color information of the pixel point corresponding to the first coordinate and the depth information of the vertex corresponding to the corresponding second coordinate in the target three-dimensional head mesh; The target three-dimensional head mesh is projected onto a preset texture space to determine a second coordinate of each vertex in the target three-dimensional head mesh and a corresponding third coordinate in the preset texture space, and texture information of a pixel point corresponding to the third coordinate is determined according to texture information of the vertex corresponding to the second coordinate.

5. The method according to claim 4, characterized in that After determining the corresponding texture information of the third coordinate according to the texture information of the second coordinate, the method further includes: Projecting the human face skin mask onto the target three-dimensional head mesh to determine a three-dimensional human face skin area in the target three-dimensional head mesh; Projecting the three-dimensional human face skin area to the preset texture space to determine a target human face skin mask in the preset texture space; According to the target facial skin mask, the texture information of the third coordinate in the texture space located in the target facial skin mask, and a preset texture completion model, the texture information of the third coordinate in the texture space located outside the target facial skin mask is completed.

6. A three-dimensional head mesh generation device, characterized in that: include: A key point determination module is configured to determine the key point information of the face in the two-dimensional image; A coordinate determination module is configured to input the facial key point information and the information of vertices in a preset standard three-dimensional head mesh into a pre-trained three-dimensional head reconstruction model to output a target three-dimensional head mesh corresponding to the face in the two-dimensional image; The three-dimensional head reconstruction model is obtained by training based on a loss function, and the loss function includes at least a first constraint term and a second constraint term. The first constraint item is used to characterize the position difference between each vertex in the target three-dimensional head mesh and the corresponding facial key point, and the first constraint item is calculated in the following manner: determining the difference between the projection coordinates of the coordinates of each vertex in the target three-dimensional head mesh projected onto the plane where the two-dimensional image is located and the coordinates of each corresponding facial key point; The second constraint item is used to characterize the difference between the relationship between each vertex and adjacent vertices in the target three-dimensional head mesh and the relationship between the corresponding vertex and adjacent vertices in the preset standard three-dimensional head mesh. The second constraint item is calculated in the following manner: determining the difference between the Laplace coordinates of each vertex in the target three-dimensional head mesh and the Laplace coordinates of each corresponding vertex in the preset standard three-dimensional head mesh.

7. The device according to claim 6, characterized in that The loss function of the model also includes a third constraint term, wherein the third constraint term is calculated in the following manner: Determine the difference between the normal vector of each vertex in the target three-dimensional head mesh and the normal vector of each corresponding vertex in the preset standard three-dimensional head mesh.

8. The device according to any one of claims 6 to 7, characterized in that The loss function of the model also includes a fourth constraint term, wherein the fourth constraint term is calculated in the following manner: The difference between the length of the edge in the target three-dimensional head mesh and the length of the corresponding edge in the preset standard three-dimensional head mesh is determined.

9. The device according to any one of claims 6 to 7, characterized in that The device also includes: A mask determination module is configured to determine a face skin mask in the two-dimensional image; a skin determination module, configured to determine a human face skin area in the two-dimensional image according to the human face skin mask; The first projection module is configured to perform projection of the human face skin area onto the target three-dimensional head mesh to determine the first coordinate of each pixel point in the human face skin area, the corresponding second coordinate in the target three-dimensional head mesh, and determine the texture information of the vertex corresponding to the corresponding second coordinate according to the color information of the pixel point corresponding to the first coordinate and the depth information of the vertex corresponding to the corresponding second coordinate in the target three-dimensional head mesh; and project the target three-dimensional head mesh onto a preset texture space to determine the second coordinate of each vertex in the target three-dimensional head mesh, the corresponding third coordinate in the preset texture space, and determine the texture information of the pixel point corresponding to the corresponding third coordinate according to the texture information of the vertex corresponding to the second coordinate.

10. The device according to claim 9, characterized in that The device also includes: A second projection module is configured to project the face skin mask onto the three-dimensional head mesh to determine a three-dimensional face skin area in the three-dimensional head mesh; and project the three-dimensional face skin area onto the preset texture space to determine a target face skin mask in the preset texture space; The texture completion module is configured to complete the texture information of the third coordinate in the texture space outside the target face skin mask based on the target face skin mask, the texture information of the third coordinate in the texture space located in the target face skin mask, and a preset texture completion model.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional head mesh generation method according to any one of claims 1 to 5.

12. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the three-dimensional head mesh generation method according to any one of claims 1 to 5.

Images