A three-dimensional head deformation model construction method and device, and an electronic device

Abstract

The application relates to computer image science, in particular to a three-dimensional head morphable model construction method, device and electronic equipment. The method comprises the following steps: acquiring face point cloud data of an RGB head image; performing affine transformation on the face point cloud data and a predefined head template grid to acquire a head grid, wherein the predefined head template grid is a head template grid provided with a plurality of vertices and provided with a mouth cavity and shoulders; and performing completion on the head grid to obtain a complete head model. The embodiment of the application realizes head template grid predefinition, constructs a three-dimensional head morphable model by using the predefined head template grid, the constructed three-dimensional head morphable model is close to a real object, the application range is expanded, the connection between the head and a digital body is facilitated, the diversity of the three-dimensional head morphable model is increased, the existing problems of the three-dimensional head morphable model are solved, and a reference thought is provided for a three-dimensional model reconstruction technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and graphics, and more particularly, to a three-dimensional head morphable model construction method and device and electronic equipment. BACKGROUND

[0002] With the improvement of device computing power, the demand for digital products is increasing, and the head as an important part of digital people is attracting attention. The three-dimensional head morphable model is a statistical model for three-dimensional face reconstruction. By analyzing a large number of three-dimensional face data sets, a model that can flexibly express different face shapes and textures is established. The three-dimensional head morphable model represents the three-dimensional shape and texture of the face as a linear combination of a number of basis vectors, and establishes the model from the linear combination.

[0003] The existing three-dimensional head morphable model has the problem of only focusing on the face, which has limitations in actual application. SUMMARY

[0004] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and to provide a three-dimensional head morphable model construction method, device and electronic equipment, so as to achieve the effect of complete construction of the three-dimensional head morphable model.

[0005] According to a first aspect of the present application, a three-dimensional head morphable model construction method is provided.

[0006] Obtaining face point cloud data of an RGB head image;

[0007] Performing affine transformation on the face point cloud data and a predefined head template grid to obtain a head grid, the predefined head template grid being a head template grid provided with a plurality of vertices and having a mouth and shoulders;

[0008] Completing the head grid to obtain a complete head model.

[0009] Performing three-dimensional morphable model construction according to the predefined head template grid and the face point cloud data, the constructed model not only focusing on the face, but also containing other parts such as shoulders. Close to the real object, the application range is expanded, which is beneficial to the connection of the head and the digital body, and increases the diversity of the three-dimensional head morphable model.

[0010] Optionally, the predefined head template grid is a head template grid provided with a plurality of vertices and having a mouth and shoulders, comprising:

[0011] The predefined head template grid includes at least an eye region, a mouth region, a mouth region and a shoulder region;

[0012] The vertex distribution mode of the eye region and the mouth region is denser than that of other regions in the predefined head template grid.

[0013] The predefined head template grid includes multiple regions such as an eye region, a mouth region, an oral cavity region, and a shoulder region, and is related to a wide range of model types, not only focusing on a face region, and the limitation in actual application is reduced; the vertex distribution of the eye region and the mouth region is denser than that of other regions in the predefined head template grid, which relieves the pressure on the display memory to some extent and is beneficial to the construction of the three-dimensional head model.

[0014] Optionally, the face point cloud data is subjected to affine transformation with the predefined head template grid to obtain a head grid, including:

[0015] Obtaining key points of the predefined head template grid from the predefined head template grid;

[0016] Obtaining key points of the face point cloud data from the face point cloud data;

[0017] Performing first affine transformation on the key points in the predefined head template grid and the face point cloud data to obtain a first registration result of the predefined head template grid and the face point cloud data;

[0018] Performing second affine transformation on the face point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result;

[0019] Obtaining a head grid according to the second affine transformation result.

[0020] The first registration is performed on the key points in the predefined head template grid and the face point cloud data, the coarse registration of the predefined head template grid and the face point cloud data is completed, a basis is provided for the second affine transformation, the loss of registration is reduced, and the accuracy of the construction of the three-dimensional morphable model is improved.

[0021] Optionally, the second affine transformation on the face point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result specifically includes:

[0022] Supposing that the total number of vertices in the predefined head template grid is n, the total number of vertices n is taken as the number of outer loops of the iterative optimization of the second affine transformation;

[0023] For each outer loop, the inner loop optimization iteration of the second affine transformation is performed on each vertex to obtain an affine transformation matrix of each vertex;

[0024] The affine transformation matrices corresponding to all the vertices are taken as the second affine transformation result;

[0025] The inner loop optimization iteration of the second affine transformation for each vertex specifically includes:

[0026] The preset initial affine transformation matrix is used to determine whether the vertex belongs to the shoulder region. If not, the affine transformation matrix of the vertex is updated based on the initial affine transformation matrix and the loss function. If yes, the initial affine transformation matrix is used as the affine transformation matrix of the vertex.

[0027] The second affine transformation minimizes the spatial position difference between the face point cloud data to obtain a more accurate rotation and translation transformation matrix, realizes image registration, ensures that parallel lines remain parallel after transformation, aligns images under different perspectives to a common coordinate system in three-dimensional model reconstruction, and improves the accuracy of three-dimensional reconstruction. According to the vertex belonging to the shoulder region, the affine transformation matrix of the vertex is not updated, which is beneficial to the connection of the head and shoulder model and the body part.

[0028] Optionally, the updating of the affine transformation matrix of the vertex based on the initial affine transformation matrix and the loss function specifically includes:

[0029] For the affine transformation matrix of the vertex, it is determined whether the number of vertices of the vertex is less than the number n of outer loops of the second affine transformation iteration optimization. If the number of vertices of the vertex is less than the number n of outer loops of the second affine transformation iteration optimization, the difference between the current affine transformation result and the previous affine transformation result of the vertex is calculated.

[0030] It is determined whether the difference between the current affine transformation result and the previous affine transformation result of the vertex is greater than a threshold value of the difference between two affine transformations. If the difference is greater than the threshold value of the difference between two affine transformations, the corresponding point of the vertex is found.

[0031] The loss function is used to solve the optimized affine transformation matrix between the vertex and the corresponding point of the vertex.

[0032] The updating of the affine transformation matrix of the vertex based on the initial affine transformation matrix and the loss function reduces the initial cost of the affine transformation, and the loss function refines the loss in the affine transformation, accurately calculates the existing loss, and improves the efficiency of three-dimensional deformation model construction.

[0033] Optionally, the loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise.

[0034] The face point cloud data and the predefined head template grid are subjected to the second affine transformation, and the distance loss exists between the vertices after the affine transformation and the vertices before the affine transformation.

[0035] calculating the distance loss by using a distance loss function , specifically:

[0036]

[0037] wherein, represents the i-th vertex in the predefined head template mesh, represents the i-th vertex in the predefined head template mesh, represents an affine transformation matrix of, represents the i-th vertex to the nearest point in the point cloud, represents a distance weight, represents the square of the distance of two vertices, represents a predefined head template mesh; There is a stiffness loss between two adjacent vertices of the predefined head template mesh;

[0038] calculating the stiffness loss by using a stiffness loss function , specifically:

[0039]

[0040]

[0041] wherein, represents an edge of the predefined head template mesh, , represents an affine transformation matrix of two adjacent vertices on, represents the square of the F-norm;

[0042] There is a key point loss between the predefined head template mesh and the key points of the face point cloud data;

[0043] calculating the key point loss by using a key point loss function , specifically:

[0044]

[0045] In the key point loss function, represents the matching relationship between the vertices of the predefined head template mesh and the key points of the point cloud data; represents the vertices of the predefined head template mesh, represents the key points of the point cloud data;

[0046] There is noise after the predefined head template mesh and the face point cloud data complete the affine transformation;

[0047] calculating the noise by using a Laplacian smoothing loss function​​​ , specifically:

[0048]

[0049] and represent the first vertex, the second vertex on the predefined head template mesh; represent the normalized weight related to the distance from to .

[0050] According to the function, the loss existing in the affine transformation can be solved, the loss generated in the process of constructing the model is reduced, and the efficiency of constructing the three-dimensional head deformable model is improved.

[0051] Optionally, the loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise, specifically, the loss function is calculated according to distance loss, stiffness loss, key point loss, and noise, and the calculation formula is as follows:

[0052]

[0053] wherein, the total loss function value is , , , respectively are the weight of stiffness loss, key point loss, and Laplacian smoothing loss.

[0054] The total loss function value accurately measures the loss generated in the process of constructing the model, numerizes the loss, and facilitates formulating a better optimization scheme for the constructed model.

[0055] Optionally, the head mesh is completed to obtain a complete head model, specifically including:

[0056] a plurality of reference points are obtained from an eye region of the predefined head template mesh

[0057] an eyeball position is obtained according to the plurality of reference points;

[0058] the head mesh is completed according to the eyeball position to obtain a complete head model.

[0059] The predefined head template mesh contains multiple regions, but does not have a specific eyeball model, and the head mesh is completed to obtain a complete three-dimensional head deformable model, and a specific and complete head model is generated.

[0060] According to a second aspect of the present application, a three-dimensional head deformable model construction device is provided, the device comprising:​​

[0061] an acquisition module configured to acquire face point cloud data of an RGB head image;

[0062] an affine transformation module configured to perform affine transformation on the face point cloud data and a predefined head template grid to obtain a head grid, the predefined head template grid being a head template grid provided with a plurality of vertices and having a mouth and a shoulder;

[0063] a supplement module configured to supplement the head grid to obtain a complete head model processing module.

[0064] The three-dimensional head morphable model device, each module is an indispensable part of the whole process, modules cooperate with each other, the RGB head image is acquired by the acquisition module, the face point cloud data of the RGB head image is acquired; the face point cloud data and the predefined head template grid are registered by the affine transformation module, the face point cloud data and the predefined head template grid are subjected to second affine transformation according to the first registration result, and the head grid is obtained from the affine transformation module; the head grid is supplemented by the supplement module, and the eyeball is supplemented to obtain a complete three-dimensional head morphable model.

[0065] According to a third aspect of the present application, an electronic device is provided, comprising:

[0066] a memory configured to store one or more computer programs;

[0067] a processor configured to implement the three-dimensional head morphable model method of the first aspect when the one or more computer programs are executed by the processor.

[0068] The electronic device provides a complete system for three-dimensional head morphable model construction, stores images, processes the stored images, and saves the processed images in the memory.

[0069] According to a fourth aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores computer instructions, the computer instructions are used to make the processor execute to implement the three-dimensional head morphable model method of the first aspect.

[0070] Based on any one of the above aspects, the three-dimensional head morphable model construction method, device and electronic equipment provided by the embodiments of the present application obtain the face point cloud data of the head image to provide data for constructing the three-dimensional head morphable model. The vertices on the predefined head template are effectively arranged by using the predefined head template grid containing at least the eye region, the mouth region, the oral cavity region and the shoulder region, so that the model can be completely established, and the details of the original image can be well expressed, and the vertices of the shoulder region are preset to improve the effect of complete construction of the three-dimensional head morphable model.

[0071] Further, the vertices belonging to the shoulder region do not update the affine transformation matrix, and the distribution of the vertices is not affected. The three-dimensional head morphable model constructed according to the construction method has high precision, small memory and is beneficial to the connection of the head and the body. BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0073] Figure 1 A schematic application scenario diagram of the three-dimensional head morphable model construction method provided by the present embodiment.

[0074] Figure 2 A flowchart of the three-dimensional head morphable model construction method provided by the present embodiment.

[0075] Figure 3 A sub-step schematic diagram of step S120 provided by the present embodiment.

[0076] Figure 4 A functional module schematic diagram of the three-dimensional head morphable model construction device provided by the present embodiment.

[0077] Figure 5 A structural schematic diagram of the electronic equipment provided by the present embodiment. DETAILED DESCRIPTION

[0078] The drawings of the present application are only used for illustrative description and cannot be understood as limitation of the present application. In order to better illustrate the following embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0079] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0080] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0081] Some three-dimensional head deformation models have some limitations in facial modeling. Some models only focus on facial features, without including independent parts such as eyeballs, oral cavities and shoulders, which limits their application in a wider field. The vertex distribution of some three-dimensional head deformation models is too dense, which will cause excessive memory occupation in actual application, affecting performance. In addition, when the basis vector coefficient of some three-dimensional head deformation models containing shoulders changes, the shoulders will also change undesirably, which will bring inconvenience when the head model is fused with a digital body model.

[0082] The embodiment provides a technical scheme that can solve the above problems. The specific embodiments of the present application will be described in detail below in conjunction with the drawings.

[0083] Exemplarily, a three-dimensional deformation model construction method provided by the embodiment of the present application is applied to a scene. As shown in Figure 1 The application scene at least includes a server 100 and a terminal 200 that can communicate with the server 100, the server 100 has the function of processing data to construct a model; the terminal device 200 has the function of displaying the constructed model, and also has the function of processing data to construct a model.

[0084] It can be understood that the server 100 can be an independent electronic device, or a cluster composed of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a vehicle-mounted terminal, etc., but is not limited thereto.

[0085] In an implementable manner, the server 100 and the terminal 200 can respectively perform the three-dimensional head morphable model construction method provided in the embodiments of the present application, or alternatively, the three-dimensional head morphable model construction method provided in the embodiments of the present application can be partially executed in the server 100 and partially executed in the terminal 200.

[0086] As shown in Figure 2 The embodiments provide a three-dimensional head morphable model construction method, which can include the following steps:

[0087] S110, obtaining face point cloud data of an RGB head image;

[0088] The face point cloud data can clearly reproduce the texture and shape of the RGB head image, and is convenient for feature extraction of head details, thereby providing a data basis for constructing a three-dimensional head model.

[0089] In the embodiments, the face point cloud data of the RGB head image is obtained in a manner including monocular vision technology, binocular vision technology, multi-view vision technology, etc.

[0090] In an alternative implementation manner, multi-view vision technology is used to obtain images of hundreds of human heads, including various age groups and various face shapes, and the number of men and women is basically equal. A plurality of cameras surround the head of the person being photographed, and the shutter trigger time of the camera is within 5 ms. The method for obtaining the face point cloud data by scanning the obtained RGB head image includes but is not limited to this.

[0091] S120, performing affine transformation on the face point cloud data and a predefined head template grid to obtain a head grid, wherein the predefined head template grid is a head template grid provided with a plurality of vertices and having an oral cavity and a shoulder;

[0092] The predefined head template grid includes an eye region, a mouth region, an oral cavity region, and a shoulder region, an ear region, etc., and the plurality of regions are combined to form a grid.

[0093] The predefined head template grid can be provided with a plurality of vertices, which can be randomly arranged or regularly arranged, and there are various arrangement schemes.

[0094] In an alternative implementation manner, the predefined head template grid includes at least an eye region, a mouth region, an oral cavity region, and a shoulder region, and the plurality of vertices are distributed more densely in the eye region and the mouth region than in other regions in the predefined head template grid. The manner includes but is not limited to this.

[0095] The affine transformation includes one or more of a translation transformation, a rotation transformation, a scale transformation, and a skew transformation.

[0096] In this embodiment, before the affine transformation is performed on the face point cloud data and the predefined head template grid, the predefined head template grid template is normalized so that the three coordinates of the template vertex range between -1 and 1.

[0097] In an optional implementation, one or more of a translation transformation, a rotation transformation, a scale transformation, and a skew transformation is performed on the key points in the predefined head template grid and the face point cloud data to obtain a first registration result of the predefined head template grid and the face point cloud data.

[0098] The key points in the predefined head template grid and the face point cloud data can be obtained by using a key point extraction algorithm such as ISS key point detection or SIFT key point detection.

[0099] In an optional implementation, the RGB head image can include one or more, and the corresponding face point cloud data is obtained by scanning the one or more RGB head images. When the RGB head image includes multiple RGB head images, the head of a person of various ages can be included according to actual needs, and the number of heads of men and women can also be allocated according to actual needs. Specifically, when the RGB head image includes hundreds of images, the face images of the front are selected from the scanned hundreds of RGB head images with natural expressions, and the two-dimensional face key points are detected from the face images. The three-dimensional key points are obtained according to the internal and external parameters of the scanning camera, and then the points closest to the three-dimensional key points are found from the face point cloud data as the key points of the face point cloud. In the predefined head template grid, the points with the same semantics as the face point cloud key points are selected as the key points of the predefined head template grid. The method for obtaining the key points includes but is not limited to the above.

[0100] According to the first registration result, one or more of a translation transformation, a rotation transformation, a scale transformation, and a skew transformation is performed on the face point cloud data and the predefined head template grid to obtain a second affine transformation result, so as to obtain a head grid.

[0101] In S130, the head grid is completed to obtain a complete head model.

[0102] In this embodiment, the head grid is completed by using a completion network or based on a model.

[0103] In an alternative implementation, a plurality of reference points are obtained from an eye region of the predefined head template mesh, the reference points are determined according to inner and outer corners of the eye region, highest and lowest points of the eye socket, an eyeball position is obtained from the reference points, a complete head model is obtained by performing eyeball completion on the head mesh through the eyeball position, and a complete head model is obtained by installing eyeballs on a plurality of head meshes, and specifically includes that a shape of a human head can be represented as:

[0104]

[0105] In the formula, S represents a shape vector, x, y, and z are three-dimensional coordinate values of the head mesh vertex, the plurality of reference points are combined with the shape vector of the human head to complete the head mesh;

[0106] The principal component analysis can also be performed on one or more head meshes to obtain:

[0107]

[0108] In the formula, represents model data, represents an average vector of one or more head vectors, represents a standard deviation, represents an eigenvector of the head mesh, and the principal component analysis on the completed head model can obtain a head model with obvious characteristics, and the completion manner is not limited.

[0109] In an alternative implementation, Figure 3 An affine transformation process schematic diagram of a three-dimensional head deformation model construction method provided by the embodiment of the application, and the specific steps of the step S120 can include:

[0110] S121, obtaining key points of the predefined head template mesh from the predefined head template mesh; and obtaining key points of the face point cloud data from the face point cloud data;

[0111] S122, performing first affine transformation on the key points in the predefined head template mesh and the face point cloud data to obtain a first registration result of the predefined head template mesh and the face point cloud data;

[0112] S123, performing second affine transformation on the face point cloud data and the predefined head template mesh according to the first registration result to obtain a second affine transformation result.

[0113] Optionally, the total number of vertices in the predefined head template mesh is n, and the total number of vertices n is used as the number of outer loops of iterative optimization of the second affine transformation.

[0114] The inner loop optimization iteration of the second affine transformation for each vertex obtains an affine transformation matrix of each vertex;

[0115] The affine transformation matrix corresponding to all vertices is taken as the second affine transformation result;

[0116] The inner loop optimization iteration of the second affine transformation for each vertex obtains an affine transformation matrix of each vertex.

[0117] In an optional implementation, an initial affine transformation matrix is preset, and it is judged whether the vertex belongs to the vertex of the shoulder region. If not, the affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and a loss function. If yes, the initial affine transformation matrix is taken as the affine transformation matrix of the vertex.

[0118] Optionally, the affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and a loss function.

[0119] In an optional implementation, for the affine transformation matrix of the vertex, it is judged whether the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation. If the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, the difference between the current affine transformation result and the previous affine transformation result of the vertex is calculated. It is judged whether the difference between the current affine transformation result and the previous affine transformation result of the vertex is greater than a threshold of the difference between the two affine transformations. If the difference is greater than the threshold of the difference between the two affine transformations, the corresponding point of the vertex is found. Specifically, the gradient descent method can be used to update the affine transformation matrix, and the gradient descent method is as follows:

[0120]

[0121] In the formula, , respectively represent the affine transformation matrix of the i-th vertex at the j-1 time and the j time, represents a learning rate, and the represents a total loss function value. For the vertex of the connecting point of the human head and the body, i.e., the vertex of the shoulder region, the transformation matrix is not updated, so these points can remain unchanged.

[0122] Optionally, the optimization affine transformation matrix between the vertex and the corresponding point of the vertex is solved based on the loss function.

[0123] In this embodiment, the loss function is obtained based on one or more of distance loss, stiffness loss, key point loss, and noise. Specifically, the loss function is calculated based on distance loss, stiffness loss, key point loss, and noise.

[0124] The facial point cloud data and the predefined head template mesh are subjected to a second affine transformation, whereby there is a distance loss between the vertices after the affine transformation and the vertices before the affine transformation;

[0125] In an optional embodiment, the distance loss function is used to calculate the distance loss , specifically:

[0126]

[0127] Where, Indicates the first vertices, express The affine transformation matrix of Indicates the The closest point of a vertex to the point cloud, represents the distance weight, represents the square of the distance between two vertices, Represents a predefined head template mesh. In an optional implementation, considering that both the predefined head template mesh and the facial point cloud data contain areas that are not important for face reconstruction, such as the top of the head, back of the head, neck, and mouth, the following settings can be made for better face reconstruction:

[0128] when When it comes to the facial area, ;when When it is in the top of the head, back of the head, or neck area, ;when In the oral area, The reason why the distance weight of the oral region is set to 0 is that the reconstructed point cloud does not contain the oral cavity.

[0129] The deformation of two adjacent points in the predefined head template mesh follows a similar pattern, described by stiffness loss. Although the vertices in the oral region have no corresponding points in the facial point cloud, due to the stiffness loss, they deform along with the adjacent vertices in the oral region, thus maintaining the coordination of the entire mesh. There is stiffness loss between adjacent vertices in the predefined head template mesh.

[0130] In an optional embodiment, the stiffness loss is calculated using a stiffness loss function , specifically:

[0131]

[0132] Where, represents the edges of the predefined head template grid, 、 express The affine transformation matrix of two adjacent vertices on , represents the square of the F norm;

[0133] During the initial affine transformation phase, the predefined head template mesh and the facial point cloud data differ significantly, making it inaccurate to use the closest point as the matching point. Since key points carry semantic information, the key points of the facial point cloud and the predefined head template mesh are matched, but there is a loss of key points between the predefined head template mesh and the facial point cloud data.

[0134] In an optional embodiment, the key point loss is calculated using a key point loss function , specifically:

[0135]

[0136] The key point loss function Representing the matching relationship between the vertices of the predefined head template mesh and the key points of the point cloud data; represents the vertices of the predefined head template mesh, Representing the key points of the point cloud data;

[0137] The predefined head template grid and facial point cloud data have noise after completing affine transformation;

[0138] In an optional embodiment, the noise is calculated using the Laplace smoothing loss function , specifically:

[0139]

[0140] and Represents the first Vertex, vertices; Represents and arrive The normalized weight related to the distance.

[0141] Optionally, the loss function is obtained based on one or more of distance loss, stiffness loss, key point loss, and noise. Specifically, the loss function is calculated based on distance loss, stiffness loss, key point loss, and noise.

[0142] In an alternative embodiment, the calculation formula is as follows:

[0143]

[0144] wherein the represents the total loss function value, , , respectively are the stiffness loss, the key point loss, and the Laplace smoothing loss weight.

[0145] It can be understood that the outer loop has n times, corresponding to n stiffness weights and key point weights. In order to maintain the stability of the head mesh structure, the stiffness weight is from large to small, so that the mesh is globally aligned, and then locally deformed, and the relationship of the stiffness weight is represented as:

[0146]

[0147] In the formula, represents the stiffness loss weight of the first outer loop, represents the stiffness loss weight of the n-th outer loop;

[0148] Since the accuracy of the key point detection is not high, the key point weight is from large to small, gradually reducing the influence of the key point error, and the relationship of the key point weight is represented as:

[0149]

[0150] In the formula, represents the key point loss weight of the first outer loop, represents the key point loss weight of the n-th outer loop;

[0151] The inner loop finds the optimal affine transformation matrix of the transformation based on the current stiffness weight and the key point weight.

[0152] The affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and the loss function. The initial affine transformation matrix reduces the initial cost of the affine transformation, and the loss function refines the loss that occurs in the affine transformation, accurately calculates the existing loss, and improves the efficiency of the three-dimensional head deformation model construction.

[0153] The technical scheme of the three-dimensional head deformation model construction method provided by the present application will be described below in conjunction with a specific embodiment.

[0154] As Figure 4 shown, the present application embodiment further provides a three-dimensional head deformation model construction device 210. Optionally, the three-dimensional head deformation model construction device can include:

[0155] The acquisition module 211 is configured to acquire face point cloud data of an RGB head image.

[0156] In this embodiment, the acquisition module 211 can be configured to perform Figure 2 The step S110 shown in the figure, the specific description of the acquisition module 211 can refer to the description of the step S110.

[0157] The affine transformation module 212 is configured to perform affine transformation on the face point cloud data and a predefined head template grid to obtain a head grid, wherein the predefined head template grid is a head template grid provided with a plurality of vertices and with a mouth and a shoulder.

[0158] In this embodiment, the affine transformation module 212 can be configured to perform Figure 2 The step S120 shown in the figure, the specific description of the affine transformation module 212 can refer to the description of the step S120.

[0159] The supplement module 213 is configured to complete the head grid to obtain a complete head model.

[0160] In this embodiment, the compensation module 213 can be configured to perform Figure 2 The step S130 shown in the figure, the specific description of the supplement module 213 can refer to the description of the step S130.

[0161] It can be understood that the above-mentioned device embodiments and the above-mentioned method embodiments can correspond to each other, and the similar description of the device embodiments can refer to the method embodiments. To avoid repetition, it will not be described here. The three-dimensional head deformation model construction device provided in the embodiment of the application can execute the three-dimensional head deformation model construction method provided in any embodiment of the application, and has the corresponding function modules and beneficial effects of executing the method. The function modules of the three-dimensional head deformation model construction method device can be realized by hardware, can be realized by software instructions, and can also be realized by a combination of hardware and software modules.

[0162] Specifically, each step of the method embodiment of the application can be completed by the integrated logic circuit of the hardware in the processor and / or the software instruction. The steps of the three-dimensional head model texture generation method in the embodiment of the application can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps in the above-mentioned method embodiment.

[0163] An electronic device 310 is provided in the embodiments of the present application, and a structure of the electronic device 310 is shown in Figure 5 The electronic device 310 can be the server 100 or the terminal 200 shown in the embodiments of the present application. Figure 1 The electronic device 310 can be the server 100 or the terminal 200 shown in the embodiments of the present application.

[0164] As shown in Figure 5 The electronic device 310 includes a memory 311, a processor 312, a communication module 313, and an input / output interface 314, and the like. Optionally, the memory 311, the processor 312, the communication module 313, and the input / output interface 314 can be connected and communicated through a bus 315.

[0165] The memory 311 is configured to store one or more computer programs and transmit codes of the computer programs to the processor 312. When the one or more computer programs are executed by the processor 311, the three-dimensional head model texture generation method in the embodiments of the present application is implemented.

[0166] Optionally, the electronic device 310 can be connected to a network through the communication module 313, so as to communicate with other devices such as terminals or servers through the network, and realize the interaction of data. The electronic device 310 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers, or other types of computers. The electronic device 310 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0167] Optionally, the electronic device 310 can connect the required input / output devices such as keyboards, display devices, etc. through the input / output interface 314. The electronic device 310 itself can have a display device, and can also be externally connected to other display devices through the input / output interface 314. Optionally, the storage device such as a hard disk can also be connected through the input / output interface 314, so as to store the data in the electronic device 310 into the storage device, or read the data in the storage device, and also store the data in the storage device into the memory 311. It can be understood that the input / output interface 314 can be a wired interface or a wireless interface. According to different actual application scenarios, the devices connected with the input / output interface 314 can be a component of the electronic device 310, or an external device connected with the electronic device 310 when needed.

[0168] Optionally, the memory 311 can be a volatile memory and / or a non-volatile memory, the volatile memory can be a random access memory, etc., and the non-volatile memory can be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0169] Optionally, the computer program stored in the processor 311 can be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to complete the method provided by the embodiment. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device 310.

[0170] Optionally, the processor 312 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and any appropriate controller, microcontroller, processor, etc. The processor 312 executes various methods and processes of the embodiment, exemplarily, a three-dimensional head model texture generation method of the embodiment.

[0171] Optionally, the bus 315 can include a channel for transmitting information. The bus 315 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 315 can be divided into an address bus, a data bus, a control bus, etc.

[0172] In an optional implementation, the embodiment further provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 311 of the electronic device 310. When the computer program is executed by the processor 312, one or more steps of the three-dimensional head model texture generation method of the embodiment can be executed.

[0173] Optionally, the computer readable storage medium can be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0174] Obviously, the above embodiments of the present application are only examples for clearly illustrating the technical solutions of the present application, and are not intended to limit the specific embodiments of the present application. Any modification, equivalent replacement and improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A method for constructing a three-dimensional head deformation model, characterized in that: include: Get facial point cloud data of RGB head image; Performing affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and an oral cavity and shoulders; Completing the head mesh to obtain a complete head model; The facial point cloud data is affine transformed with a predefined head template mesh to obtain a head mesh, comprising: Acquire key points of a predefined head template grid from the predefined head template grid; Acquire key points of facial point cloud data from the facial point cloud data; Performing a first affine transformation on the key points in the predefined head template grid and the facial point cloud data to obtain a first registration result of the predefined head template grid and the facial point cloud data; performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result; Obtaining a head mesh according to the second affine transformation result; The performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result specifically includes: Assuming that the total number of vertices in the predefined head template mesh is n, the total number of vertices n is used as the number of outer loops of the iterative optimization of the second affine transformation; For each outer loop, performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex; Taking the affine transformation matrix corresponding to all vertices as the second affine transformation result; The step of performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex specifically includes: Presetting an initial affine transformation matrix, determining whether the vertex belongs to a vertex in the shoulder region, and if not, iteratively updating the affine transformation matrix of the vertex based on the initial affine transformation matrix and a loss function; and if so, using the initial affine transformation matrix as the affine transformation matrix of the vertex; The iterative updating of the affine transformation matrix of the vertex based on the initial affine transformation matrix and the loss function specifically includes: For the affine transformation matrix of the vertex, determining whether the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, and if the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, calculating the difference between the current affine transformation result of the vertex and the previous affine transformation result; Determine whether the difference between the current affine transformation result of the vertex and the previous affine transformation result is greater than a threshold of the difference between the two affine transformations, and if the difference is greater than the threshold of the two affine transformations, find the corresponding point of the vertex; Solving the optimized affine transformation matrix between the vertex and the corresponding point of the vertex based on the loss function; The gradient descent method is used to update the affine transformation matrix. The gradient descent method is as follows: Where, 、 They represent the j-1th and jth affine transformation matrices of the i-th vertex respectively. represents the learning rate, Represents the total loss function value; The loss function is based on the distance loss , stiffness loss , key point loss ,noise The noise is calculated using the Laplace smoothing loss function. , the calculation formula of the loss function is as follows: Among them, the Represents the total loss function value, 、 、 They are the weights of stiffness loss, key point loss, and Laplace smoothing loss respectively; The inner loop finds the affine transformation matrix for the optimal transformation based on the current stiffness weights and keypoint weights.

2. The method for constructing a three-dimensional head deformation model according to claim 1, characterized in that: The predefined head template mesh is a head template mesh provided with a plurality of vertices and having an oral cavity and shoulders, including: The predefined head template grid includes at least an eye region, a mouth region, an oral cavity region and a shoulder region; The plurality of vertices are distributed in a manner such that vertices in the eye and mouth regions are denser than those in other regions of the predefined head template grid.

3. The method for constructing a three-dimensional head deformation model according to claim 1, wherein: The facial point cloud data and the predefined head template mesh are subjected to a second affine transformation, whereby there is a distance loss between the vertices after the affine transformation and the vertices before the affine transformation; The distance loss is calculated using the distance loss function , specifically: Where, Indicates the first vertices, express The affine transformation matrix of Indicates the The closest point of a vertex to the point cloud, represents the distance weight, represents the square of the distance between two vertices, Represents a predefined head template grid; There is stiffness loss between two adjacent vertices of the predefined head template mesh; The stiffness loss is calculated using the stiffness loss function , specifically: Where, represents the edges of the predefined head template grid, 、 express The affine transformation matrix of two adjacent vertices on , represents the square of the F norm; There is a key point loss in the key points of the predefined head template mesh and the facial point cloud data; The key point loss is calculated using the key point loss function , specifically: The key point loss function Representing the matching relationship between the vertices of the predefined head template mesh and the key points of the point cloud data; represents the vertices of the predefined head template mesh, Representing the key points of the point cloud data; The predefined head template grid and facial point cloud data have noise after completing affine transformation; The noise is calculated using the Laplace smoothing loss function , specifically: and Represents the first Vertex, vertices; Represents and arrive The normalized weight related to the distance.

4. A method for constructing a three-dimensional head deformation model according to any one of claims 1 to 3, characterized in that: Completing the head mesh to obtain a complete head model specifically includes: Obtaining a number of reference points from the eye region of the predefined head template mesh; obtaining an eyeball position according to the plurality of reference points; The head mesh is subjected to eyeball completion according to the eyeball positions to obtain a complete head model.

5. A three-dimensional head deformation model construction device, characterized in that: The device comprises: Acquisition module, used to obtain facial point cloud data of RGB head image; an affine transformation module, configured to perform affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and a mouth and shoulders; A supplementing module, configured to complete the head mesh to obtain a complete head model; The facial point cloud data is affine transformed with a predefined head template mesh to obtain a head mesh, comprising: Acquire key points of a predefined head template grid from the predefined head template grid; Acquire key points of facial point cloud data from the facial point cloud data; Performing a first affine transformation on the key points in the predefined head template grid and the facial point cloud data to obtain a first registration result of the predefined head template grid and the facial point cloud data; performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result; Obtaining a head mesh according to the second affine transformation result; The performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result specifically includes: Assuming that the total number of vertices in the predefined head template mesh is n, the total number of vertices n is used as the number of outer loops of the iterative optimization of the second affine transformation; For each outer loop, performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex; Taking the affine transformation matrix corresponding to all vertices as the second affine transformation result; The step of performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex specifically includes: Presetting an initial affine transformation matrix, determining whether the vertex belongs to a vertex in the shoulder region, and if not, iteratively updating the affine transformation matrix of the vertex based on the initial affine transformation matrix and a loss function; and if so, using the initial affine transformation matrix as the affine transformation matrix of the vertex; The iterative updating of the affine transformation matrix of the vertex based on the initial affine transformation matrix and the loss function specifically includes: For the affine transformation matrix of the vertex, determining whether the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, and if the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, calculating the difference between the current affine transformation result of the vertex and the previous affine transformation result; Determine whether the difference between the current affine transformation result of the vertex and the previous affine transformation result is greater than a threshold of the difference between the two affine transformations, and if the difference is greater than the threshold of the two affine transformations, find the corresponding point of the vertex; Solving the optimized affine transformation matrix between the vertex and the corresponding point of the vertex based on the loss function; The gradient descent method is used to update the affine transformation matrix. The gradient descent method is as follows: Where, 、 They represent the j-1th and jth affine transformation matrices of the i-th vertex respectively. represents the learning rate, Represents the total loss function value; The loss function is based on the distance loss , stiffness loss , key point loss ,noise The noise is calculated using the Laplace smoothing loss function. , the calculation formula of the loss function is as follows: Among them, the Represents the total loss function value, 、 、 They are the weights of stiffness loss, key point loss, and Laplace smoothing loss respectively; The inner loop finds the affine transformation matrix for the optimal transformation based on the current stiffness weights and keypoint weights.

6. An electronic device, characterized in that: include: a memory for storing one or more computer programs; The processor implements the method for constructing a three-dimensional head deformable model as claimed in any one of claims 1 to 4 when the one or more computer programs are executed by the processor.

Images