Three-dimensional head deformation model construction method and device, and electronic equipment

Abstract

The invention relates to computer imageology, in particular to a three-dimensional head deformation model construction method and device and electronic equipment. The method comprises the following steps: acquiring face point cloud data of an RGB head image; performing affine transformation according to the face point cloud data and a predefined head template grid to obtain a head grid, wherein the predefined head template grid is a head template grid which is provided with a plurality of vertexes and is provided with an oral cavity and shoulders; and complementing the head grid to obtain a complete head model. According to the embodiment of the invention, the pre-definition of the head template grid is realized, the pre-defined head template grid is utilized to construct the three-dimensional head deformation model, the constructed three-dimensional head deformation model is close to a real object, the application range is expanded, the connection of the head and the digital body is facilitated, the diversity of the three-dimensional head deformation model is increased, and the user experience is improved. The problems existing in an existing three-dimensional head deformation model are solved, and a reference thought is provided for the three-dimensional model reconstruction technology.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and graphics, and more specifically, to a method, device and electronic equipment for constructing a three-dimensional head deformation model. Background Art

[0002] As the computing power of devices increases, the demand for digital products is growing. As an important component of digital humans, the head has attracted much attention. The 3D head deformation model is a statistical model used for 3D reconstruction of the face. By analyzing a large number of 3D face data sets, a model that can flexibly express different facial shapes and textures is established. The 3D head deformation model represents the 3D shape and texture of the face as a linear combination of several basis vectors, and builds a model from its linear combination.

[0003] Problems with current 3D head deformation models include focusing only on the face, which has limitations in practical applications. Summary of the invention

[0004] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a method, device and electronic device for constructing a three-dimensional head deformation model, so as to achieve the effect of completely constructing a three-dimensional head deformation model.

[0005] According to a first aspect of the present application, a method for constructing a three-dimensional head deformation model is provided.

[0006] Get the facial point cloud data of the RGB head image;

[0007] Performing affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and having an oral cavity and shoulders;

[0008] The head mesh is completed to obtain a complete head model.

[0009] The 3D morphable model is constructed based on the predefined head template mesh and facial point cloud data. The constructed model not only focuses on the face, but also includes other parts such as shoulders. It is close to the real object, has a wider range of applications, is conducive to the connection between the head and the digital body, and increases the diversity of the 3D head morphable model.

[0010] Optionally, the predefined head template mesh is a head template mesh provided with a plurality of vertices and having an oral cavity and shoulders, including:

[0011] The predefined head template grid includes at least an eye region, a mouth region, an oral cavity region and a shoulder region;

[0012] The plurality of vertices are distributed in a manner that vertices in the eye and mouth regions are densely distributed than vertices in other regions in the predefined head template mesh.

[0013] The predefined head template grid includes multiple areas such as the eye area, mouth area, oral cavity area and shoulder area, covering a wide range of model types and not only focusing on the face area, reducing limitations in practical applications; the vertices of the eye and mouth areas are densely distributed than other areas in the predefined head template grid, which alleviates the pressure on the video memory to a certain extent and is conducive to the construction of a three-dimensional head model.

[0014] Optionally, performing affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh includes:

[0015] Acquire key points of a predefined head template grid from the predefined head template grid;

[0016] Acquire key points of facial point cloud data from the facial point cloud data;

[0017] Performing a first affine transformation on the key points in the predefined head template grid and the facial point cloud data to obtain a first registration result of the predefined head template grid and the facial point cloud data;

[0018] performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result;

[0019] A head mesh is obtained according to the second affine transformation result.

[0020] A first registration is performed on the key points in the predefined head template grid and the facial point cloud data to complete a rough registration of the predefined head template grid and the facial point cloud data, which provides a basis for the second affine transformation, reduces the loss of registration, and improves the accuracy of constructing the three-dimensional deformation model.

[0021] Optionally, performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result specifically includes:

[0022] Assuming that the total number of vertices in the predefined head template mesh is n, the total number of vertices n is used as the number of outer loops of the iterative optimization of the second affine transformation;

[0023] For each outer loop, performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex;

[0024] Taking the affine transformation matrices corresponding to all vertices as the second affine transformation result;

[0025] The step of performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex specifically includes:

[0026] An initial affine transformation matrix is ​​preset to determine whether the vertex belongs to the vertex in the shoulder area. If not, the affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and the loss function. If it does, the initial affine transformation matrix is ​​used as the affine transformation matrix of the vertex.

[0027] Through the second affine transformation, the spatial position difference between the facial point cloud data is minimized to obtain a more accurate rotation and translation transformation matrix, so as to achieve image registration and ensure that parallel lines remain parallel after transformation. In the reconstruction of the three-dimensional model, images from different perspectives are aligned to a common coordinate system to improve the accuracy of the three-dimensional reconstruction. According to the vertex belonging to the shoulder area, the affine transformation matrix of the vertex is not updated, which is conducive to the connection between the head and shoulder model and the body part.

[0028] Optionally, the iteratively updating the affine transformation matrix of the vertex based on the initial affine transformation matrix and the loss function specifically includes:

[0029] For the affine transformation matrix of the vertex, determine whether the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation; if the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, calculate the difference between the current affine transformation result of the vertex and the previous affine transformation result;

[0030] Determine whether the difference between the current affine transformation result of the vertex and the previous affine transformation result is greater than a threshold of the difference between the two affine transformations, and if the difference is greater than the threshold of the two affine transformations, find the corresponding point of the vertex;

[0031] An optimized affine transformation matrix between the vertex and the corresponding point of the vertex is solved based on the loss function.

[0032] The affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and the loss function. The initial affine transformation matrix reduces the initial cost of the affine transformation, and the loss function refines the loss occurring in the affine transformation, accurately calculates the existing loss, and improves the efficiency of constructing the three-dimensional deformable model.

[0033] Optionally, the loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise;

[0034] The facial point cloud data and the predefined head template mesh are subjected to a second affine transformation, and there is a distance loss between the vertices after the affine transformation and the vertices before the affine transformation;

[0035] The distance loss L is calculated using the distance loss function dist , specifically:

[0036]

[0037] In the formula, v i represents the i-th vertex in the predefined head template mesh, X i represents the affine transformation matrix of vi, u i represents the closest point from the i-th vertex to the point cloud, w i represents the distance weight, Represents the square of the distance between two vertices, V represents the predefined head template mesh;

[0038] There is a stiffness loss between two adjacent vertices of the predefined head template mesh;

[0039] The stiffness loss L is calculated using the stiffness loss function stif , specifically:

[0040]

[0041] Where E represents the edge of the predefined head template grid, X a , X b represents the affine transformation matrix of two adjacent vertices on E, represents the square of the F norm;

[0042] There is a key point loss in the key points of the predefined head template grid and the facial point cloud data;

[0043] The key point loss L is calculated using the key point loss function lan , specifically:

[0044]

[0045] In the key point loss function, L = {(v i1 ,l1),...,(v in , l n )} represents the matching relationship between the vertices of the predefined template mesh and the key points of the point cloud data; v in represents the vertices of the predefined template mesh, l n Representing the key points of the point cloud data;

[0046] The predefined head template grid and facial point cloud data have noise after affine transformation;

[0047] The noise L is calculated using the Laplace smoothing loss function lap , specifically:

[0048]

[0049] v i and v j represents the i-th vertex and j-th vertex on the predefined head template mesh; w ij Indicates and v j to v i The normalized weight related to the distance.

[0050] According to the function, the loss existing in the affine transformation can be solved, the loss generated in the process of building the model can be reduced, and the efficiency of building the three-dimensional head deformable model can be improved.

[0051] Optionally, the loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise. Specifically, the loss function is calculated according to distance loss, stiffness loss, key point loss, and noise, and the calculation formula is as follows:

[0052] L total =L dist +αL stif +βL lan +γL lap

[0053] Among them, the L total Represents the total loss function value, α, β, and γ are the weights of stiffness loss, key point loss, and Laplace smoothing loss, respectively.

[0054] The total loss function value accurately measures the loss generated in the process of building the model, quantifies the loss, and facilitates the formulation of a better optimization plan for the constructed model.

[0055] Optionally, completing the head mesh to obtain a complete head model specifically includes:

[0056] Get several reference points from the eye area of ​​the predefined head template mesh

[0057] Obtaining the eyeball position according to the plurality of reference points;

[0058] The eyeballs of the head mesh are completed according to the eyeball positions to obtain a complete head model.

[0059] The predefined head template grid contains multiple regions, but no specific eyeball model. Completing the head grid can obtain a complete three-dimensional head deformable model, thereby generating a specific and complete head model.

[0060] According to a second aspect of the present application, a three-dimensional head deformation model construction device is provided, the device comprising:

[0061] An acquisition module is used to acquire facial point cloud data of an RGB head image;

[0062] An affine transformation module, used for performing affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and having an oral cavity and shoulders;

[0063] A supplementing module is used to complete the head mesh to obtain a complete head model processing module.

[0064] The three-dimensional head deformation model device, each module is an indispensable part of the whole process, and the modules cooperate with each other. The RGB head image is obtained by the acquisition module to obtain the facial point cloud data of the RGB head image; the facial point cloud data and the predefined head template grid are first registered by the affine transformation module, and the facial point cloud data and the predefined head template grid are second affine transformed according to the first registration result, and the head grid is obtained from the affine transformation module; the head grid is supplemented with eyeballs by the supplement module to obtain a complete three-dimensional head deformation model.

[0065] According to a third aspect of the present application, an electronic device is provided, including:

[0066] a memory for storing one or more computer programs;

[0067] The processor implements the three-dimensional head deformation model method described in the first aspect when the one or more computer programs are executed by the processor.

[0068] The electronic device provides a complete system for constructing a three-dimensional head deformation model, stores images, processes the stored images, and the processed images are then stored in the memory.

[0069] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a three-dimensional head deformation model method described in the first aspect when executed.

[0070] Based on any of the above aspects, the method, device, and electronic device for constructing a three-dimensional head deformation model provided by the embodiments of the present application obtain facial point cloud data of a head image to provide data for constructing a three-dimensional head deformation model. By using a predefined head template grid including at least an eye area, a mouth area, an oral cavity area, and a shoulder area, the vertices on the predefined head template are effectively laid out, which can not only completely build a model, but also well express the details of the original image. By presetting the vertices in the shoulder area, the effect of completely constructing the three-dimensional head deformation model is improved.

[0071] Furthermore, the affine transformation matrix of the vertices belonging to the shoulder region is not updated, and the distribution of the vertices is not affected. The three-dimensional head deformation model established according to the construction method has high precision and small video memory, and is conducive to the connection between the head and the body. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0073] Figure 1 A schematic application scenario diagram of a three-dimensional head deformation model construction method provided in this embodiment.

[0074] Figure 2 A flowchart of a method for constructing a three-dimensional head deformation model provided in this embodiment.

[0075] Figure 3 This embodiment provides a schematic diagram of sub-steps of step S120.

[0076] Figure 4 A schematic diagram of functional modules of a device for constructing a three-dimensional head deformation model provided in this embodiment.

[0077] Figure 5 A schematic diagram of the structure of an electronic device provided by this embodiment. DETAILED DESCRIPTION

[0078] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0079] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0081] At present, some 3D head morphing models have some limitations in facial modeling. Some models only focus on facial features without including independent parts such as eyeballs, mouth and shoulders, which limits their application in a wider range of fields. The vertex distribution of some 3D head morphing models is too dense, which will cause excessive memory usage and affect performance in practical applications. In addition, some 3D head morphing models that include shoulders will also undergo unexpected changes in the shoulders when the basis vector coefficients change, which will cause inconvenience when fusing the head model with the digital body model.

[0082] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0083] For example, a schematic diagram of an application scenario of a three-dimensional deformable model construction method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has the function of processing data to build a model; the terminal device 200 has the function of displaying the completed model and also has the function of processing data to build a model.

[0084] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0085] In one practicable manner, the server 100 and the terminal 200 may respectively execute the three-dimensional head deformation model construction method provided in the embodiment of the present application, or, optionally, the three-dimensional head deformation model construction method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0086] like Figure 2 As shown, this embodiment provides a method for constructing a three-dimensional head deformation model, which may include the following steps:

[0087] S110, obtaining facial point cloud data of the RGB head image;

[0088] The RGB head image is scanned to obtain facial point cloud data, which can clearly reproduce the texture and shape of the RGB head image, facilitate feature extraction of head details, and provide a data basis for building a three-dimensional head model.

[0089] In this embodiment, the facial point cloud data of the RGB head image is obtained by using monocular vision technology, binocular vision technology, multi-eye vision technology, etc., and the facial point cloud data is obtained by scanning the RGB head image.

[0090] In an optional implementation, multi-eye vision technology is used to acquire images of hundreds of heads, including people of all ages and various face shapes, with a substantially equal number of men and women. Multiple cameras surround the heads of the subjects, and the shutter trigger time of the cameras is within 5ms. Methods for scanning the acquired RGB head images to obtain facial point cloud data include but are not limited to the following.

[0091] S120, performing affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and having an oral cavity and shoulders;

[0092] The predefined head template grid includes an eye area, a mouth area, an oral cavity area, a shoulder area, an ear area, etc., and the plurality of areas are combined and matched into a grid;

[0093] The predefined head template grid may be provided with a number of vertices, and the vertices may be randomly arranged or regularly arranged, and there are various arrangement schemes.

[0094] In an optional implementation, the predefined head template grid includes at least an eye area, a mouth area, an oral cavity area and a shoulder area, and the plurality of vertices are distributed so that the vertices in the eye and mouth areas are denser than those in other areas of the predefined head template grid. The method includes but is not limited to this.

[0095] The affine transformation includes one or more of translation transformation, rotation transformation, scale transformation and tilt transformation.

[0096] In this embodiment, before performing affine transformation on the facial point cloud data and the predefined head template mesh, the predefined head template mesh template is normalized so that the three coordinates of the template vertices are in the range of -1 to 1.

[0097] In an optional implementation, one or more of translation transformation, rotation transformation, scale transformation, and tilt transformation are performed on the key points in the predefined head template grid and the facial point cloud data to obtain a first registration result of the predefined head template grid and the facial point cloud data;

[0098] The key points in the predefined head template grid and the facial point cloud data can be obtained by using key point extraction algorithms such as ISS key point detection and SIFT key point detection.

[0099] In an optional implementation, the RGB head image may include one or more, and the corresponding facial point cloud data is obtained by scanning based on one or more RGB head images, wherein when multiple RGB head images are included, heads of various age groups may be included according to actual needs, and the number of male and female heads may also be allocated according to actual needs. Specifically, when the RGB head images include hundreds of images, frontal facial images are selected from the hundreds of scanned RGB head images of natural expressions, and two-dimensional facial key points are detected from these facial images. Three-dimensional key points are obtained based on the internal and external parameters of the scanning camera, and then the points closest to these three-dimensional key points are found in the facial point cloud data as the key points of the facial point cloud. In the predefined head template grid, points with the same semantics as the key points of the facial point cloud are selected as the key points of the predefined head template grid, and the method of obtaining the key points includes but is not limited to this.

[0100] According to the first registration result, one or more of translation transformation, rotation transformation, scale transformation and tilt transformation are performed on the facial point cloud data and the predefined head template mesh to obtain a second affine transformation result to acquire a head mesh.

[0101] S 1 30. Complete the head mesh to obtain a complete head model.

[0102] In this embodiment, the head mesh is completed by using a completion network or based on a model.

[0103] In an optional implementation, a plurality of reference points are obtained from the eye region of the predefined head template mesh, the reference points are determined according to the inner and outer eye corners, the highest point and the lowest point of the eye socket of the eye region, the eyeball position is obtained from the reference points, the eyeball of the head mesh is completed according to the eyeball position to obtain a complete head model, and the eyeballs are installed on hundreds of head meshes to obtain a complete head model, which specifically includes: according to the three-dimensional coordinate values ​​of the vertices of the head mesh, the shape of a human head can be expressed as:

[0104] S=(x1,y1,z1,...,x m， y m , z m ) T

[0105] Wherein, S represents the shape vector, x, y, and z are the three-dimensional coordinate values ​​of the vertices of the head mesh respectively, and the head mesh is completed by combining the several reference points with the shape vector of the human head;

[0106] You can also perform principal component analysis on one or more head meshes to obtain:

[0107] M=(μ,σ,U)

[0108] Where M represents the model data, μ∈R 3m represents the average vector of one or more head vectors, σ∈R 3m Represents the standard deviation, U represents the characteristic vector of the head grid, and performing the principal component analysis on the completed head model can obtain a head model with obvious characteristics, and the completion method is not limited.

[0109] In an optional embodiment, illustratively, Figure 3 A schematic diagram of an affine transformation process of a method for constructing a three-dimensional head deformation model provided in an embodiment of the present application, wherein the specific steps of step S120 may include:

[0110] S121, acquiring key points of a predefined head template grid from the predefined head template grid; acquiring key points of facial point cloud data from the facial point cloud data;

[0111] S122, performing a first affine transformation on the key points in the predefined head template grid and the facial point cloud data to obtain a first registration result of the predefined head template grid and the facial point cloud data;

[0112] S123. Perform a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result.

[0113] Optionally, the total number of vertices in the predefined head template mesh is set to n, and the total number of vertices n is used as the number of outer loops of the iterative optimization of the second affine transformation;

[0114] For each outer loop, performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex;

[0115] Taking the affine transformation matrices corresponding to all vertices as the second affine transformation result;

[0116] The inner loop optimization iteration of the second affine transformation is performed on each vertex to obtain the affine transformation matrix of each vertex.

[0117] In an optional embodiment, an initial affine transformation matrix is ​​preset to determine whether the vertex belongs to the vertex of the shoulder area. If not, the affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and the loss function. If it does, the initial affine transformation matrix is ​​used as the affine transformation matrix of the vertex.

[0118] Optionally, the affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and a loss function.

[0119] In an optional implementation, for the affine transformation matrix of the vertex, determine whether the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation; if the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, calculate the difference between the current affine transformation result of the vertex and the previous affine transformation result; determine whether the difference between the current affine transformation result of the vertex and the previous affine transformation result is greater than the threshold of the difference between the two affine transformations; if the difference is greater than the threshold of the two affine transformations, find the corresponding point of the vertex; specifically, the gradient descent method can be used to update the affine transformation matrix, and the gradient descent method is as follows:

[0120]

[0121] In the formula, They represent the j-1th and jth affine transformation matrices of the i-th vertex, ρ represents the learning rate, and L total Represents the total loss function value. For the connection points between the head and the body, that is, the vertices in the shoulder area, if their transformation matrix is ​​not updated, these points can remain unchanged.

[0122] Optionally, an optimized affine transformation matrix between the vertex and the corresponding point of the vertex is solved based on the loss function.

[0123] In this embodiment, the loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise. Specifically, the loss function is calculated according to distance loss, stiffness loss, key point loss, and noise.

[0124] The facial point cloud data and the predefined head template mesh are subjected to a second affine transformation, and there is a distance loss between the vertices after the affine transformation and the vertices before the affine transformation;

[0125] In an optional implementation, the distance loss function is used to calculate the distance loss L dist , specifically:

[0126]

[0127] In the formula, v i represents the i-th vertex in the predefined head template mesh, X i Indicates v i The affine transformation matrix, u i represents the closest point from the i-th vertex to the point cloud, w i represents the distance weight, Represents the square of the distance between two vertices, V represents the predefined head template mesh;

[0128] In an optional implementation, considering that the predefined head template grid and the facial point cloud data both contain areas that are not important for face reconstruction, such as the top of the head, the back of the head, the neck, and the mouth, the following settings can be made for better face reconstruction:

[0129] When v i When it belongs to the face area, w i =1; when v i When it belongs to the top of the head, the back of the head, and the neck area, w i =0.1; when v i When it belongs to the oral area, w i = 0. The reason for setting the distance weight of the oral cavity area to 0 is that the reconstructed point cloud does not contain the oral cavity.

[0130] The deformation of two adjacent points in the predefined head template mesh has a similar law, and the stiffness loss is used to describe this law. Although the vertices in the oral region have no corresponding points in the facial point cloud, they will deform along with the adjacent vertices in the oral region under the effect of stiffness loss, thereby maintaining the coordination of the entire mesh. There is stiffness loss between two adjacent vertices in the predefined head template mesh;

[0131] In an optional embodiment, the stiffness loss L is calculated using a stiffness loss function. stif , specifically:

[0132]

[0133] Where E represents the edge of the predefined head template grid, X a , X b represents the affine transformation matrix of two adjacent vertices on E, represents the square of the F norm;

[0134] In the initial affine transformation stage, the predefined head template grid and the facial point cloud data are quite different, and it is inaccurate to use the closest point as the matching point. Since the key points have semantic information, the key points of the facial point cloud and the key points of the predefined head template grid are matched, and there is a key point loss between the key points of the predefined head template grid and the facial point cloud data;

[0135] In an optional implementation, the key point loss L is calculated using a key point loss function lan , specifically:

[0136]

[0137] In the key point loss function, L = {(v i1 ,l1),...,(v in , l n )} represents the matching relationship between the vertices of the predefined template mesh and the key points of the point cloud data; v in represents the vertices of the predefined template mesh, l n Representing the key points of the point cloud data;

[0138] The predefined head template grid and facial point cloud data have noise after affine transformation;

[0139] In an optional implementation, the noise L is calculated using the Laplace smoothing loss function. lap , specifically:

[0140]

[0141] v i and v j represents the i-th vertex and j-th vertex on the predefined head template mesh; w ij Indicates and v j to v i The normalized weight related to the distance.

[0142] Optionally, the loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise. Specifically, the loss function is calculated according to distance loss, stiffness loss, key point loss, and noise.

[0143] In an optional implementation, the calculation formula is as follows:

[0144] L total =L dist +αL stif +βL lan +γL lap

[0145] Among them, the L total Represents the total loss function value, α, β, and γ are the weights of stiffness loss, key point loss, and Laplace smoothing loss, respectively.

[0146] It can be understood that the outer loop has n times, corresponding to n stiffness weights and key point weights. In order to maintain the stability of the head mesh structure, the stiffness weights are from large to small, so that the mesh is globally aligned and then locally deformed. The relationship between the stiffness weights is expressed as:

[0147] α 1 >...>α n

[0148] In the formula, α 1 represents the stiffness loss weight of the first outer loop, α n represents the stiffness loss weight of the nth outer loop;

[0149] Since the accuracy of the key point detection is not high, the key point weights are gradually reduced from large to small to gradually reduce the impact of key point errors. The relationship between the key point weights is expressed as:

[0150] β 1 >...>β n

[0151] In the formula, β 1 represents the key point loss weight of the first outer loop, β n represents the key point loss weight of the nth outer loop;

[0152] The inner loop finds the affine transformation matrix for the optimal transformation based on the current stiffness weights and keypoint weights.

[0153] The affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and the loss function. The initial affine transformation matrix reduces the initial cost of the affine transformation, and the loss function refines the loss occurring in the affine transformation, accurately calculates the existing loss, and improves the efficiency of constructing the three-dimensional deformable model.

[0154] The technical solution of the method for constructing a three-dimensional head deformation model provided by the present application is described below in conjunction with a specific embodiment.

[0155] like Figure 4 As shown, the embodiment of the present application further provides a three-dimensional head deformation model construction device 210. Optionally, the three-dimensional head deformation model construction device may include:

[0156] An acquisition module 211 is used to acquire facial point cloud data of an RGB head image;

[0157] In this embodiment, the acquisition module 211 can be used to execute Figure 2 As shown in step S110, for the specific description of the acquisition module 211, reference may be made to the description of step S110.

[0158] The affine transformation module 212 is used to perform affine transformation on the facial point cloud data and the predefined head template mesh to obtain a head mesh. The predefined head template mesh is a head template mesh with a plurality of vertices and an oral cavity and shoulders.

[0159] In this embodiment, the conversion module 212 can be used to perform Figure 2 As shown in step S120, for a detailed description of the conversion module 212, reference may be made to the description of step S120.

[0160] A supplementing module 213 is used to complete the head mesh to obtain a complete head model;

[0161] In this embodiment, the conversion module 213 can be used to perform Figure 2 As shown in step S130 , for a detailed description of the conversion module 213 , reference may be made to the description of step S130 .

[0162] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A three-dimensional head deformation model construction device provided in an embodiment of the present application can execute a three-dimensional head deformation model construction method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the three-dimensional head deformation model construction method device can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0163] Specifically, each step of the method embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor. The steps of the three-dimensional head model texture generation method in combination with the embodiment of the present application can be directly embodied as a hardware encoding processor, or a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, and other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0164] The present application embodiment provides an electronic device 310, whose structure is as follows: Figure 5 The electronic device 310 may be Figure 1 The server 100 or the terminal 200 is shown.

[0165] like Figure 5 As shown, the electronic device 310 includes a memory 311, a processor 312, a communication module 313 and an input / output interface 314, etc. Optionally, the memory 311, the processor 312, the communication module 313 and the input / output interface 314 can be connected and communicated through a bus 315.

[0166] The memory 311 is used to store one or more computer programs and transmit the code of the computer program to the processor 312; when the one or more computer programs are executed by the processor 311, the three-dimensional head model texture generation method in the embodiment of the present application is implemented.

[0167] Optionally, the electronic device 310 can be connected to a network via a communication module 313 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 310 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 310 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0168] Optionally, the electronic device 310 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 314. The electronic device 310 itself can have a display device, and can also be connected to other display devices through the input / output interface 314. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 314, so that data in the electronic device 310 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 311. It can be understood that the input / output interface 314 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 314 can be a component of the electronic device 310, or it can be an external device connected to the electronic device 310 when needed.

[0169] Optionally, the memory 311 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0170] Optionally, the computer program stored in the processor 311 may be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 310.

[0171] Optionally, the processor 312 may be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 312 executes various methods and processes of this embodiment, illustratively, such as a three-dimensional head model texture generation method in an embodiment of the present application.

[0172] Optionally, the bus 315 may include a path for transmitting information. The bus 315 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 315 may be divided into an address bus, a data bus, a control bus, etc.

[0173] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 311 of the electronic device 310. When the computer program is executed by the processor 312, one or more steps of a three-dimensional head model texture generation method of the embodiment of the present application can be performed.

[0174] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0175] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for constructing a three-dimensional head deformation model, characterized in that: include: Get the facial point cloud data of the RGB head image; Performing affine transformation on the facial point cloud data and a predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and having an oral cavity and shoulders; The head mesh is completed to obtain a complete head model.

2. A method for constructing a three-dimensional head deformation model according to claim 1, characterized in that: The predefined head template mesh is a head template mesh provided with a plurality of vertices and having an oral cavity and shoulders, including: The predefined head template grid includes at least an eye region, a mouth region, an oral cavity region and a shoulder region; The plurality of vertices are distributed in a manner that vertices in the eye and mouth regions are densely distributed than vertices in other regions in the predefined head template mesh.

3. The method for constructing a three-dimensional head deformation model according to claim 1, characterized in that: The facial point cloud data is affine transformed with a predefined head template mesh to obtain a head mesh, including: Acquire key points of a predefined head template grid from the predefined head template grid; Acquire key points of facial point cloud data from the facial point cloud data; Performing a first affine transformation on the key points in the predefined head template grid and the facial point cloud data to obtain a first registration result of the predefined head template grid and the facial point cloud data; performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result; A head mesh is obtained according to the second affine transformation result.

4. A method for constructing a three-dimensional head deformation model according to claim 3, characterized in that: The performing a second affine transformation on the facial point cloud data and the predefined head template grid according to the first registration result to obtain a second affine transformation result specifically includes: Assuming that the total number of vertices in the predefined head template mesh is n, the total number of vertices n is used as the number of outer loops of the iterative optimization of the second affine transformation; For each outer loop, performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex; Taking the affine transformation matrices corresponding to all vertices as the second affine transformation result; The step of performing the inner loop optimization iteration of the second affine transformation on each vertex to obtain the affine transformation matrix of each vertex specifically includes: An initial affine transformation matrix is ​​preset to determine whether the vertex belongs to the vertex in the shoulder area. If not, the affine transformation matrix of the vertex is iteratively updated based on the initial affine transformation matrix and the loss function. If it does, the initial affine transformation matrix is ​​used as the affine transformation matrix of the vertex.

5. The method for constructing a three-dimensional head deformation model according to claim 3, characterized in that: The iterative updating of the affine transformation matrix of the vertex based on the initial affine transformation matrix and the loss function specifically includes: For the affine transformation matrix of the vertex, determine whether the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation; if the number of vertices of the vertex is less than the number n of outer loops of the iterative optimization of the second affine transformation, calculate the difference between the current affine transformation result of the vertex and the previous affine transformation result; Determine whether the difference between the current affine transformation result of the vertex and the previous affine transformation result is greater than a threshold of the difference between the two affine transformations, and if the difference is greater than the threshold of the two affine transformations, find the corresponding point of the vertex; An optimized affine transformation matrix between the vertex and the corresponding point of the vertex is solved based on the loss function.

6. A method for constructing a three-dimensional head deformation model according to claim 5, characterized in that: The loss function is obtained according to one or more of distance loss, stiffness loss, key point loss and noise; The facial point cloud data and the predefined head template mesh are subjected to a second affine transformation, and there is a distance loss between the vertices after the affine transformation and the vertices before the affine transformation; The distance loss L is calculated using the distance loss function dist , specifically: In the formula, v i represents the i-th vertex in the predefined head template mesh, X i Indicates v i The affine transformation matrix, u i represents the closest point from the i-th vertex to the point cloud, w i represents the distance weight, Represents the square of the distance between two vertices, V represents the predefined head template mesh; There is a stiffness loss between two adjacent vertices of the predefined head template mesh; The stiffness loss L is calculated using the stiffness loss function stif , specifically: Where E represents the edge of the predefined head template grid, X a , X b represents the affine transformation matrix of two adjacent vertices on E, represents the square of the F norm; There is a key point loss in the key points of the predefined head template grid and the facial point cloud data; The key point loss L is calculated using the key point loss function lan , specifically: In the key point loss function, L = {(v i1 ,l1),...,(v in , l n )} represents the matching relationship between the vertices of the predefined template mesh and the key points of the point cloud data; v in represents the vertices of the predefined template mesh, l n Representing the key points of the point cloud data; The predefined head template grid and facial point cloud data have noise after affine transformation; The noise L is calculated using the Laplace smoothing loss function lap , specifically: v i and v j represents the i-th vertex and j-th vertex on the predefined head template mesh; w ij Indicates and v j to v i The normalized weight related to the distance.

7. A method for constructing a three-dimensional head deformation model according to claim 6, characterized in that: The loss function is obtained according to one or more of distance loss, stiffness loss, key point loss, and noise. Specifically, the loss function is calculated according to distance loss, stiffness loss, key point loss, and noise. The calculation formula is as follows: L total =L dist +αL stif +βL lan +γL lap Among them, the L total Represents the total loss function value, α, β, and γ are the weights of stiffness loss, key point loss, and Laplace smoothing loss, respectively.

8. A method for constructing a three-dimensional head deformation model according to any one of claims 1 to 7, characterized in that: Completing the head mesh to obtain a complete head model specifically includes: Acquire a plurality of reference points from the eye region of the predefined head template grid; Obtaining the eyeball position according to the plurality of reference points; The eyeballs of the head mesh are completed according to the eyeball positions to obtain a complete head model.

9. A three-dimensional head deformation model construction device, characterized in that: The device comprises: An acquisition module is used to acquire facial point cloud data of an RGB head image; An affine transformation module, used for performing affine transformation on the facial point cloud data and the predefined head template mesh to obtain a head mesh, wherein the predefined head template mesh is a head template mesh having a plurality of vertices and having an oral cavity and shoulders; The supplementing module is used to complete the head mesh to obtain a complete head model.

10. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the three-dimensional head deformation model construction method as described in any one of claims 1-8.

Images