Digital human synthesis method, computer program product, device and medium
Through mesh vectorized facial deformation modeling and three-dimensional Gaussian rendering technology, the problem of unreal changes in facial expressions of digital people is solved, efficient and accurate digital people synthesis is achieved, and the expression and sense of reality of digital people are enhanced.
Patent Information
- Application Number
- CN202410822893.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-06-25
AI Technical Summary
Existing digital human synthesis technology is difficult to generate realistic and high-quality facial expressions, especially when dealing with large postures or expression changes, there are problems of distortion and inconsistency in geometric timing.
Using face deformation modeling technology based on mesh vectorization, combined with three-dimensional Gaussian cell microscopic microscopic microscopic microscopic microscopic data, maps the face to three-dimensional standard points in the standard space, and constructs a three-dimensional Gaussian set. Render digital face images from the target perspective to achieve dynamic facial expression changes.
It improves the rendering efficiency and accuracy of digital people, generates dynamic facial expressions with high fidelity, and enhances the authenticity and expressiveness of digital people.
Smart Images

Figure CN118379438B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a digital human synthesis method, computer program product, device and medium. Background Art
[0002] Digital humans are created and used by computer means, and are the comprehensive products of multiple human characteristics, such as appearance characteristics, human performance ability and interaction ability. As digital humans develop rapidly, their application scope is also expanding, gradually forming specific industries, and their business models are also undergoing continuous evolution and diversification.
[0003] In order to improve the realism and expressiveness of digital humans, it is necessary to optimize the facial features of digital humans. However, the facial details of digital humans generated by existing related technologies are average and the simulated expressions are stiff.
[0004] Therefore, there is an urgent need for a method to improve the synthesis quality and performance of digital humans to solve the above technical problems. Summary of the invention
[0005] Based on this, it is necessary to provide a digital human synthesis method, computer program product, device and medium to address the above technical problems.
[0006] In a first aspect, the present application provides a method for synthesizing a digital human, comprising: obtaining three-dimensional point cloud data of a face corresponding to target video data; mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; constructing a three-dimensional Gaussian set according to the three-dimensional standard point, and using the three-dimensional Gaussian set to render a target face image under a target perspective to synthesize and obtain a digital human under the target perspective.
[0007] In some embodiments, before mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to at least one three-dimensional standard point in the standard space according to a preset facial deformation model, the method includes: converting the three-dimensional point cloud data of the face into a continuous grid vectorization model according to a preset three-dimensional reconstruction algorithm; mapping the three-dimensional deformation point to a triangular mesh in the deformation space of the grid vectorization model.
[0008] In some embodiments, the three-dimensional point cloud data of the human face is converted into a continuous mesh vectorization model according to a preset three-dimensional reconstruction algorithm, including: converting the three-dimensional point cloud data of the human face into a continuous mesh vectorization model according to the Phong surface method; wherein the human face in the mesh vectorization model is composed of multiple triangular meshes, and any triangular mesh can be represented by the vertex coordinates of the triangle vertices and the vertex normal vectors of the triangle vertices.
[0009] In some embodiments, at least one three-dimensional deformation point in the three-dimensional point cloud data of the face is mapped to at least one three-dimensional standard point in the standard space according to a preset facial deformation model, including: establishing the facial deformation model according to a first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation point is located in the deformation space and a second rotation matrix corresponding to the triangular mesh where the three-dimensional standard point is located in the standard space; calling the facial deformation model to map the three-dimensional deformation point in the deformation space to the three-dimensional standard point in the standard space; wherein the first rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the deformation space; wherein the second rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the standard space.
[0010] In some embodiments, the facial deformation model is established according to a first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation point is located in the deformation space and a second rotation matrix corresponding to the triangular mesh where the three-dimensional standard point is located in the standard space, including: using the first rotation matrix and the second rotation matrix to establish the facial deformation model, the facial deformation model is:
[0011] ;
[0012] ;
[0013] in, represents the three-dimensional deformation point, d represents the deformation space, represents a three-dimensional standard point, c represents a standard space, Represents a mapping, represents the first rotation matrix, represents the second rotation matrix.
[0014] In some embodiments, the three-dimensional deformation point is mapped to a triangular mesh in the deformation space of the mesh vectorization model, including: obtaining the centroid coordinates of the three-dimensional deformation point located on the triangular mesh and the vertex coordinates and vertex normal vectors of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; mapping the centroid coordinates of the three-dimensional deformation point to a coordinate point on the triangular mesh in the deformation space according to a first preset formula and the vertex coordinates of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; and calculating the normal vector of the three-dimensional deformation point in the deformation space according to a second preset formula and the vertex normal vectors of the fixed vertices of the triangle.
[0015] In some embodiments, the first preset formula is: ;in, represents the three-dimensional deformation point, The index number of the triangle mesh where the three-dimensional deformation point is located, represents the centroid coordinates of the three-dimensional deformation point, Respectively represent the vertex coordinates of the three vertices of the triangular mesh, Indicates mapping the triangle mesh index value and barycentric coordinates to coordinate points on the triangle mesh.
[0016] In some embodiments, the second preset formula is: ;in, represents the normal vector of the 3D deformation point, Represent the vertex normal vectors of the three vertices of the triangle mesh, The normal vector used to calculate the 3D deformation point by interpolation.
[0017] In some embodiments, constructing a three-dimensional Gaussian set based on the three-dimensional standard points includes: taking the three-dimensional standard points as three-dimensional Gaussian points; calculating the mean and covariance matrix of the three-dimensional Gaussian points; and using the mean and covariance matrix to represent the three-dimensional Gaussians in the three-dimensional Gaussian set to construct the three-dimensional Gaussian set.
[0018] In some embodiments, a method for calculating the mean corresponding to a three-dimensional Gaussian point includes: calculating the mean of the three-dimensional Gaussian point according to the position and normal vector of the three-dimensional standard point corresponding to the three-dimensional Gaussian point and a third preset formula, wherein the third preset formula is:
[0019] ;
[0020] in, represents the mean of three-dimensional Gaussian points, represents the normal vector of the 3D deformation point, represents a three-dimensional standard point, c represents a standard space, represents mapping and d represents offset.
[0021] In some embodiments, a method for calculating a covariance matrix corresponding to a three-dimensional Gaussian point includes: calculating a covariance matrix corresponding to the i-th three-dimensional Gaussian point at time t according to a rotation amount and a scale change amount corresponding to the three-dimensional Gaussian point and a fourth preset formula, wherein the fourth preset formula is:
[0022] ;
[0023] in, It is based on the scale change corresponding to the three-dimensional Gaussian point The scale matrix constructed, It is the rotation amount corresponding to the three-dimensional Gaussian point Construct the rotation matrix, the scale change The rotation amount is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space at time t. It is calculated based on the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount.
[0024] In some embodiments, the rotation amount corresponding to the three-dimensional Gaussian point The rotation amount corresponding to the three-dimensional Gaussian point is calculated according to the rotation amount and the centroid interpolation rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t, including: according to the rotation amount and the centroid interpolation rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the fifth preset formula, , the fifth preset formula is:
[0025] ;
[0026] ;
[0027] represents the quaternion corresponding to the three vertices of the triangle mesh where the three-dimensional Gaussian point is located in the deformation space, represents the interpolation rotation of the center of gravity at time t, represents the rotation amount of the three-dimensional Gaussian point in the standard space; wherein the calculation formula of the quaternion is: ;
[0028] Represents any vertex of a triangle mesh The quaternion of Represents the vertices on the triangle mesh The neighborhood triangle set of represents the neighborhood triangle area, A quaternion representing the domain triangle.
[0029] In some embodiments, the scale change The scale change is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the area of the triangular mesh where the three-dimensional Gaussian point is located in the standard space, including: calculating the scale change amount according to the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the area of the triangular mesh where the three-dimensional Gaussian point is located in the standard space and a sixth preset formula , the sixth preset formula is:
[0030] ;
[0031] in, represents the area of the triangular mesh in standard space, represents the area of the triangular mesh in the deformation space, Represents the scale change value in the standard space.
[0032] In some embodiments, a target facial image under a target perspective is rendered using a three-dimensional Gaussian set, including: determining a two-dimensional spatial influence value matching the three-dimensional Gaussian by mapping at least one three-dimensional Gaussian projection in the three-dimensional Gaussian set to a two-dimensional image plane in a differentiable form; calculating the color value of a pixel point in the two-dimensional plane according to the two-dimensional spatial influence value corresponding to at least one of the three-dimensional Gaussians; and rendering the target facial image under the target perspective according to the color value of the pixel point in the two-dimensional plane.
[0033] In some embodiments, the Gaussian parameters corresponding to the three-dimensional Gaussian set are optimized, wherein the optimization process of the Gaussian parameters includes: obtaining three-dimensional point cloud sample data of the face corresponding to the historical video data, wherein the three-dimensional point cloud sample data of the face includes three-dimensional point cloud training data of the face and three-dimensional point cloud verification data of the face; mapping at least one three-dimensional deformation sample point in the three-dimensional point cloud sample data of the face to at least one three-dimensional standard sample point in the standard space according to a preset facial deformation model; constructing a three-dimensional Gaussian set to be trained based on the three-dimensional standard sample points, and using the three-dimensional Gaussian set to be trained to render a face image to be optimized under a target perspective; calculating the reconstruction loss function of the face image to be optimized and the three-dimensional point cloud verification data of the face; taking the reconstruction loss function as the target, performing parameter optimization training based on error minimization; and alternately optimizing the parameters of each three-dimensional Gaussian in the three-dimensional Gaussian set to be trained through an adaptive density control algorithm until the preset conditions are met.
[0034] In some embodiments, before optimizing the Gaussian parameters, the method further includes: initializing Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set, including: obtaining sparse point cloud data, and using the sparse point cloud data to initialize Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set.
[0035] In some embodiments, obtaining three-dimensional point cloud data of a face corresponding to target video data includes: extracting facial expression parameters of the face corresponding to the target video data according to a preset facial expression detail capture and animation production model; extracting facial key point parameters corresponding to the target video data according to a preset facial key point fitting model; extracting camera parameters and camera posture parameters corresponding to the target video data according to a preset motion recovery structure model; and inputting the extracted facial expression parameters of the face, the facial key point parameters, the camera parameters and the camera posture parameters into a preset three-dimensional face data generation model to obtain three-dimensional point cloud data of the face.
[0036] In a second aspect, the present application provides a digital human synthesis system, the system comprising: a data preparation module, used to obtain three-dimensional face point cloud data corresponding to target video data; a deformation mapping module, used to map at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; an image rendering module, used to construct a three-dimensional Gaussian set based on the three-dimensional standard point, and use the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain a digital human under the target perspective.
[0037] In a third aspect, the present application provides a computer program product, which, when executed by a processor, implements the steps of the following method: obtaining three-dimensional face point cloud data corresponding to target video data; mapping at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; constructing a three-dimensional Gaussian set based on the three-dimensional standard point, and using the three-dimensional Gaussian set to render a target face image under a target perspective to synthesize and obtain a digital human under the target perspective.
[0038] In a fourth aspect, the present application provides an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, performing the following operations: obtaining three-dimensional face point cloud data corresponding to target video data; mapping at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; constructing a three-dimensional Gaussian set based on the three-dimensional standard points, and using the three-dimensional Gaussian set to render a target face image under a target perspective to synthesize a digital human under the target perspective.
[0039] In a fifth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program enables the computer to perform the following operations: obtain three-dimensional face point cloud data corresponding to the target video data; map at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; construct a three-dimensional Gaussian set based on the three-dimensional standard point, and use the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain a digital human under the target perspective.
[0040] The beneficial effects achieved by the present application are as follows: the present application provides a method for synthesizing a digital human, including obtaining three-dimensional face point cloud data corresponding to target video data; mapping at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; constructing a three-dimensional Gaussian set according to the three-dimensional standard point, and using the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain a digital human under the target perspective. The present application is based on the facial deformation modeling technology of grid vectorization, combined with the three-dimensional Gaussian fragment differentiable rendering technology, to synthesize a high-fidelity digital human with dynamic facial expression changes, further improving the rendering efficiency and accuracy of the digital human. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0042] Figure 1 is a schematic diagram of a digital human synthesis method provided in an embodiment of the present application;
[0043] Figure 2 It is a three-dimensional Gaussian parameter optimization flow chart provided in an embodiment of the present application;
[0044] Figure 3 This is a digital human synthesis system architecture diagram provided by an embodiment of the present application;
[0045] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like in the entire specification and claims should be interpreted as inclusive rather than exclusive or exhaustive; that is, the meaning of "including but not limited to".
[0048] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.
[0049] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.
[0050] As the background technology, digital human (Head Avatar) synthesis is a basic computer graphics technology, which can be widely used in many practical scenarios such as virtual anchors, remote meetings, film production, video games, digital assistants, etc. However, synthesizing digital humans with realism and expressiveness is a very challenging task. The technical difficulty lies not only in the real-time synchronization of the mouth and the details of geometric dynamic changes such as eye contact, but also in the under-constrained optimization problem caused by the lack of 3D supervision information.
[0051] Existing digital human synthesis technology solutions are a hot research field, and the main technical solutions can be divided into three categories: image-based models, implicit models, and explicit models. Among them, image-based models do not rely on any expression in 3D space, and use a deformation field to transform an image to match a new posture or expression; or use a codec structure, the encoder extracts an identification code from a given source image, and the decoder synthesizes the output image based on this identification code and input features. The decoder may rely on facial key points, facial contours and other information. Although image-based models can synthesize high-quality results, distortion may occur when dealing with large posture or expression changes, and because the deformation information of the three-dimensional surface is extracted from the two-dimensional image, the digital human synthesized by the image-based model still lacks geometric and temporal consistency.
[0052] Among them, the digital human synthesis technology method based on implicit models usually uses implicit surface functions such as signed distance functions (SDF) or voxel representation to achieve the synthesis of digital humans. The digital human synthesis technology solution based on implicit models is to represent the human face as a discrete implicit feature voxel grid to synthesize dynamic transformations. In addition, the combination of neural radiance field (NeRF) and voxel rendering is also a technical method that synthesizes digital speakers by utilizing low-dimensional parameters of the facial model or audio signals. Although the digital human synthesis technology method based on implicit models can solve the consistency problems of spatial geometry and timing to a certain extent, it is limited by the reconstruction of static scenes and is difficult to derive to expressions or postures that have not appeared.
[0053] Among them, the digital human synthesis technology method based on explicit models is mainly implemented by explicit triangular mesh feature representation; at this time, the deformation model parameters are used as prior information to reconstruct the facial features of the digital speaker from incomplete (partially occluded) or noisy data (depth map); the explicit deformation model is constructed by fitting from a series of three-dimensional head scans to provide statistical information on facial shape, movement expression and geometric texture. In addition, in the technical field of digital human synthesis based on display models, generative adversarial networks (GAN) are also used to synthesize and optimize digital humans, as well as two-dimensional neural rendering methods to synthesize and optimize digital humans. Although the above schemes can synthesize digital humans with geometric consistency and easy editing and control, it is understandable that the above schemes are limited to the skull structure and cannot synthesize hair information, or the loose constraints of the trunk geometry lead to time and space inconsistencies.
[0054] On this basis, for the new perspective synthesis task of digital human, the existing technology proposes a digital human reconstruction synthesis scheme based on dynamic neural radiance fields (Dynamic Neural Radiance Fields for Monocular 4D FacialAvatar Reconstruction, NeRFace) to synthesize new expressions and postures of digital human. The digital human reconstruction synthesis scheme based on dynamic neural radiance fields is based on a given video containing a human face and a background picture without a human face as the model input, and is based on 3D Morphable Models (3DMM) to track and estimate facial expressions. Then, based on the estimated facial posture and expression, a volume rendering method is used to synthesize new perspectives and new expressions of the face. Among them, the sampling point information on the perspective ray and the learnable frame hidden code (Per-frame Learnable Codes) can be input into the dynamic neural radiance field together to realize the training of the model, and finally the trained model is used to output the final synthesized color and density. However, the background in the digital human reconstruction synthesis scheme based on dynamic neural radiance fields must be static, and the color of the last sample on each ray defaults to the corresponding value of the background color; and the digital human reconstruction synthesis scheme cannot synthesize eye movements and blinks, and the rendering is slow.
[0055] In this field, there is also a digital human face geometry and appearance modeling scheme (Neuralhead Avatars, NHA) based on monocular video to achieve digital human synthesis, in which a hybrid representation method is used to first roughly estimate the low-dimensional shape, expression and posture parameters in the FLAME model based on a real-time face tracker; then a geometry network is used to modify the geometry mesh generated by the FLAME model, and then rasterization is performed; then a texture network is used to synthesize the mesh texture of the rasterized surface; finally, the parameters of the FLAME model and the neural network are jointly optimized based on a differentiable optimization method. Although this scheme can generate avatars with high texture details when a large perspective flip occurs, a certain degree of distortion will still occur when the mouth movement expression changes significantly.
[0056] Different from all the above technical solutions, in order to further improve the synthesis quality and performance of digital humans, this application mainly proposes a facial deformation modeling technology based on grid vectorization, combined with three-dimensional Gaussian fragment differentiable rendering technology, to synthesize high-fidelity digital humans with dynamic facial expression changes, further improving the rendering efficiency and accuracy of digital humans.
[0057] Embodiment 1
[0058] The embodiment of the present application provides a method for synthesizing a digital human. Specifically, the method disclosed in the embodiment of the present application is used to realize the synthesis of a digital human under a new perspective, including the following contents: S1. Acquire three-dimensional point cloud data of a face corresponding to target video data.
[0059] The above-mentioned target video data is specifically video data containing faces, such as monocular RGB (Red Green Blue color mode) face video or selfie face video, or it can be an image data set composed of multiple face images. The form of the above-mentioned target video data does not affect the implementation of the present invention. Further, the embodiment of the present application also proposes to obtain target video data containing faces with dynamic facial expression changes according to a preset frame sampling frequency, wherein the preset frame sampling frequency is defined by the staff according to the actual scene and needs, and the present application does not limit this; it can be understood that by sampling any form of video data containing faces at a preset frame sampling frequency, target video data containing faces can be obtained for later processing. For example, the news anchor's speech video can be sampled at a frame rate of 25FPS to obtain target video data containing the news anchor's face.
[0060] Furthermore, in order to synthesize a digital human corresponding to the face contained in the target video data, it is necessary to process the target video data to obtain the three-dimensional point cloud data of the face, so as to facilitate the subsequent use of the three-dimensional point cloud data of the face to synthesize the digital human from the target perspective.
[0061] In a specific implementation scenario, the above-mentioned acquisition of human face three-dimensional point cloud data corresponding to the target video data includes: extracting human face facial expression parameters corresponding to the target video data according to a preset facial expression detail capture and animation production model; extracting facial key point parameters corresponding to the target video data according to a preset facial key point fitting model; extracting camera parameters and camera posture parameters corresponding to the target video data according to a preset motion recovery structure model; and inputting the extracted human face facial expression parameters, facial key point parameters, camera parameters and camera posture parameters into a preset facial statistical model to obtain human face three-dimensional point cloud data.
[0062] Among them, the above-mentioned preset three-dimensional face data generation model is any trained model that can generate three-dimensional point cloud data of the face according to the input facial expression parameters, facial key point parameters, camera parameters and camera posture parameters. The above-mentioned three-dimensional face data generation model is used to perform three-dimensional face recognition and construct a three-dimensional face model, and can output three-dimensional face data. Those skilled in the art can flexibly select a suitable three-dimensional face model that can achieve the above-mentioned effect according to actual conditions. This application does not limit the specific type of the above-mentioned three-dimensional face data generation model. In one specific embodiment, the above-mentioned three-dimensional face data generation model can be a trained facial statistical model (FLAME, Faces Learned with an Articulated Model and Expressions), and the facial statistical model outputs a three-dimensional face mesh by inputting a parameter vector including shape parameters, posture parameters and expression parameters that can be used to control the identity features of the face, the rotation and translation of the head, and the expression changes of the face. The three-dimensional face mesh is obtained by adding the template mesh and the mixed shape function and transforming it through the skin function. It can be used to represent different face shapes and postures, and can also be used to fit three-dimensional scanning data or two-dimensional image data. It has a good face representation effect and is conducive to improving the synthesis accuracy of digital humans.
[0063] Among them, the Detailed Expression Capture and Animation (DECA) model is a three-dimensional face modeling model based on artificial neural networks. It obtains detailed facial expression information based on deep learning algorithms, and then applies the obtained detailed facial expression information to the expression and actions of virtual characters. The Detailed Expression Capture and Animation model can capture and restore various muscle movements and micro-expressions of the human face by collecting details of human facial expressions, realize efficient encoding and reconstruction of facial features, and further realize high-accuracy expression capture. It is understandable that if further animation synthesis is required, the Detailed Expression Capture and Animation model can also apply the captured facial expression information to the virtual character, so that it can show rich facial expressions and actions in a realistic way. The landmark fitting model detects facial recognition points first and then matches the facial recognition points to the three-dimensional face model. The structure from motion model (Structure from motion) analyzes the image sequence, sequentially performs image feature extraction, estimates the initial structure of the scene and camera motion, optimizes the estimated results, calibrates the camera, obtains a dense description of the scene, derives the geometry, texture, and reflection properties of the scene, obtains camera parameters, and performs three-dimensional reconstruction. The embodiment of the present application utilizes the structure from motion model to extract the camera parameters and camera attitude parameters corresponding to the target video data, which greatly improves the accuracy of the acquired camera parameters and camera attitude parameters, and facilitates improving the accuracy of subsequent data processing.
[0064] S2. Mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to a three-dimensional standard point in a standard space according to a preset facial deformation model.
[0065] It is understandable that since a person's facial expressions and postures change dynamically over time, it is necessary to build a facial deformation model to map the three-dimensional deformed point to the three-dimensional canonical point in the standard space, and then perform rendering and synthesis in the standard space.
[0066] In some specific implementation scenarios, before mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to at least one three-dimensional standard point in the standard space according to the preset facial deformation model, the method includes: converting the three-dimensional point cloud data of the face into a continuous grid vector model according to the preset three-dimensional reconstruction algorithm; mapping the three-dimensional deformation point to a triangular grid in the deformation space of the grid vector model. It is understandable that the above-mentioned preset three-dimensional reconstruction algorithm can also include other methods that can perform three-dimensional reconstruction and realize the establishment of a grid vector model, and the present application does not limit the specific method. Before using the facial deformation model to map the deformation point to the standard point, the present application first converts the three-dimensional point cloud data of the face into a continuous grid vector model, so as to facilitate the use of the characteristics of the grid and the vector to represent the three-dimensional deformation point in the later stage, facilitate the subsequent facial deformation model processing, further speed up the synthesis of digital humans, and improve efficiency.
[0067] Preferably, the above-mentioned conversion of the three-dimensional point cloud data of the face into a continuous mesh vectorization model according to a preset three-dimensional reconstruction algorithm includes: converting the three-dimensional point cloud data of the face into a continuous mesh vectorization model according to the Phong surface method; wherein the face in the mesh vectorization model is composed of multiple triangular meshes, and any triangular mesh can be represented by the vertex coordinates of the triangle vertices and the vertex normal vectors of the triangle vertices. The present application adopts the Phong shading method to convert the three-dimensional point cloud data of the face into a continuous mesh vectorization model. Different from the existing facial deformation modeling method based on linear blend skinning (LBS), the Phong surface in the classic Phong shading method (Phong shading) is used to simulate and model the face, which has the characteristics of high efficiency and ease of use, and improves the compatibility of the digital human subsequently synthesized by the present application, so as to achieve compatibility of the synthesized digital human with the existing animation control process.
[0068] Furthermore, in a specific implementation scenario, the above-mentioned mapping of the three-dimensional deformation point to the triangular mesh in the deformation space of the mesh vectorization model includes: obtaining the centroid coordinates of the three-dimensional deformation point located on the triangular mesh and the vertex coordinates and vertex normal vectors of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; mapping the centroid coordinates of the three-dimensional deformation point to the coordinate point on the triangular mesh in the deformation space according to a first preset formula and the vertex coordinates of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; and calculating the normal vector of the three-dimensional deformation point in the deformation space according to a second preset formula and the vertex normal vectors of the fixed vertices of the triangle. According to the properties of the Phong surface, the triangular mesh located on the Phong surface points within , and its centroid coordinates are , then the position and normal vector can be represented by the triangle vertices and the vertex normal vector It is represented by linear interpolation.
[0069] Specifically, the first preset formula is: ;in, represents the three-dimensional deformation point, The index number of the triangle mesh where the 3D deformation point is located. represents the barycentric coordinates of the three-dimensional deformation point, Respectively represent the vertex coordinates of the three vertices of the triangular mesh, Indicates mapping the triangle mesh index value and the centroid coordinates to the coordinate points on the triangle mesh. The second preset formula is: ;in, represents the normal vector of the 3D deformation point, Represent the vertex normal vectors of the three vertices of the triangle mesh, The application uses the characteristics of the grid to first represent the three-dimensional deformation points in the three-dimensional face data as the content related to the triangles on the grid surface, and then uses the facial deformation model to map the three-dimensional deformation points to three-dimensional standard points, which is convenient for calculation and later calculation of the mean and covariance matrix to represent the three-dimensional Gaussian points.
[0070] Further, in a specific embodiment, in the vector The formed first-order continuous space can be used to approximate the mesh surface. The above-mentioned mapping of at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to at least one three-dimensional standard point in the standard space according to the preset facial deformation model includes: establishing a facial deformation model according to a first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation point is located in the deformation space and a second rotation matrix corresponding to the triangular mesh where the three-dimensional standard point is located in the standard space, calling the facial deformation model, and mapping the three-dimensional deformation point in the deformation space to the three-dimensional standard point in the standard space; wherein the first rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the deformation space; wherein the second rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the standard space.
[0071] Among them, the facial deformation model is established according to the first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation point is located in the deformation space and the second rotation matrix corresponding to the triangular mesh where the three-dimensional standard point is located in the standard space, including: using the first rotation matrix and the second rotation matrix to establish the facial deformation model, the facial deformation model is:
[0072] ;
[0073] ;
[0074] in, represents the three-dimensional deformation point, d represents the deformation space, represents a three-dimensional standard point, c represents a standard space, Represents a mapping, represents the first rotation matrix, represents the second rotation matrix. That is, the point on the triangular mesh in the standard space at any time and points on the triangular mesh in the deformation space The correspondence between them can be modeled based on the rotation matrix consisting of the tangent, bitangent and normal on the triangular face where the point is located.
[0075] Preferably, the Frenet frame method can be used for implementation. The Frenet frame method is used in many fields, including mathematics, physics, engineering, computer graphics, etc. Frenet frame modeling is an important method for describing the geometric properties of curves and performing spatial transformations. This method is based on the Frenet frame, also known as the curve frame or motion frame, which consists of the tangent vector, the principal normal vector and the binormal vector of the curve. These vectors not only describe the tangent direction of the curve at a certain point and the concave side direction of the curve, but also obtain the binormal vector perpendicular to them through the cross product of the tangent vector and the principal normal vector. The Frenet frame is an orthogonal normalized frame, that is, these three vectors are orthogonal to each other and have a length of 1. This application establishes a facial deformation model through the Frenet frame method, and maps the three-dimensional deformation points in the deformation space to the three-dimensional standard points in the standard space. It only needs to obtain the first rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation points are located in the deformation space and the second rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the corresponding three-dimensional standard points are located in the standard space. The first rotation matrix and the second rotation matrix can be used for modeling, and the three-dimensional deformation points can be mapped to the three-dimensional standard points. The calculation is fast, which ensures the accuracy and reliability of the modeling, and the mapping relationship between the three-dimensional standard points and the three-dimensional deformation points is simply and intuitively displayed, which simplifies the solution problem during subsequent rendering.
[0076] S3. Construct a three-dimensional Gaussian set according to the three-dimensional standard points, and use the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain the digital human under the target perspective.
[0077] It can be understood that three-dimensional reconstruction of the human face and synthesis from new perspectives are essentially the first under-constrained problems due to the complexity and diversity of facial expressions and the shape changes of facial geometry; therefore, in an embodiment of the present application, a three-dimensional Gaussian set is established to simulate facial geometry and problems in order to improve the authenticity of the synthesized digital human.
[0078] Specifically, in some implementation scenarios, the construction of the above three-dimensional Gaussian set includes: using three-dimensional standard points as three-dimensional Gaussian points; calculating the mean and covariance matrix of the three-dimensional Gaussian points; using the mean and covariance matrix to represent the three-dimensional Gaussians in the three-dimensional Gaussian set to construct the three-dimensional Gaussian set. Among them, the three-dimensional standard points used as three-dimensional Gaussian points are three-dimensional standard points mapped by three-dimensional deformation points contained in the three-dimensional point cloud data of the face.
[0079] The method for calculating the mean corresponding to the three-dimensional Gaussian point includes: calculating the mean of the three-dimensional Gaussian point according to the position and normal vector of the three-dimensional standard point corresponding to the three-dimensional Gaussian point and a third preset formula, and the third preset formula is: ;in, represents the mean of three-dimensional Gaussian points, represents the normal vector of the 3D deformation point, represents a three-dimensional standard point, c represents a standard space, Represents mapping, and d represents offset. The present application does not limit the method for determining the offset. Preferably, an offset prediction model can be constructed based on a fully connected neural network to obtain a sample data set of dynamic changes in facial expressions. The sample data set of dynamic changes in facial expressions is the training sample data of the offset prediction model, and its data scale can be flexibly selected according to actual needs. Each training sample data carries a label, which identifies the offset of the facial expression over time. The offset prediction model is trained using the sample data set of dynamic changes in facial expressions until the corresponding model iteration end condition is met; the three-dimensional point cloud data of the face is input into the trained offset prediction model to obtain the offset of the three-dimensional points of the three-dimensional point cloud data of the face under the process of dynamic changes in facial expressions and postures over time.
[0080] The present application utilizes the Phong surface method to construct a vector mesh model, and maps the three-dimensional deformation points to three-dimensional standard points according to the facial deformation model disclosed in the above content. Therefore, the three-dimensional Gaussian constructed based on the three-dimensional standard points can be correspondingly represented as content related to the triangular mesh, providing a simple and efficient method for representing the three-dimensional Gaussian, facilitating the subsequent rendering of the target face image, and further improving the effect and efficiency of subsequent digital human synthesis.
[0081] Among them, the calculation method of the covariance matrix corresponding to the three-dimensional Gaussian point includes: calculating the covariance matrix corresponding to the i-th three-dimensional Gaussian point at time t according to the rotation amount and scale change amount corresponding to the three-dimensional Gaussian point and the fourth preset formula, and the fourth preset formula is: ;in, It is based on the scale change corresponding to the three-dimensional Gaussian point The scale matrix constructed, It is the rotation amount corresponding to the three-dimensional Gaussian point Constructed rotation matrix, scale change The rotation amount is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space and the area of the triangular mesh where the three-dimensional Gaussian point is located in the standard space at time t. It is calculated based on the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount.
[0082] Among them, the rotation amount corresponding to the three-dimensional Gaussian point The rotation amount corresponding to the three-dimensional Gaussian point is calculated according to the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount. The specific calculation method includes: according to the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount and the fifth preset formula, the rotation amount corresponding to the three-dimensional Gaussian point is calculated. , the fifth preset formula is:
[0083] ;
[0084] ;
[0085] in, The quaternion corresponding to the three vertices of the triangular mesh where the three-dimensional Gaussian point is located in the deformation space, represents the interpolation rotation of the center of gravity at time t, Represents the rotation of a three-dimensional Gaussian point in standard space; among them, the quaternion The calculation formula is: , Represents any vertex of a triangle mesh The quaternion can be based on the quaternion The calculation formula is used to calculate the quaternions corresponding to the three vertices of the triangular mesh where the three-dimensional Gaussian point is located in the deformation space. ; Represents the vertices on the triangle mesh The neighborhood triangle set of represents the neighborhood triangle area, A quaternion representing the domain triangle.
[0086] Among them, the scale change The scale change is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the area of the triangular mesh where the three-dimensional Gaussian point is located in the standard space, specifically including: calculating the scale change amount according to the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the area of the triangular mesh where the three-dimensional Gaussian point is located in the standard space and the sixth preset formula , the sixth preset formula is: ;in, represents the area of the triangular mesh in standard space, represents the area of the triangular mesh in the deformation space, Represents the scale change value in the standard space.
[0087] After constructing a three-dimensional Gaussian set for rendering, the embodiment of the present application also discloses using the constructed three-dimensional Gaussian set to render the target face image under the target perspective, specifically including: mapping at least one three-dimensional Gaussian projection in the three-dimensional Gaussian set to a two-dimensional plane in a differentiable form, determining the two-dimensional spatial influence value matching the three-dimensional Gaussian; calculating the color value of the pixel point in the two-dimensional plane according to the two-dimensional spatial influence value corresponding to at least one three-dimensional Gaussian; rendering the target face image under the target perspective according to the color value of the pixel point in the two-dimensional plane. Mapping the three-dimensional Gaussian projection to the two-dimensional picture plane in a differentiable form improves the accuracy of the facial geometric texture representation of the face, so that the final synthesized digital human has a better facial representation effect and improves the accuracy of digital human synthesis.
[0088] Specifically, the three-dimensional spatial influence value of the three-dimensional Gaussian is calculated according to the three-dimensional position information, opacity value, covariance matrix and mean value corresponding to the three-dimensional Gaussian point, and then the two-dimensional spatial influence value of the corresponding three-dimensional Gaussian is determined by projecting the calculated three-dimensional Gaussian onto a two-dimensional plane; finally, the color value of the pixel point in the two-dimensional plane is calculated according to at least one determined two-dimensional spatial influence value, so as to render the target face image according to the color value of the pixel point in each two-dimensional plane. The three-dimensional spatial influence value is the influence value of each three-dimensional Gaussian on the local area space. After the three-dimensional spatial influence value is obtained, since the three-dimensional Gaussian is used as the weighted value of the color synthesis for the final rendering, in order to calculate the color value of each pixel point in the two-dimensional plane, the three-dimensional spatial influence value needs to be mapped to the two-dimensional plane. When the color value of the pixel point in the two-dimensional plane is determined, the position of each three-dimensional Gaussian can be optimized by gradient flow so as to move to the correct area. When the position and value are determined, the target face image can be obtained.
[0089] In a specific implementation scenario, the three-dimensional spatial influence value of at least one three-dimensional Gaussian may be calculated according to a preset three-dimensional spatial influence value calculation formula, wherein the three-dimensional spatial influence value calculation formula is expressed as: ;
[0090] in represents the three-dimensional spatial influence value of the i-th three-dimensional Gaussian in the three-dimensional Gaussian set at time t, represents a three-dimensional Gaussian, is a point in three-dimensional space, represents the deformation space, is the opacity value of the i-th three-dimensional Gaussian, represents the sigmoid function, Indicated in Moment The mean of three-dimensional Gaussians (the specific calculation method has been disclosed in the above content), Indicated in Moment The mean covariance matrix of Gaussians (the specific calculation method has been disclosed in the above content).
[0091] After calculating the three-dimensional spatial influence value corresponding to the three-dimensional Gaussian, the three-dimensional Gaussian projection is mapped to a two-dimensional plane to determine the two-dimensional spatial influence value of the corresponding three-dimensional Gaussian. This application does not restrict how to map the specific two-dimensional spatial influence value. Preferably, the mean value of the three-dimensional Gaussian fragment can be first and the camera extrinsic matrix Multiply, then perform deep regularization, and finally pass with the internal parameter matrix Multiply to map to a two-dimensional plane, expressed as follows: ,in, is the two-dimensional mean of the three-dimensional Gaussian, and z represents the depth regularization. The formula for projecting the covariance matrix of the three-dimensional Gaussian onto the two-dimensional plane is as follows: , where J is the point projection transformation ( ) is the Jacobian matrix of .
[0092] After mapping the three-dimensional spatial influence value corresponding to the three-dimensional Gaussian to the two-dimensional plane to obtain the two-dimensional spatial influence value corresponding to the three-dimensional Gaussian, the color value of the pixel point in the two-dimensional plane is calculated. The embodiment of the present application discloses a preferred color value calculation method, which specifically includes: sorting each three-dimensional Gaussian according to the corresponding two-dimensional spatial influence value, determining the Gaussian influence value group of each three-dimensional Gaussian; for each three-dimensional Gaussian, weighted calculation is performed on each target three-dimensional Gaussian in the Gaussian influence value group of the current three-dimensional Gaussian, and the color value of each pixel point in the two-dimensional plane by the current three-dimensional Gaussian is obtained. The influence of all three-dimensional Gaussians in the three-dimensional Gaussian set on the pixel points of the two-dimensional plane can be calculated by weighting the Gaussians sorted by depth from front to back. For ease of description, a Gaussian influence value group can be defined, and the Gaussian influence value group is composed of all three-dimensional Gaussians before the current three-dimensional Gaussian. The calculation of the color value of each three-dimensional Gaussian on the pixel point of the two-dimensional plane can be weighted by multiplying the transparency value composed of the influence value of the three-dimensional Gaussian in the two-dimensional space and the Gaussian influence value before the Gaussian.
[0093] As an efficient and convenient implementation method, the present application calculates the color value of a pixel point in a two-dimensional plane according to a preset color calculation formula. The above color calculation formula can be expressed as ,in, is a facial geometry texture scene represented by a three-dimensional Gaussian set, is the color of the i-th three-dimensional Gaussian, is the two-dimensional spatial influence value of the i-th three-dimensional Gaussian in the two-dimensional space, and j represents the j-th target three-dimensional Gaussian in the Gaussian influence value group of the i-th three-dimensional Gaussian.
[0094] After rendering the target face image, the target face image is matched and combined with the three-dimensional face through the corresponding relationship of feature points, and the current three-dimensional face can have corresponding dynamic facial expressions. If the target video data is an image set consisting of multiple continuous images, multiple target face images will be obtained. Matching and combining each target face image with the three-dimensional face can make the final digital human have dynamic facial expression changes, thereby solving the problem that the dynamic expression changes of digital human faces in related technologies are not obvious.
[0095] In addition, in the digital human synthesis method disclosed in the present application, a new target face image under the target perspective is obtained by rendering using a three-dimensional Gaussian set to synthesize a digital human under the target perspective, wherein the three-dimensional Gaussian parameters corresponding to the three-dimensional Gaussian set used are optimized. The present application does not limit the specific optimization method. Preferably, Figure 2 As shown, the embodiment of the present application also provides a method for optimizing Gaussian parameters corresponding to a three-dimensional Gaussian set, including: obtaining human face three-dimensional point cloud sample data corresponding to historical video data, wherein the human face three-dimensional point cloud sample data includes human face three-dimensional point cloud training data and human face three-dimensional point cloud verification data; mapping at least one three-dimensional deformation sample point in the human face three-dimensional point cloud sample data to at least one three-dimensional standard sample point in the standard space according to a preset facial deformation model; constructing a three-dimensional Gaussian set to be trained based on the three-dimensional standard sample points, and using the three-dimensional Gaussian set to be trained to render the human face image to be optimized under the target perspective; calculating the reconstruction loss function of the human face image to be optimized and the human face three-dimensional point cloud verification data; taking the reconstruction loss function as the goal, performing parameter optimization training based on error minimization; and alternately optimizing the parameters of each three-dimensional Gaussian in the three-dimensional Gaussian set to be trained by an adaptive density control algorithm until the preset conditions are met. Among them, the error loss refers to the difference, inconsistency or gap between the human face image to be optimized and the human face three-dimensional point cloud verification data. By continuously optimizing the error loss, the gap between the human face image to be optimized and the human face three-dimensional point cloud verification data is reduced, so that the two tend to be the same. The preset condition is a pre-set iteration cutoff condition, which can be set based on the error loss being less than a preset threshold, or can be set based on a training cycle or training duration, which does not affect the implementation of the present application.
[0096] In addition, before optimizing the parameters corresponding to the three-dimensional Gaussian set, the present application also proposes to initialize the Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set, including: obtaining sparse point cloud data, and using the sparse point cloud data to initialize the Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set. This improves the accuracy of the optimization results of the subsequent Gaussian parameters and further improves the effect of digital human synthesis.
[0097] In order to improve the control accuracy and reliability, and improve the accuracy and reliability of image processing, this embodiment uses an adaptive density control algorithm to control the optimization process. The adaptive density control algorithm is also an adaptive threshold algorithm based on three-dimensional Gaussian density control. It is an adaptive threshold algorithm designed for images with uneven density. It can determine the thresholds of different areas by analyzing the pixel density. The face image to be optimized and the three-dimensional point cloud verification data of the face are divided into several sub-areas, and then the threshold of each sub-area is adaptively determined according to the pixels in the sub-area, which can effectively handle the grayscale differences in different density areas. The parameters of the three-dimensional Gaussian are alternately optimized through the adaptive density control algorithm, including three-dimensional position information, opacity, anisotropic covariance and spherical harmonic coefficients. In the optimization process, a more compact and accurate facial geometric texture representation is generated by continuously adding and removing three-dimensional Gaussians, which solves the problem of insufficient synthesis quality of digital humans and effectively improves the fluency and authenticity of the three-dimensional visual representation of digital humans.
[0098] The process of establishing the three-dimensional Gaussian set to be trained is consistent with the process of establishing the three-dimensional Gaussian set disclosed in the aforementioned content, and the face image to be optimized under the target perspective obtained by rendering according to the three-dimensional Gaussian set to be trained is consistent with the process of rendering the target face image under the target perspective according to the three-dimensional Gaussian set disclosed in the aforementioned content, and this application will not repeat them here. After that, the reconstruction loss function of the face image to be optimized and the face 3D point cloud verification data is calculated. This application discloses a preferred implementation method: constructing a reconstruction loss function based on the Huber function; specifically, after rendering the face image to be optimized from a new perspective, it can be input into the Huber function together with the corresponding original face 3D point cloud verification data, and the Huber-based image reconstruction loss function can be calculated. After the loss function is calculated, the common stochastic gradient descent method can be used for parameter optimization training based on error minimization. The above reconstruction loss function calculation relationship is expressed as:
[0099] ;
[0100] in, Is a weighting factor used to adjust the impact of the loss function. represents the reconstruction loss function, represents the face image to be optimized, Represents face 3D point cloud verification data.
[0101] This application provides a new digital human synthesis method based on a facial deformation model construction method based on grid vectorization and combined with a three-dimensional Gaussian rendering method, which effectively improves the expressiveness of digital humans.
[0102] Embodiment 2
[0103] Corresponding to the above-mentioned embodiment 1, the present application also provides a digital human synthesis system according to an embodiment, such as Figure 3 As shown, specifically including:
[0104] The data preparation module 310 is used to obtain the three-dimensional point cloud data of the face corresponding to the target video data; the deformation mapping module 320 is used to map at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; the image rendering module 330 is used to construct a three-dimensional Gaussian set according to the three-dimensional standard point, and use the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain a digital human under the target perspective.
[0105] In some implementation scenarios, the deformation mapping module 320 is also used to convert the three-dimensional point cloud data of the face into a continuous grid vector model according to a preset three-dimensional reconstruction algorithm; and map the three-dimensional deformation points to a triangular mesh in the deformation space of the grid vector model.
[0106] In some implementation scenarios, the deformation mapping module 320 is also used to convert the three-dimensional point cloud data of the human face into a continuous mesh vectorization model according to the Phong surface method; wherein the human face in the mesh vectorization model is composed of multiple triangular meshes, and any triangular mesh can be represented by the vertex coordinates of the triangle vertices and the vertex normal vectors of the triangle vertices.
[0107] In some implementation scenarios, the deformation mapping module 320 is also used to establish the facial deformation model based on a first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation points are located in the deformation space and a second rotation matrix corresponding to the triangular mesh where the three-dimensional standard points are located in the standard space; calling the facial deformation model to map the three-dimensional deformation points in the deformation space to the three-dimensional standard points in the standard space; wherein the first rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation points are located in the deformation space; wherein the second rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional standard points are located in the standard space.
[0108] In some implementation scenarios, the deformation mapping module 320 is further used to establish the facial deformation model using the first rotation matrix and the second rotation matrix, and the facial deformation model is:
[0109] ;
[0110] ;
[0111] in, represents the three-dimensional deformation point, d represents the deformation space, represents a three-dimensional standard point, c represents a standard space, Represents a mapping, represents the first rotation matrix, represents the second rotation matrix.
[0112] In some implementation scenarios, the deformation mapping module 320 is also used to obtain the centroid coordinates of a three-dimensional deformation point located on a triangular mesh and the vertex coordinates and vertex normal vectors of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; according to a first preset formula and the vertex coordinates of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located, the centroid coordinates of the three-dimensional deformation point are mapped to coordinate points on the triangular mesh in the deformation space; according to a second preset formula and the vertex normal vectors of the fixed vertices of the triangle, the normal vector of the three-dimensional deformation point in the deformation space is calculated.
[0113] The first preset formula is: ;in, represents the three-dimensional deformation point, The index number of the triangle mesh where the three-dimensional deformation point is located, represents the centroid coordinates of the three-dimensional deformation point, Respectively represent the vertex coordinates of the three vertices of the triangular mesh, Indicates mapping the triangle mesh index value and barycentric coordinates to coordinate points on the triangle mesh.
[0114] The second preset formula is: ;in, represents the normal vector of the 3D deformation point, Represent the vertex normal vectors of the three vertices of the triangle mesh, The normal vector used to calculate the 3D deformation point by interpolation.
[0115] In some implementation scenarios, the image rendering module 330 is further used to use the three-dimensional standard points as three-dimensional Gaussian points; calculate the mean and covariance matrix of the three-dimensional Gaussian points; and use the mean and covariance matrix to represent the three-dimensional Gaussians in the three-dimensional Gaussian set to construct a three-dimensional Gaussian set.
[0116] In some implementation scenarios, the image rendering module 330 is further used to calculate the mean of the three-dimensional Gaussian points according to the position and normal vector of the three-dimensional standard point corresponding to the three-dimensional Gaussian point and a third preset formula, and the third preset formula is:
[0117] ;
[0118] in, represents the mean of three-dimensional Gaussian points, represents the normal vector of the 3D deformation point, represents a three-dimensional standard point, c represents a standard space, represents mapping and d represents offset.
[0119] In some implementation scenarios, the image rendering module 330 is further used to calculate the covariance matrix corresponding to the i-th three-dimensional Gaussian point at time t according to the rotation amount and scale change amount corresponding to the three-dimensional Gaussian point and a fourth preset formula, and the fourth preset formula is:
[0120] ;
[0121] in, It is based on the scale change corresponding to the three-dimensional Gaussian point The scale matrix constructed, It is the rotation amount corresponding to the three-dimensional Gaussian point Construct the rotation matrix, the scale change The rotation amount is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space at time t. It is calculated based on the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount.
[0122] In some implementation scenarios, the image rendering module 330 is further used to calculate the rotation amount corresponding to the three-dimensional Gaussian point according to the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount and the fifth preset formula. , the fifth preset formula is:
[0123] ;
[0124] ;
[0125] represents the quaternion corresponding to the three vertices of the triangle mesh where the three-dimensional Gaussian point is located in the deformation space, represents the interpolation rotation of the center of gravity at time t, represents the rotation amount of the three-dimensional Gaussian point in the standard space; wherein the calculation formula of the quaternion is: ;
[0126] Represents any vertex of a triangle mesh The quaternion of Represents the vertices on the triangle mesh The neighborhood triangle set of represents the neighborhood triangle area, A quaternion representing the domain triangle.
[0127] In some implementation scenarios, the image rendering module 330 is further used to calculate the scale change according to the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space and the sixth preset formula , the sixth preset formula is:
[0128] ;
[0129] in, represents the area of the triangular mesh in standard space, represents the area of the triangular mesh in the deformation space, Represents the scale change value in the standard space.
[0130] In some implementation scenarios, the image rendering module 330 is also used to determine the two-dimensional spatial influence value matching the three-dimensional Gaussian by mapping at least one three-dimensional Gaussian projection in the three-dimensional Gaussian set to a two-dimensional image plane in a differentiable form; calculate the color value of the pixel point in the two-dimensional plane according to the two-dimensional spatial influence value corresponding to at least one of the three-dimensional Gaussian; and render the target face image under the target perspective according to the color value of the pixel point in the two-dimensional plane.
[0131] In some implementation scenarios, the system also includes a parameter optimization module (not shown in the figure), which is used to obtain face three-dimensional point cloud sample data corresponding to historical video data, wherein the face three-dimensional point cloud sample data includes face three-dimensional point cloud training data and face three-dimensional point cloud verification data; according to a preset facial deformation model, at least one three-dimensional deformation sample point in the face three-dimensional point cloud sample data is mapped to at least one three-dimensional standard sample point in the standard space; based on the three-dimensional standard sample points, a three-dimensional Gaussian set to be trained is constructed, and the three-dimensional Gaussian set to be trained is used to render the face image to be optimized under the target perspective; the reconstruction loss function of the face image to be optimized and the face three-dimensional point cloud verification data is calculated; with the reconstruction loss function as the target, parameter optimization training is performed based on error minimization; and the parameters of each three-dimensional Gaussian in the three-dimensional Gaussian set to be trained are alternately optimized through an adaptive density control algorithm until the preset conditions are met.
[0132] In some implementation scenarios, the parameter optimization module is also used to initialize the Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set, including: obtaining sparse point cloud data, and using the sparse point cloud data to initialize the Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set.
[0133] In some implementation scenarios, the data preparation module 310 is also used to extract facial expression parameters of the face corresponding to the target video data according to a preset facial expression detail capture and animation model; extract facial key point parameters corresponding to the target video data according to a preset facial key point fitting model; extract camera parameters and camera posture parameters corresponding to the target video data according to a preset motion recovery structure model; and input the extracted facial expression parameters of the face, the facial key point parameters, the camera parameters and the camera posture parameters into a preset three-dimensional face data generation model to obtain three-dimensional point cloud data of the face.
[0134] Embodiment 3
[0135] Corresponding to the above-mentioned embodiment 1 and embodiment 2, the present application also provides a computer program product in an embodiment, which implements the steps of the following method when executed by a processor: obtaining three-dimensional face point cloud data corresponding to the target video data; mapping at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; constructing a three-dimensional Gaussian set based on the three-dimensional standard point, and using the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain a digital human under the target perspective.
[0136] In some implementation scenarios, when the computer program is executed by a processor, it also implements the steps of the following method: converting the three-dimensional point cloud data of the face into a continuous grid vectorized model according to a preset three-dimensional reconstruction algorithm; mapping the three-dimensional deformation points to a triangular mesh in the deformation space of the grid vectorized model.
[0137] In some implementation scenarios, when the computer program is executed by a processor, the steps of the following method are also implemented: converting the three-dimensional point cloud data of the human face into a continuous mesh vectorization model according to the Phong surface method; wherein the human face in the mesh vectorization model is composed of multiple triangular meshes, and any triangular mesh can be represented by the vertex coordinates of the triangle vertices and the vertex normal vectors of the triangle vertices.
[0138] In some implementation scenarios, when the computer program is executed by the processor, the steps of the following method are also implemented: establishing the facial deformation model according to the first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation point is located in the deformation space and the second rotation matrix corresponding to the triangular mesh where the three-dimensional standard point is located in the standard space; calling the facial deformation model to map the three-dimensional deformation point in the deformation space to the three-dimensional standard point in the standard space; wherein the first rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the deformation space; wherein the second rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the standard space.
[0139] In some implementation scenarios, when the computer program is executed by a processor, the steps of the following method are further implemented: using the first rotation matrix and the second rotation matrix to establish the facial deformation model, the facial deformation model being:
[0140] ;
[0141] ;
[0142] in, represents the three-dimensional deformation point, d represents the deformation space, represents a three-dimensional standard point, c represents a standard space, Represents a mapping, represents the first rotation matrix, represents the second rotation matrix.
[0143] In some implementation scenarios, when the computer program is executed by a processor, it also implements the steps of the following method: obtaining the centroid coordinates of a three-dimensional deformation point located on a triangular mesh and the vertex coordinates and vertex normal vectors of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; mapping the centroid coordinates of the three-dimensional deformation point to a coordinate point on the triangular mesh in the deformation space according to a first preset formula and the vertex coordinates of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; and calculating the normal vector of the three-dimensional deformation point in the deformation space according to a second preset formula and the vertex normal vectors of the fixed vertices of the triangle.
[0144] The first preset formula is: ;
[0145] in, represents the three-dimensional deformation point, The index number of the triangle mesh where the three-dimensional deformation point is located, represents the centroid coordinates of the three-dimensional deformation point, Respectively represent the vertex coordinates of the three vertices of the triangular mesh, Indicates mapping the triangle mesh index value and barycentric coordinates to coordinate points on the triangle mesh.
[0146] The second preset formula is: ;
[0147] in, represents the normal vector of the 3D deformation point, Represent the vertex normal vectors of the three vertices of the triangle mesh, The normal vector used to calculate the 3D deformation point by interpolation.
[0148] In some implementation scenarios, when the computer program is executed by a processor, it also implements the steps of the following method: taking the three-dimensional standard points as three-dimensional Gaussian points; calculating the mean and covariance matrix of the three-dimensional Gaussian points; and using the mean and covariance matrix to represent the three-dimensional Gaussians in the three-dimensional Gaussian set to construct a three-dimensional Gaussian set.
[0149] In some implementation scenarios, when the computer program is executed by the processor, the steps of the following method are further implemented: according to the position and normal vector of the three-dimensional standard point corresponding to the three-dimensional Gaussian point and a third preset formula, the mean of the three-dimensional Gaussian point is calculated, and the third preset formula is:
[0150] ;
[0151] in, represents the mean of three-dimensional Gaussian points, represents the normal vector of the 3D deformation point, represents a three-dimensional standard point, c represents a standard space, represents mapping and d represents offset.
[0152] The covariance matrix corresponding to the i-th three-dimensional Gaussian point at time t is calculated according to the rotation amount and scale change amount corresponding to the three-dimensional Gaussian point and the fourth preset formula, and the fourth preset formula is:
[0153] ;
[0154] in, It is based on the scale change corresponding to the three-dimensional Gaussian point The scale matrix constructed, It is the rotation amount corresponding to the three-dimensional Gaussian point Construct the rotation matrix, the scale change The rotation amount is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space at time t. It is calculated based on the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount.
[0155] In some implementation scenarios, when the computer program is executed by the processor, the steps of the following method are further implemented: according to the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount and the fifth preset formula, the rotation amount corresponding to the three-dimensional Gaussian point is calculated. , the fifth preset formula is:
[0156] ;
[0157] ;
[0158] represents the quaternion corresponding to the three vertices of the triangle mesh where the three-dimensional Gaussian point is located in the deformation space, represents the interpolation rotation of the center of gravity at time t, represents the rotation amount of the three-dimensional Gaussian point in the standard space; wherein the calculation formula of the quaternion is: ;
[0159] Represents any vertex of a triangle mesh The quaternion of Represents the vertices on the triangle mesh The neighborhood triangle set of represents the neighborhood triangle area, A quaternion representing the domain triangle.
[0160] In some implementation scenarios, when the computer program is executed by the processor, the steps of the following method are further implemented: calculating the scale change amount according to the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space and the sixth preset formula , the sixth preset formula is:
[0161] ;
[0162] in, represents the area of the triangular mesh in standard space, represents the area of the triangular mesh in the deformation space, Represents the scale change value in the standard space.
[0163] In some implementation scenarios, when the computer program is executed by a processor, the steps of the following method are also implemented: determining a two-dimensional spatial influence value matching the three-dimensional Gaussian by mapping at least one three-dimensional Gaussian projection in the three-dimensional Gaussian set to a two-dimensional image plane in a differentiable form; calculating the color value of the pixel point in the two-dimensional plane according to the two-dimensional spatial influence value corresponding to at least one of the three-dimensional Gaussians; and rendering the target face image under the target perspective according to the color value of the pixel point in the two-dimensional plane.
[0164] In some implementation scenarios, when the computer program is executed by a processor, the steps of the following method are also implemented: obtaining three-dimensional point cloud sample data of a face corresponding to historical video data, wherein the three-dimensional point cloud sample data of a face includes three-dimensional point cloud training data of a face and three-dimensional point cloud verification data of a face; mapping at least one three-dimensional deformation sample point in the three-dimensional point cloud sample data of a face to at least one three-dimensional standard sample point in a standard space according to a preset facial deformation model; constructing a three-dimensional Gaussian set to be trained based on the three-dimensional standard sample points, and using the three-dimensional Gaussian set to be trained to render a face image to be optimized under a target perspective; calculating a reconstruction loss function of the face image to be optimized and the three-dimensional point cloud verification data of a face; taking the reconstruction loss function as a target, performing parameter optimization training based on error minimization; and alternately optimizing the parameters of each three-dimensional Gaussian in the three-dimensional Gaussian set to be trained through an adaptive density control algorithm until preset conditions are met.
[0165] In some implementation scenarios, when the computer program is executed by a processor, it also implements the steps of the following method: initializing Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set, including: obtaining sparse point cloud data, and using the sparse point cloud data to initialize Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set.
[0166] In some implementation scenarios, when the computer program is executed by a processor, the steps of the following method are also implemented:
[0167] According to a preset facial expression detail capture and animation production model, facial expression parameters of the face corresponding to the target video data are extracted; according to a preset facial key point fitting model, facial key point parameters corresponding to the target video data are extracted; according to a preset motion recovery structure model, camera parameters and camera posture parameters corresponding to the target video data are extracted; the extracted facial expression parameters of the face, the facial key point parameters, the camera parameters and the camera posture parameters are input into a preset three-dimensional face data generation model to obtain three-dimensional point cloud data of the face.
[0168] Embodiment 4
[0169] Corresponding to all the above embodiments, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, performing the following operations: obtaining three-dimensional face point cloud data corresponding to target video data; mapping at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; constructing a three-dimensional Gaussian set based on the three-dimensional standard points, and using the three-dimensional Gaussian set to render a target face image under a target perspective to synthesize and obtain a digital human under the target perspective.
[0170] in, Figure 4 The architecture of the electronic device is exemplarily shown, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a bus 430.
[0171] Among them, the processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.
[0172] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 420 can store an operating system 421 for controlling the execution of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operations of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and an icon font processing system 425, etc. can also be stored. The above-mentioned icon font processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.
[0173] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0174] The network interface 414 is used to connect to a communication module (not shown) to achieve communication interaction between the device and other devices. The communication module can achieve communication through a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).
[0175] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410 , the video display adapter 411 , the disk drive 412 , the input / output interface 413 , the network interface 414 , and the memory 420 ).
[0176] In addition, the electronic device 400 can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment, etc.
[0177] It should be noted that, although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.
[0178] It can be seen from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a cloud service end, or a network device, etc.) to execute the methods of various embodiments of the present application or certain parts of the embodiments.
[0179] Embodiment 5
[0180] Corresponding to all the above embodiments, the embodiments of the present application also provide a computer-readable storage medium, characterized in that it stores a computer program, which enables the computer to perform the following operations: obtain three-dimensional face point cloud data corresponding to the target video data; map at least one three-dimensional deformation point in the three-dimensional face point cloud data to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; construct a three-dimensional Gaussian set based on the three-dimensional standard point, and use the three-dimensional Gaussian set to render the target face image under the target perspective to synthesize and obtain a digital human under the target perspective.
[0181] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0182] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A digital human synthesis method, characterized in that: The method comprises: Obtaining three-dimensional point cloud data of a face corresponding to the target video data; Mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the human face to a three-dimensional standard point in a standard space according to a preset facial deformation model, wherein the three-dimensional standard point is represented in the form of a grid and a vector; Constructing a three-dimensional Gaussian set according to the three-dimensional standard points, and using the three-dimensional Gaussian set to render a target face image under a target perspective to synthesize and obtain a digital human under the target perspective; Before mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to at least one three-dimensional standard point in a standard space according to a preset facial deformation model, the method comprises: Converting the three-dimensional point cloud data of the face into a continuous grid vectorized model according to the Feng surface method; Mapping the three-dimensional deformation points to a triangular mesh in the deformation space of the mesh vectorization model; The step of mapping at least one three-dimensional deformation point in the three-dimensional point cloud data of the face to at least one three-dimensional standard point in a standard space according to a preset facial deformation model includes: Establishing the facial deformation model according to a first rotation matrix corresponding to the triangular mesh where the three-dimensional deformation point is located in the deformation space and a second rotation matrix corresponding to the triangular mesh where the three-dimensional standard point is located in the standard space; Calling the facial deformation model to map the three-dimensional deformation points in the deformation space to the three-dimensional standard points in the standard space; The first rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional deformation point is located in the deformation space; The second rotation matrix is a rotation matrix composed of the tangent, bitangent and normal of the triangular mesh where the three-dimensional standard point is located in the standard space; The constructing of a three-dimensional Gaussian set according to the three-dimensional standard points comprises: Taking the three-dimensional standard point as a three-dimensional Gaussian point; Calculating the mean and covariance matrix of the three-dimensional Gaussian points; The three-dimensional Gaussian set is constructed by using the mean and covariance matrix to represent the three-dimensional Gaussian in the three-dimensional Gaussian set; The calculation method of the covariance matrix corresponding to the three-dimensional Gaussian points includes: The covariance matrix corresponding to the i-th three-dimensional Gaussian point at time t is calculated according to the rotation amount and scale change amount corresponding to the three-dimensional Gaussian point and the fourth preset formula, and the fourth preset formula is: ; in, It is based on the scale change corresponding to the three-dimensional Gaussian point The scale matrix constructed, It is the rotation amount corresponding to the three-dimensional Gaussian point Construct the rotation matrix, the scale change The rotation amount is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space at time t. It is calculated based on the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount.
2. The method according to claim 1, characterized in that: The human face in the mesh vectorization model is composed of multiple triangular meshes, and any triangular mesh can be represented by the vertex coordinates of the triangle vertices and the vertex normal vectors of the triangle vertices.
3. The method according to claim 2, characterized in that The facial deformation model is established according to a first rotation matrix corresponding to a triangular mesh where the three-dimensional deformation point is located in the deformation space and a second rotation matrix corresponding to a triangular mesh where the three-dimensional standard point is located in the standard space, including: The facial deformation model is established by using the first rotation matrix and the second rotation matrix. The facial deformation model is: ; ; in, represents the three-dimensional deformation point, d represents the deformation space, represents a three-dimensional standard point, c represents a standard space, Represents a mapping, represents the first rotation matrix, represents the second rotation matrix.
4. The method according to claim 3, characterized in that The mapping of the three-dimensional deformation points to a triangular mesh in the deformation space of the mesh vectorized model comprises: Obtaining the centroid coordinates of a three-dimensional deformation point located on a triangular mesh and the vertex coordinates and vertex normal vectors of the triangle vertices of the triangular mesh where the three-dimensional deformation point is located; According to a first preset formula and vertex coordinates of triangle vertices of the triangular mesh where the three-dimensional deformation point is located, mapping the centroid coordinates of the three-dimensional deformation point to coordinate points on the triangular mesh in the deformation space; According to the second preset formula and the vertex normal vectors of the fixed vertices of the triangle, the normal vectors of the three-dimensional deformation points in the deformation space are calculated.
5. The method according to claim 4, characterized in that The first preset formula is: ; in, represents the three-dimensional deformation point, The index number of the triangle mesh where the three-dimensional deformation point is located, represents the centroid coordinates of the three-dimensional deformation point, Respectively represent the vertex coordinates of the three vertices of the triangular mesh, Indicates mapping the triangle mesh index value and barycentric coordinates to coordinate points on the triangle mesh.
6. The method according to claim 4, characterized in that The second preset formula is: ; in, represents the normal vector of the 3D deformation point, Represent the vertex normal vectors of the three vertices of the triangle mesh, The normal vector used to calculate the 3D deformation point by interpolation.
7. The method according to claim 1, characterized in that The method for calculating the mean corresponding to the three-dimensional Gaussian points includes: The mean of the three-dimensional Gaussian points is calculated according to the position and normal vector of the three-dimensional standard point corresponding to the three-dimensional Gaussian point and a third preset formula, and the third preset formula is: ; in, represents the mean of three-dimensional Gaussian points, represents the normal vector of the 3D deformation point, represents a three-dimensional standard point, c represents a standard space, represents mapping and d represents offset.
8. The method according to claim 1, characterized in that The rotation amount corresponding to the three-dimensional Gaussian point It is calculated based on the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point in the deformation space is located at time t and the centroid interpolation rotation amount, including: According to the rotation amount of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t and the centroid interpolation rotation amount and the fifth preset formula, the rotation amount corresponding to the three-dimensional Gaussian point is calculated. , the fifth preset formula is: ; ; represents the quaternion corresponding to the three vertices of the triangle mesh where the three-dimensional Gaussian point is located in the deformation space, represents the interpolation rotation of the center of gravity at time t, represents the rotation amount of the three-dimensional Gaussian point in the standard space; The calculation formula of quaternion is: ; Represents any vertex of a triangle mesh The quaternion of Represents the vertices on the triangle mesh The neighborhood triangle set of represents the neighborhood triangle area, A quaternion representing the domain triangle.
9. The method according to claim 1, characterized in that: The scale change is obtained according to the change of the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space and the area of the triangular mesh of the three-dimensional Gaussian point in the standard space at time t, including: The scale change is calculated according to the area of the triangular mesh where the i-th three-dimensional Gaussian point is located in the deformation space at time t, the area of the triangular mesh where the three-dimensional Gaussian point is located in the standard space, and the sixth preset formula , the sixth preset formula is: ; in, represents the area of the triangular mesh in standard space, represents the area of the triangular mesh in the deformation space, Represents the scale change value in the standard space.
10. The method according to any one of claims 1 to 9, characterized in that: The method of using a three-dimensional Gaussian set to render a target face image under a target perspective includes: Determine a two-dimensional spatial influence value matching the three-dimensional Gaussian by mapping at least one three-dimensional Gaussian projection in the three-dimensional Gaussian set to a two-dimensional image plane in a differentiable form; Calculate the color value of a pixel point in a two-dimensional plane according to a two-dimensional spatial influence value corresponding to at least one of the three-dimensional Gaussians; The target face image under the target viewing angle is obtained by rendering according to the color values of the pixels in the two-dimensional plane.
11. The method according to claim 10, characterized in that The Gaussian parameters corresponding to the three-dimensional Gaussian set are optimized, wherein the optimization process of the Gaussian parameters includes: Acquire human face three-dimensional point cloud sample data corresponding to the historical video data, wherein the human face three-dimensional point cloud sample data includes human face three-dimensional point cloud training data and human face three-dimensional point cloud verification data; Mapping at least one three-dimensional deformation sample point in the three-dimensional point cloud sample data of the human face to at least one three-dimensional standard sample point in a standard space according to a preset facial deformation model; Constructing a three-dimensional Gaussian set to be trained based on the three-dimensional standard sample points, and rendering the three-dimensional Gaussian set to be trained to obtain a face image to be optimized under a target viewing angle; Calculating the reconstruction loss function of the face image to be optimized and the face three-dimensional point cloud verification data; Taking the reconstruction loss function as the target, parameter optimization training is performed based on error minimization; The parameters of each three-dimensional Gaussian in the three-dimensional Gaussian set to be trained are alternately optimized through an adaptive density control algorithm until a preset condition is met.
12. The method according to claim 11, characterized in that Before optimizing the Gaussian parameters, the method further includes: Initializing Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set, including: Sparse point cloud data is acquired, and Gaussian parameters corresponding to at least one three-dimensional Gaussian in the three-dimensional Gaussian set are initialized using the sparse point cloud data.
13. The method according to claim 10, characterized in that The step of obtaining the three-dimensional point cloud data of a face corresponding to the target video data includes: Extracting facial expression parameters of a human face corresponding to the target video data according to a preset facial expression detail capture and animation model; Extracting facial key point parameters corresponding to the target video data according to a preset facial key point fitting model; Extracting camera parameters and camera posture parameters corresponding to the target video data according to a preset motion recovery structure model; The extracted facial expression parameters, facial key point parameters, camera parameters and camera posture parameters are input into a preset three-dimensional face data generation model to obtain three-dimensional face point cloud data.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 13 are implemented.
15. An electronic device, characterized in that: The electronic device comprises: one or more processors; And a memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the method according to any one of claims 1-13 is executed.
16. A computer-readable storage medium, characterized in that: The computer program stores a computer program, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 13.
