A method for three-dimensional reconstruction of an animatable face based on triangle mesh differentiable rendering

Abstract

The application discloses a kind of based on triangulated mesh differentiable rendering animated face three-dimensional reconstruction method, this method takes single-view or multi-view portrait head portrait video as input, single-frame reconstruction and sequence frame reconstruction are successively carried out, finally output the portrait head portrait basic model of triangular mesh format, face main joint position information and control weight, face expression mixed shape and multiple face color map.The geometric structure of the face three-dimensional model reconstructed by the method is accurate, the color map is high in definition, and the animation can be edited by joint parameters and expression mixing, which can meet the needs of film and television, game industry for face animation model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics, and in particular to an animated 3D face reconstruction method based on triangular mesh differentiable rendering. Background Technology

[0002] 3D facial reconstruction technology aims to create digital facial models and has wide applications in film and animation, video game development, and the emerging field of digital humans. Obtaining animable 3D facial models through scanning and motion capture is costly; therefore, there is a desire to directly reconstruct 3D facial models from 2D photographs and videos. Triangular meshes are the most common 3D geometric representation in computer graphics. Facial models represented by triangular meshes are directly compatible with rasterization rendering processes and are the mainstream geometric representation scheme.

[0003] Existing face triangulation models that use video frames as input include parametric models such as 3DMM and FLAME, and improved parametric models such as NHA and FLAME. The former learns from large datasets of 3D face meshes, applying principal component analysis to decompose the vertex positions of the face mesh into linear spaces representing shape, pose, and expression, and then combines the basis functions of these linear spaces to form a new face model. However, limited by the training data and the finite dimensions of the linear spaces themselves, parametric models lack the ability to represent individual details when reconstructing new sample faces. Improved parametric models use neural networks for targeted training on specific samples, optimizing the linear space parameters of the parametric model to improve the representation of specific samples. While these models offer improvements in geometric and color representation compared to parametric models, they still suffer from limited improvement and tend to average out results. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing an animated 3D face reconstruction method based on triangular mesh differentiable rendering. This approach bypasses the limitations of existing parametric models' parameter space through direct end-to-end mesh vertex coordinate optimization. Simultaneously, it innovatively adds semantic map constraints to promote local geometry optimization, achieving face geometry reconstruction results of higher quality than existing models. This approach designs a step-by-step optimization process from single frame to multiple frames, resulting in a smoother and more stable optimization process that is less prone to getting trapped in local optima or incorrect solutions, achieving high-quality reconstruction results even with a small number of training frames.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] An animable 3D face reconstruction method based on triangular mesh differentiable rendering includes:

[0007] The video of the face to be reconstructed is preprocessed to obtain RGB sequence frame images, face region mask sequence frame images, face region semantic label sequence frame images, video frame camera parameters, and FLAME model parameters for each RGB sequence frame image.

[0008] Take a single RGB sequence frame from the video of the face to be reconstructed as a neutral expression frame, optimize a static single-frame face model using a differentiable rendering method, and obtain the semantic texture of the face at the same time.

[0009] Cluster analysis was performed based on the FLAME model parameters of each RGB sequence image, and several RGB sequence images with large differences in expression were selected as video frames to participate in the training of the reconstructed animated face model.

[0010] Using the optimized static single-frame face model as the initial mesh, an expression triangle mesh is optimized for each video frame participating in the training of the reconstructed animated face model. Then, the shading map shared between different frames is trained, and the expression blending shape and shading map are output to complete the animated 3D face reconstruction of the face video to be reconstructed.

[0011] Furthermore, the semantic tags of the facial region semantic tag sequence frame image include at least the background, skin, left and right eyebrows, left and right eyes, nose, upper and lower lips, inner mouth area, left and right ears, and hair.

[0012] Furthermore, the step of taking a single RGB sequence frame image from the video of the face to be reconstructed as a neutral expression frame, optimizing a static single-frame face model using a differentiable rendering method, and simultaneously obtaining the semantic texture of the face, specifically involves:

[0013] The initial triangular mesh is a neutral facial expression triangular mesh model with the shape, expression, and pose space blending coefficients of the FLAME model all set to 0. At the same time, the texture coordinates of the model vertices and the initial semantic labels are specified.

[0014] Shading map initialization: Albedo map initialization pixel value is 0, Roughness map initialization pixel value is 0.5, Specular map initialization pixel value is 0, and Lightmap initialization pixel value is 1.

[0015] A differentiable renderer is used to render triangular meshes, shading maps, and semantic maps separately through a rasterization process at a specified camera viewpoint, resulting in differentiable RGB images and differentiable semantic images. A loss function is calculated, and its gradient is backpropagated through an automatic differentiation framework to the vertex coordinates of the triangular meshes and the pixel values ​​of the maps, achieving iterative optimization of the triangular meshes and maps to obtain a static single-frame face model. The model optimization is divided into three stages, each incorporating a smoothing loss. The first stage adds a geometric loss, optimizing only the vertex positions of the triangular meshes. The second stage adds additional image loss and shading parameter regularization loss, simultaneously optimizing both the vertex positions of the triangular meshes and the pixel values ​​of the maps. In the first two stages, the semantic maps are derived from the rasterization results of manually labeled vertex semantic tags. The third stage sets the semantic maps to a learnable state and adds an additional semantic map loss to optimize the pixel values ​​of the semantic maps.

[0016] Furthermore, using the optimized static single-frame face model as the initial mesh, an expression triangle mesh is optimized for each video frame participating in the training of the reconstructed animated face model. Then, a shared shading map between different frames is trained, and the expression blending shape and shading map are output to complete the animated 3D face reconstruction of the video to be reconstructed. Specifically:

[0017] The initial triangular mesh is obtained by adding the FLAME pose and expression offset of each video frame that participated in the training of the reconstructed animated face model to the optimized static single-frame face model.

[0018] A differentiable renderer is used to render triangular meshes, shading maps, and semantic maps separately through a rasterization process at a specified camera viewpoint, resulting in differentiable RGB images and differentiable semantic images. A loss function is calculated, and its gradient is backpropagated through an automatic differentiation framework to the vertex coordinates of the triangular meshes and the pixel values ​​of the maps, enabling iterative optimization of the triangular meshes and maps. The model optimization consists of three stages: the first stage adds geometric loss and smoothing loss, optimizing only the facial expression mesh for each frame; the second stage adds additional image loss and shading parameter regularization loss, simultaneously optimizing both the facial expression mesh and the shading map to obtain a blurred map; the third stage uses only image loss and shading parameter regularization loss, optimizing only the shading map.

[0019] Furthermore, multi-resolution hash encoding technology is used to improve texture clarity, resulting in clear textures.

[0020] Furthermore, different video frames independently optimize their respective facial expression meshes while sharing the same shading maps and semantic maps.

[0021] Furthermore, the geometric loss includes: semantic loss, L1 loss of the difference between the semantic label sequence frame image of the face region and the differentiable rendered semantic image; and masking loss, L1 loss of the difference between the mask sequence frame image of the face region and the non-transparent region mask obtained from the differentiable rendered RGB image.

[0022] The image loss includes the L1 loss between the RGB sequence frame image and the differentiable RGB image, and the SSIM loss between the RGB sequence frame image and the differentiable RGB image.

[0023] The smoothing loss includes the Laplacian loss obtained by calculating the neighborhood Laplacian of each vertex and the normal regularization loss obtained by calculating the normal cosine of each adjacent triangle face;

[0024] The shading parameter regularization loss includes: roughness regularization, which constrains roughness to approximate a normal distribution with a mean of 0.5 and a standard deviation of 0.1; specular reflection regularization, which constrains specular reflection in the skin region to approximate a normal distribution with a mean of 0.375 and a standard deviation of 0.166; and light source regularization, which constrains the light color of the lightmap to approximate white light.

[0025] The semantic mapping loss is the L1 loss of differentiable renderable semantic images and semantic label sequence frames of face regions.

[0026] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the described method for animable 3D face reconstruction based on triangular mesh differentiable rendering.

[0027] A storage medium containing computer-executable instructions, which, when executed by a computer processor, implement the described method for animable 3D face reconstruction based on triangular mesh differentiable rendering.

[0028] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the described method for animable 3D face reconstruction based on triangular mesh differentiable rendering.

[0029] The beneficial effects of this invention are that it uses video frames as input to reconstruct a complete 3D facial model with triangular meshes as geometric representation, physically based textures as shading representation, and joint and facial expression hybrid shapes as animation methods. The reconstruction result is geometrically accurate, with clear textures, rich details, and flexible animation editing. Furthermore, it is compatible with existing 3D pipelines in the game and film industries, demonstrating high practical value. Attached Figure Description

[0030] Figure 1This is a flowchart of an animated 3D face reconstruction method based on triangular mesh differentiable rendering;

[0031] Figure 2 This is a flowchart of single-frame reconstruction of an animated 3D face;

[0032] Figure 3 This is a flowchart of the animated 3D face reconstruction sequence frame reconstruction.

[0033] Figure 4 This is a display of the multi-frame reconstruction results of this scheme. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the accompanying drawings.

[0035] This invention relates to an animated 3D face reconstruction method based on triangular mesh differentiable rendering, comprising:

[0036] The video of the face to be reconstructed is preprocessed to obtain RGB sequence frame images, face region mask sequence frame images, face region semantic label sequence frame images, video frame camera parameters, and FLAME model parameters for each RGB sequence frame image.

[0037] Take a single RGB sequence frame from the video of the face to be reconstructed as a neutral expression frame, optimize a static single-frame face model using a differentiable rendering method, and obtain the semantic texture of the face at the same time.

[0038] Cluster analysis was performed based on the FLAME model parameters of each RGB sequence image, and several RGB sequence images with large differences in expression were selected as video frames to participate in the training of the reconstructed animated face model.

[0039] Using the optimized static single-frame face model as the initial mesh, an expression triangle mesh is optimized for each video frame participating in the training of the reconstructed animated face model. Then, the shading map shared between different frames is trained, and the expression blending shape and shading map are output to complete the animated 3D face reconstruction of the face video to be reconstructed.

[0040] The present invention employs a step-by-step optimization process from single frame to multiple frames, resulting in a smoother and more stable optimization process that is less prone to getting trapped in local optima or incorrect solutions. It can still achieve high-quality reconstruction results even with a small number of training frames. The following specific embodiment further illustrates the effectiveness of the present invention:

[0041] like Figure 1 As shown, the present invention provides an animated 3D face reconstruction method based on triangular mesh differentiable rendering, comprising the following steps:

[0042] Step 1: Preprocess the input face video to obtain the following data as input for this reconstruction method: RGB sequence frame images, face region mask sequence frame images, face region semantic label sequence frame images, video frame camera parameters, and FLAME (a face parameter model) model parameters. The acquisition methods for each type of input information are as follows:

[0043] (1.1) RGB sequence frame images: If the video is not in sequence frame image format, convert it to sequence frame images using video processing library tools such as OpenCV or ffmpeg;

[0044] (1.2) Face region masking sequence frame images: If the video background is a single solid color and there are no other interfering objects except for the head area, the background color area is directly removed to obtain the face region mask. Otherwise, the non-background area of ​​the semantic tag sequence frame image obtained in the following step (1.3) is used as the face region mask.

[0045] (1.3) Semantic label sequence frames of face regions: Predict the semantic label of each pixel on the sequence frames using a pre-trained label prediction network (such as the face_parsing open-source model based on the RoITanh-polar Transformer network). The prediction results of this network should be able to distinguish at least the following face regions: background, skin, left and right eyebrows, left and right eyes, nose, upper and lower lips, intraoral region, left and right ears, and hair.

[0046] (1.4) Video frame camera parameters: These need to be provided with the video. If it is a multi-view video, camera parameters need to be provided for each camera position.

[0047] (1.5) FLAME model parameters: Input the RGB sequence frame images into the open-source model DECA to obtain the FLAME model parameters for each frame. In this embodiment, the FLAME model version is 2023.

[0048] Step Two: Reconstructing a Single-Frame Face Model: Take one RGB sequence frame from the video of the face to be reconstructed as the neutral expression frame. In this embodiment, the initial frame of the face video is taken as the neutral expression frame. A static face triangle mesh is optimized using a differentiable rendering method, while simultaneously obtaining the semantic texture of the face. For example... Figure 2 As shown, step two is achieved through the following sub-steps:

[0049] (2.1) Triangular Mesh Initialization: A neutral facial expression triangular mesh model with shape, expression, and pose space blending coefficients all set to 0 is used as the initial triangular mesh. In this embodiment, the texture coordinates of the model vertices and the initial semantic labels are manually specified using the open-source 3D editing software Blender. However, any 3D editing software with UV unwrapping capabilities can perform this step. Texture coordinates should avoid overlap between regions, and a larger texture space area should be allocated to important shading areas such as the face. Initial semantic labels only mark the left and right eyes, left and right ears, and upper and lower lips; all other areas are marked as skin regions.

[0050] (2.2) Texture Initialization: The four textures required for the physically based shading optimization model are the albedo map, roughness map, specular map, and lightmap. The albedo map is initialized with a pixel value of 0, the roughness map with a pixel value of 0.5, the specular map with a pixel value of 0, and the lightmap with a pixel value of 1. This scheme uses physically based rendering (PBR) to perform light integration on the lightmap pixel by pixel. To improve optimization speed, the lightmap size is set relatively small at 32×16 pixels, while the other textures are all 1024×1024 pixels. Let the incident direction of the pixel light source on the lightmap to the model surface be... The direction of launch is The light map is Albedo map is The surface normal of the model is The solid angle coverage of the pixel light source is The diffuse reflection component of color The calculation formula is:

[0051]

[0052] The specular reflection component uses the Cook-Torrance microsurface shading model, assuming... For half-range vectors, This represents the value of the specular reflection map at the model's lighting location. The roughness map values ​​are taken at the model's lighting location, and the specular reflection component of the shading is... The calculation formula is:

[0053]

[0054]

[0055] Where D is the normal distribution function, and its calculation formula is:

[0056]

[0057]

[0058] The geometric occlusion function is calculated using the following formula:

[0059]

[0060]

[0061] For Fresnel terms, For non-metals, the value is 0.04:

[0062]

[0063] (2.3) Model Optimization Based on Differentiable Rendering: The model uses a differentiable renderer to render the triangular mesh, shading, and semantic map into an image from a specified camera viewpoint through a rasterization process. The rasterization process projects the model's triangular faces onto a two-dimensional image space based on the camera parameters and the model's vertex coordinates. The texture coordinates are sampled based on the centroid coordinates of the triangular faces to obtain the texture shading parameters. Finally, the pixel color is obtained through the physically based rendering process in (2.2). Subsequently, image loss, geometric loss, smoothing loss, shading parameter regularization loss, and semantic map loss are calculated. The gradient of the loss function is backpropagated through the gradient of the automatic differentiation framework to the vertex coordinates of the triangular mesh and the pixel value of the texture, thereby achieving iterative optimization of the triangular mesh and texture. In this embodiment, the differentiable rendering framework uses the NVDiffRast framework, and the automatic differentiation framework uses the PyTorch framework, both of which are open source frameworks. The model optimization is divided into three stages, each with a smoothing loss. The first stage adds a geometric loss, optimizing only the vertex positions of the triangular mesh. The second stage adds an image loss and a shading parameter regularization loss, optimizing both the vertex positions of the triangular mesh and the pixel values ​​of the texture. The third stage adds a semantic map loss, optimizing the pixel values ​​of the semantic map. In the first two stages of training, the semantic map comes from the rasterization results of manually labeled vertex semantic tags (the manually specified vertex semantic tags in (2.1) are projected onto the texture space using a rasterization method to obtain the semantic map. In this step, the vertex semantic tags replace the vertex colors in the rasterization process, and the resulting texture pixels are no longer RGB values ​​but semantic vectors). Since it is not easy to distinguish the extension range of areas such as hair, eyebrows, and nose, these areas are uniformly labeled as skin. In the third stage of training, the semantic map is set to a learnable state, mapping the more detailed and accurate semantic information in the semantic tag sequence frames of the face region onto the semantic map. The high-quality semantic map obtained from training is fixed and plays an important guiding role in the geometric optimization process of multi-frame training.

[0064] The following is the design of the loss function for each part:

[0065] Image loss consists of two parts: the first part is the L1 loss, which is the difference between the RGB sequence frame image and the differentiable rendered RGB image; the second part is the SSIM loss, which is the difference between the RGB sequence frame image and the differentiable rendered RGB image.

[0066] Geometric loss consists of two parts. The first part is semantic loss, which is the L1 loss of the difference between the semantic label sequence image of the face region and the differentiable rendered semantic image. The process of differentiable rendering of the semantic image is the same as rendering the RGB image, except that the texture is replaced from the PBR shading texture to the semantic texture. The second part is masking loss, which is the L1 loss of the difference between the face region masking sequence image and the non-transparent region mask obtained from the differentiable rendered RGB image. The rasterization process of the NVDiffRast framework outputs the model triangle index (greater than 0) corresponding to each pixel in the image. Detecting this output can obtain the non-transparent region of the model in the output image, which is equivalent to obtaining the non-transparent region mask obtained from rendering the RGB image.

[0067] Smoothing loss consists of two parts: the first part is the Laplacian loss, calculated by the neighborhood Laplacian of each vertex, to avoid excessive mesh deformation. The calculation formula is as follows:

[0068]

[0069] Summation is performed on all vertices. It is the number of vertices. These are the coordinates of the i-th vertex. It is the number of adjacent vertices of the i-th vertex, and the internal summation is performed on all adjacent vertices of the i-th vertex. The second part is the normal regularization loss, which calculates the cosine of the normal to each adjacent triangular face to avoid overly sharp local structures. The calculation formula is:

[0070]

[0071] Summation of adjacency relationships of triangular faces proceed, where each element This indicates that triangles i and j are adjacent. This represents the size of the adjacency table. For surface normals.

[0072] Coloring parameter regularization loss consists of three parts: the first part is roughness regularization, which constrains the roughness to approximate a normal distribution with a mean of 0.5 and a standard deviation of 0.1. The calculation formula is as follows:

[0073]

[0074] Summation is performed on all pixels; It is the roughness value of the i-th pixel. The first part is the absolute value function. The second part is specular reflection regularization, which constrains the specular reflection of the skin area to approximate a normal distribution with a mean of 0.375 and a standard deviation of 0.166. The calculation formula is:

[0075]

[0076] Summation is performed on all pixels; This is the specular reflection value of the i-th pixel. The third part is light source regularization, which constrains the light color of the light map to be approximately white light. It is obtained by calculating the L1 loss of the actual light map and the light map with all channels uniformly set to the RGB average value.

[0077] Semantic map loss: Used only in the third stage of the optimization process, it is used to learn to divide the semantic map into more refined and accurate parts. It is obtained by calculating the L1 loss of the learned differentiable renderable semantic image and the semantic label sequence frame image of the face region.

[0078] Step 3: Selection of Training Video Frames: The facial expression space parameters of the FLAME model input in Step 1 are analyzed using clustering methods. Video frames with significant differences in facial expressions are selected from multiple video frames as training frames to participate in the construction of the final facial expression mixing shape space. This method differs from common improved parametric models in that it does not use all video frames for training, but only selects representative facial expression frames to construct the facial expression space. The purpose is to reduce unnecessary training and ensure that the generated facial expression space is representative.

[0079] Step 4: Reconstruct an animable face model: Using the neutral expression static mesh trained in Step 2 as the initial mesh, optimize an expression triangle mesh for each video frame participating in the training. Then, train the shared shading map between different frames, outputting the expression blending shape and shading map to complete the animable 3D face reconstruction of the video to be reconstructed. Step 3 is implemented through the following sub-steps:

[0080] (3.1) Triangular mesh and texture initialization: The triangular mesh model generated in step two is given an initial triangular mesh by adding the FLAME pose and expression offset of the current frame. After adding the FLAME pose and expression offset, the initial model is closer to the target state, thus avoiding getting stuck in local optima during the geometric optimization process. The texture initialization method is the same as in (2.2).

[0081] (3.2) Optimization of multi-frame face reconstruction model: The differentiable rendering architecture and loss function of (2.3) are used for optimization. Different video frames independently optimize their own expression meshes, while sharing the same shading map and semantic map. Only the vertex position and shading map are optimized, while the semantic map remains fixed, which plays a role in guiding the optimization of triangular mesh geometry.

[0082] Specifically, the multi-frame phased optimization process is as follows: The multi-frame optimization process is divided into three stages. In the first stage, geometric loss and smoothing loss are added to optimize only the facial expression mesh in each frame. In the second stage, image loss and shading parameter regularization loss are added to optimize both the facial expression mesh and the shading map, resulting in a blurred map. In the third stage, only image loss and shading parameter regularization loss are used to optimize only the shading map.

[0083] Furthermore, multi-resolution hashing encoding is used to improve texture clarity, resulting in sharper textures. The hashing encoding technique, proposed by the InstantNGP algorithm, involves setting a multi-resolution hash grid where each grid point carries a learnable feature vector. When querying pixel color using 3D coordinates, the coordinates are hashed, generating a high-dimensional feature vector which is then mapped to the pixel color. This model uses 2D texture coordinates to query texture pixels, so the algorithm is modified to set the input dimension to 2. In this case, each 2D texture coordinate is mapped to the texture pixel color through a high-dimensional feature vector. During training, the high-resolution hash grid can adaptively learn high-frequency detail information. The source code for this algorithm can be obtained from the open-source GitHub repository tiny-cuda-nn.

[0084] The final model output and its sources are summarized as follows:

[0085] The basic model of the human head in triangular mesh format: obtained as the result of single-frame training in step two;

[0086] The position information and control weights of the main facial joints are: using the pose space parameters and mixing weights of the FLAME model input in step one;

[0087] Facial expression blending shape: obtained as the result of multi-frame training in step four;

[0088] Face shading maps: including albedo maps, roughness maps, and specular reflection maps, are obtained as the result of multi-frame training in step four.

[0089] In this embodiment, the training dataset is the NeResemble dataset, with 15 training views and 1 test view. This invention uses the method mentioned in (3.1) to select 100 frames from the training frames provided by the dataset for training. Baseline model one (flare) and baseline model two (GaussianAvatar) are trained using all training frames. All metric calculations only consider the head region.

[0090] Table 1 compares the results of the multi-frame reconstruction process of this invention with those of other schemes. Compared with the baseline model one flare method, which also uses triangular meshes as geometric representation, the face model reconstructed by this invention has higher geometric and rendering quality, and performs more naturally under extreme expressions. Compared with the baseline model two GaussianAvatar method, which uses 3D Gaussian splashing, although the rendering quality is slightly lower due to the limitations of the triangular mesh itself, this scheme has the advantages of being more adaptable to existing industrial processes and having good consistency in multi-view rendering.

[0091] Table 1

[0092] method This invention Baseline Model 1 Baseline Model 2 Peak signal-to-noise ratio ↑ 28.20 24.78 31.60 Structural similarity index ↑ 0.9740 0.8421 0.9380 Learning to perceive image similarity ↓ 0.07408 0.1225 0.065

[0093] Note: A higher Peak Signal-to-Noise Ratio (PSNR) indicates that the rendered result is closer to the original image; Structural Similarity Index (SSIM) measures the similarity of images in terms of brightness, contrast, and structure, with higher indices indicating greater similarity; Learning Perceptual Image Similarity (LPIPS) is a deep learning-based perceptual similarity that measures the differences in human visual perception of images, with lower indices indicating greater similarity.

[0094] Figure 4 The rendering results of the reconstructed facial model of this invention under several expressions are presented. Each row corresponds to one expression, and each column, from left to right, shows the original image, the rendered model, the model normal map, and the semantic map. Since this invention does not involve modeling the interior of the oral cavity, the rendering result of the oral cavity is blurred. Apart from this, other areas of the face are reconstructed in a relatively detailed and realistic manner, with coloring consistent with the original. Figure 1 It can also simulate the local geometric deformation caused by changes in facial expressions.

[0095] Corresponding to the aforementioned embodiment of an animated 3D face reconstruction method based on triangular mesh differentiable rendering, the present invention also provides an embodiment of an electronic device for implementing the aforementioned animated 3D face reconstruction method based on triangular mesh differentiable rendering.

[0096] An electronic device provided in this invention includes one or more processors for implementing the aforementioned method for animable 3D face reconstruction based on triangular mesh differentiable rendering in the above embodiments.

[0097] The embodiments of the electronic device of the present invention can be applied to any device with data processing capabilities, such as a computer or other equipment or apparatus.

[0098] The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device in which it resides reading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, in addition to the processor, memory, network interface, and non-volatile memory, the data processing device in which the device in the embodiment resides may also include other hardware depending on the actual function of that data processing device, which will not be elaborated further.

[0099] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0100] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0101] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the aforementioned method for animable 3D face reconstruction based on triangular mesh differentiable rendering as described in the above embodiments.

[0102] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0103] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for animable 3D face reconstruction based on triangular mesh differentiable rendering, characterized in that, include: The video of the face to be reconstructed is preprocessed to obtain RGB sequence frame images, face region mask sequence frame images, face region semantic label sequence frame images, video frame camera parameters, and FLAME model parameters for each RGB sequence frame image. A single RGB frame from the video of the face to be reconstructed is taken as a neutral expression frame. A static single-frame face model is optimized using a differentiable rendering method, while simultaneously obtaining the semantic texture of the face. Specifically: The initial triangular mesh is a neutral facial expression triangular mesh model with the shape, expression, and pose space blending coefficients of the FLAME model all set to 0. At the same time, the texture coordinates of the model vertices and the initial semantic labels are specified. Shading map initialization: Albedo map initialization pixel value is 0, Roughness map initialization pixel value is 0.5, Specular map initialization pixel value is 0, and Lightmap initialization pixel value is 1. A differentiable renderer is used to render triangular meshes, shading maps, and semantic maps separately through a rasterization process at a specified camera viewpoint, resulting in differentiable RGB images and differentiable semantic images. A loss function is calculated, and its gradient is backpropagated through an automatic differentiation framework to the vertex coordinates of the triangular meshes and the pixel values ​​of the textures, achieving iterative optimization of the triangular meshes and textures to obtain a static single-frame face model. The model optimization is divided into three stages, each incorporating a smoothing loss. The first stage adds a geometric loss, optimizing only the vertex positions of the triangular meshes. The second stage adds additional image loss and shading parameter regularization loss, simultaneously optimizing both the vertex positions of the triangular meshes and the pixel values ​​of the textures. In the first two stages, the semantic maps are derived from the rasterization results of manually labeled vertex semantic tags. The third stage sets the semantic maps to a learnable state and adds an additional semantic map loss to optimize the pixel values ​​of the semantic maps. The trained semantic maps are fixed and play a guiding role in the geometric optimization process during multi-frame training. Cluster analysis was performed based on the FLAME model parameters of each RGB sequence image, and several RGB sequence images with large differences in expression were selected as video frames to participate in the training of the reconstructed animated face model. Using the optimized static single-frame face model as the initial mesh, an expression triangle mesh is optimized for each video frame participating in the training of the reconstructed animated face model. Then, the shading map shared between different frames is trained, and the expression blending shape and shading map are output to complete the animated 3D face reconstruction of the face video to be reconstructed.

2. The method according to claim 1, characterized in that, The semantic tags of the facial region semantic tag sequence frame image include at least the background, skin, left and right eyebrows, left and right eyes, nose, upper and lower lips, inside the mouth, left and right ears, and hair.

3. The method according to claim 1, characterized in that, The process involves using the optimized static single-frame face model as the initial mesh, optimizing an expression triangle mesh for each video frame participating in the training of the reconstructed animated face model, then training a shared shading map between different frames, outputting the expression blending shape and shading map, and completing the animated 3D face reconstruction of the video to be reconstructed. Specifically: The initial triangular mesh is obtained by adding the FLAME pose and expression offset of each video frame that participated in the training of the reconstructed animated face model to the optimized static single-frame face model. A differentiable renderer is used to render triangular meshes, shading maps, and semantic maps separately through a rasterization process at a specified camera viewpoint, resulting in differentiable RGB images and differentiable semantic images. A loss function is calculated, and its gradient is backpropagated through an automatic differentiation framework to the vertex coordinates of the triangular meshes and the pixel values ​​of the maps, enabling iterative optimization of the triangular meshes and maps. The model optimization consists of three stages: the first stage adds geometric loss and smoothing loss, optimizing only the facial expression mesh for each frame; the second stage adds additional image loss and shading parameter regularization loss, simultaneously optimizing both the facial expression mesh and the shading map to obtain a blurred map; the third stage uses only image loss and shading parameter regularization loss, optimizing only the shading map.

4. The method according to claim 3, characterized in that, Multi-resolution hash encoding technology is used to improve texture clarity, resulting in clear textures.

5. The method according to claim 3, characterized in that, Different video frames independently optimize their respective facial expression meshes while sharing the same shading maps and semantic maps.

6. The method according to claim 1 or 3, characterized in that, The geometric loss includes: semantic loss, L1 loss of the difference between the semantic label sequence frame image of the face region and the differentiable rendered semantic image; and masking loss, L1 loss of the difference between the masking sequence frame image of the face region and the non-transparent region masking obtained from the differentiable rendered RGB image. The image loss includes the L1 loss between the RGB sequence frame image and the differentiable RGB image, and the SSIM loss between the RGB sequence frame image and the differentiable RGB image. The smoothing loss includes the Laplacian loss obtained by calculating the neighborhood Laplacian of each vertex and the normal regularization loss obtained by calculating the normal cosine of each adjacent triangle face; The shading parameter regularization loss includes: roughness regularization, which constrains roughness to approximate a normal distribution with a mean of 0.5 and a standard deviation of 0.1; specular reflection regularization, which constrains specular reflection in the skin region to approximate a normal distribution with a mean of 0.375 and a standard deviation of 0.166; and light source regularization, which constrains the light color of the lightmap to approximate white light. The semantic mapping loss is the L1 loss of differentiable renderable semantic images and semantic label sequence frames of face regions.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements an animated 3D face reconstruction method based on triangular mesh differentiable rendering as described in any one of claims 1-6.

8. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, implement the animable 3D face reconstruction method based on triangular mesh differentiable rendering as described in any one of claims 1-6.

9. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the animable 3D face reconstruction method based on triangular mesh differentiable rendering as described in any one of claims 1-6.

Images