Method and system for generating dynamic face texture based on expression semantic features

Abstract

This invention discloses a dynamic face texture generation method based on facial expression semantic features, belonging to the interdisciplinary field of computer vision and graphics. It includes: extracting geometric parameters from a parametric face model tracking video, driving model deformation, and anchoring 3D Gaussian points; constructing a dynamic texture generation network, using an identity latent code as a basis, and modulating the texture feature map layer by layer using facial expression parameters through a feature modulation module to output a dynamic UV texture map; calculating the gradient magnitude of the UV texture map, and adaptively encrypting the Gaussian points based on the texture gradient and geometric gradient; obtaining Gaussian point attributes based on UV coordinate sampling, and inputting them into a differentiable rasterizer renderer after viewpoint-related color compensation to generate an image. This achieves dynamic texture generation that is linked to facial expressions, significantly improving the dynamic realism and detail fidelity of digital human reconstruction, and solving the problem of high-frequency dynamic detail loss caused by static texture representation in existing monocular video 3D face reconstruction.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and computer graphics, and in particular to a method and system for generating dynamic facial textures based on facial expression semantic features. Background Technology

[0002] 3D digital human head reconstruction technology is one of the core pillars of the virtual reality, augmented reality, and digital entertainment industries. Traditional high-fidelity reconstruction methods typically rely on multi-view camera arrays or professional 3D scanning equipment, whose high cost and complex operation limit their widespread adoption in consumer-grade scenarios. In recent years, reconstruction methods based on monocular RGB video have attracted much attention due to their convenience and low cost. Among them, 3D Gaussian splashing, as an emerging explicit radiation field representation method, is gradually becoming a mainstream technology due to its faster training speed and real-time rendering capabilities compared to neural radiation fields.

[0003] In existing technologies, monocular head reconstruction methods based on parametric models using 3D Gaussian splashing generally employ a strategy of decoupling geometry and texture. First, a parametric face model (such as the FLAME model) is used as a geometric prior. Expression and pose parameters extracted from the video drive mesh deformation, thereby determining the basic position and rotation of 3D Gaussian points in space. Second, to overcome the difficulty of editing disordered point clouds from 3D Gaussian splashing, a UV spatial mapping mechanism is introduced: typically using a convolutional neural network, inputting a static latent code representing the target person's identity, to generate a static UV texture map (including color and opacity maps). Finally, based on the UV coordinates of the 3D Gaussian points on the FLAME model surface, the attribute values ​​of each Gaussian point are sampled on the generated texture map, and rendering is completed using a rasterizer.

[0004] Although this geometric decoupling plus UV texture mapping method achieves editable head reconstruction to some extent, it still has significant technical bottlenecks in practical applications, mainly manifested in the loss of high-frequency dynamic details.

[0005] Specifically, existing texture generation networks rely solely on static identity latent codes, resulting in unchanging UV texture maps regardless of the target person's facial expression (e.g., laughing, angry, or surprised). While the FLAME model's geometric mesh deforms with facial expression parameters, it is inherently a low-polygon mesh and cannot represent the subtle wrinkles not included in the UV texture map through geometric deformation. Real human faces possess complex physical characteristics; when facial muscles move, the skin is compressed and stretched, generating transient high-frequency texture details. For example, crow's feet appear at the corners of the eyes when laughing, nasolabial folds deepen on both sides of the nose, or the skin on the chin becomes uneven when closing the mouth forcefully.

[0006] Because existing technologies use fixed texture maps, systems can only simulate these changes through geometric stretching or learn an averaged texture. This results in reconstructed models having overly smooth skin when making large facial expressions, creating a stiff, mask-like appearance that severely impacts the realism and immersion of digital humans. Furthermore, facial expressions are often accompanied by changes in skin color due to blood flow, such as facial redness from anger or bulging neck veins from exertion. Static texture mapping mechanisms cannot capture and reproduce these physiological texture features that are linked to facial expressions. Summary of the Invention

[0007] To achieve the above objectives, this invention proposes a geometry-texture dual-driven method for reconstructing a 3D digital human head. This method overcomes the asymmetric limitations of dynamic geometry and static texture in traditional techniques. By constructing a dynamic texture generation network based on facial expression semantic feature modulation, a dynamic mapping channel is established between facial expression parameters and texture features, enabling texture details to move in real-time with changes in facial expression. Simultaneously, to effectively accommodate the generated dynamic high-frequency details, an adaptive densification strategy guided by texture gradients is introduced. By calculating the gradient magnitude of the dynamic UV texture map and mapping it back to 3D Gaussian points, combined with the view space geometric gradient, the splitting and cloning of Gaussian points are guided, ensuring that the point cloud has sufficient resolution in complex texture areas such as the corners of the eyes and forehead to resolve and render minute wrinkles and skin color variations.

[0008] Through the above-mentioned technical means, the present invention achieves high-fidelity, driveable 3D digital human head reconstruction, while maintaining the real-time rendering capability of 3D Gaussian splash, and significantly improves the realism of skin details under dynamic expressions.

[0009] On the one hand, a method for generating dynamic facial textures based on facial expression semantic features is provided, including: A video sequence containing the target object is acquired, and a parametric face model is used for tracking to extract geometric parameters; wherein, the geometric parameters include shape parameters, pose parameters, and expression parameters; The parametric face model is deformed according to the geometric parameters to obtain the deformed mesh vertex model. The initialized 3D Gaussian points are anchored on the surface of the deformed mesh vertex model, and fixed UV coordinates are assigned to the 3D Gaussian points. The facial expression parameters are input into a pre-constructed dynamic texture generation network. The dynamic texture generation network uses a static identity latent code as the content base and uses the facial expression semantic features obtained by mapping the facial expression parameters to modulate the texture feature map layer by layer through the feature modulation module, and outputs a dynamic UV texture map containing high-frequency details. Calculate the gradient magnitude of the dynamic UV texture map, and refine the Gaussian points in the complex texture region according to the preset densification scoring standard; Based on the UV coordinates of the 3D Gaussian points, the attribute values ​​of each Gaussian point are sampled on the dynamic UV texture map and rendered using a differentiable rasterizer to generate a predicted image.

[0010] On the other hand, a dynamic face texture generation system based on facial expression semantic feature modulation is provided, including: The data preprocessing module is configured to: acquire a video sequence containing the target object, track it using a parametric face model, and extract geometric parameters; wherein the geometric parameters include shape parameters, pose parameters, and expression parameters; The geometry driving module is configured to: drive the deformation of the parameterized face model according to the geometric parameters to obtain the deformed mesh vertex model, anchor the initialized 3D Gaussian points on the surface of the deformed mesh vertex model, and assign fixed UV coordinates to the 3D Gaussian points; The dynamic texture generation module is configured to: input the expression parameters into a pre-constructed dynamic texture generation network, wherein the dynamic texture generation network uses a static identity latent code as the content base, and modulates the texture feature map layer by layer using the expression semantic features obtained by mapping the expression parameters through a feature modulation module, and outputs a dynamic UV texture map containing high-frequency details. The adaptive densification module is configured to: calculate the gradient magnitude of the dynamic UV texture map, and densify the Gaussian points in the complex texture region according to a preset densification scoring standard; The rendering output module is configured to: sample the attribute values ​​of each Gaussian point on the dynamic UV texture map based on the UV coordinates of the 3D Gaussian point, and render the image through a differentiable rasterizer to generate a predicted image.

[0011] In another aspect, a computer device is also provided, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, when the processor executes the program, it performs the method described in the first aspect.

[0012] In another aspect, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the method described in the first aspect.

[0013] The above technical solution has the following advantages or beneficial effects: (1) This invention breaks through the limitations of static textures in traditional methods, significantly improving the dynamic realism of digital humans. Existing reconstruction methods based on monocular video typically employ a strategy of dynamic geometry plus static textures, resulting in an unnatural smoothness of the skin surface under exaggerated facial expressions. This invention establishes a nonlinear mapping relationship between expression parameters and texture features by constructing a dynamic texture generation network based on facial expression semantic feature modulation. This network utilizes an adaptive instance normalization module to inject high-frequency details into the texture feature map in real time according to the current expression. It can not only accurately reproduce the dynamic physical wrinkles generated by muscle movement, such as the deepening of crow's feet and nasolabial folds, but also simulate physiological skin color changes such as facial congestion and bulging veins. This makes the skin texture of the generated digital human vivid and delicate when making micro-expressions or exaggerated expressions, greatly improving visual realism and immersion.

[0014] (2) This invention solves the undersampling problem in areas with complex textures, ensuring the clear presentation of high-frequency details. Traditional 3D Gaussian splashing methods only rely on the position gradient in view space to densify point clouds, often ignoring the reconstruction needs of geometrically flat but texture-rich areas such as the forehead and corners of the eyes, resulting in blurry dynamic textures. To address this deficiency, this invention proposes an adaptive densification strategy based on texture gradient guidance, mapping the texture gradient magnitude in UV space back to 3D space, and combining it with geometric gradient as the criteria for Gaussian point splitting and cloning. This method can automatically detect high-frequency areas where wrinkles are generated and densify the Gaussian point cloud at these locations. This geometry-texture dual guidance mechanism ensures that the Gaussian point cloud has sufficient resolution to carry minute dynamic textures from the data structure level, effectively eliminating blur and jaggedness in the rendered image.

[0015] (3) This invention introduces a dynamic texture generation mechanism while maintaining low computational cost and real-time rendering capabilities. Compared to existing technologies that sacrifice rendering efficiency to improve image quality, this invention employs a lightweight feature mapping network and a U-Net decoder, without altering the underlying rasterization rendering pipeline of 3D Gaussian splashing. Feature modulation and texture sampling are both efficient matrix operations, adding only minimal inference overhead. While significantly improving rendering quality, it still maintains the core advantages of real-time rendering inherent in the 3D Gaussian splashing method, possessing excellent computational efficiency.

[0016] (4) This invention lowers the data acquisition threshold and achieves low-cost, high-fidelity reconstruction. Traditional high-fidelity facial reconstruction methods often rely on expensive light field camera arrays or professional 3D scanning equipment to obtain high-precision geometric and texture maps, making them difficult to popularize in consumer-grade scenarios. This invention, however, employs a self-supervised learning framework, requiring only a single ordinary monocular RGB video for training. Through the constraints of the joint loss function, the network can automatically decouple from and learn geometric and texture information from the video. This significantly reduces the hardware cost and data threshold for digital human creation, making high-fidelity 3D avatar modeling possible on consumer-grade devices and possessing broad commercial application prospects. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention.

[0019] Figure 2 This is a schematic diagram of the overall framework of the method in Embodiment 1 of the present invention.

[0020] Figure 3 This is a schematic diagram of the dynamic texture generation network structure according to Embodiment 1 of the present invention.

[0021] Figure 4 This is a schematic diagram of the texture gradient and densification process in Embodiment 1 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] In this embodiment of the invention, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of this invention, "multiple" refers to two or more.

[0025] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0028] Example 1 To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. Those skilled in the art should understand that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0029] This embodiment provides a method for generating dynamic facial textures based on facial expression semantic features. Figure 1 This is an overall flowchart of the dynamic face texture generation method based on facial expression semantic features according to an embodiment of the present invention, as follows: Figure 1 As shown, the method includes the following steps: S101: Obtain a video sequence containing the target object, track it using a parametric face model, and extract geometric parameters; S102: Drive the parametric face model to deform according to the geometric parameters to obtain the deformed mesh vertex model, anchor the initialized 3D Gaussian points on the surface of the deformed mesh vertex model, and assign fixed UV coordinates to the 3D Gaussian points. S103: Input the facial expression parameters into the pre-built dynamic texture generation network. The dynamic texture generation network uses the static identity latent code as the content base and modulates the texture feature map layer by layer through the feature modulation module using the facial expression semantic features obtained by mapping the facial expression parameters, and outputs a dynamic UV texture map containing high-frequency details. S104: Calculate the gradient magnitude of the dynamic UV texture map and refine the Gaussian points in the complex texture area according to the preset densification scoring standard. S105: Based on the UV coordinates of the 3D Gaussian points, sample the attribute values ​​of each Gaussian point on the dynamic UV texture map, and render it through a differentiable rasterizer to generate a predicted image.

[0030] The overall framework of the method in Embodiment 1 of the present invention is as follows: Figure 2 As shown, in step S101: a video sequence containing the target object is acquired, and tracking is performed using a parametric face model to extract geometric parameters, specifically including: First, obtain a monocular RGB video sequence containing the target object. To drive the 3D face model, the video needs to be decoupled and parameterized frame by frame. This embodiment preferably uses the FLAME model as the geometric prior for the parameterized face model. A gradient-based optimization algorithm or encoder network is used to decouple each frame of the video. Tracking and fitting are performed to extract the geometric parameters for each frame.

[0031] Furthermore, the extracted geometric parameters include shape parameter β, pose parameter θ, and expression parameter. Among them, the shape parameter β is used to describe identity geometric features such as weight and facial shape, and this parameter is usually kept fixed in the same video sequence; the pose parameter θ includes neck joint rotation and chin joint rotation, used to describe the spatial orientation of the head and the degree of chin opening and closing; expression parameters... This is used to numerically describe the facial muscle movement state of the current frame, such as the movement of the corners of the mouth and eyebrows, which will serve as a key driving signal for subsequent dynamic texture generation. In addition to geometric parameters, it is also necessary to obtain the face region mask extracted by the semantic segmentation network to eliminate the interference of background information, and at the same time obtain the camera intrinsic parameter K and extrinsic parameter matrix [R|T], which are used to project the 3D model onto the two-dimensional image space.

[0032] Optionally, this embodiment uses the FLAME model as a parametric face model. In other implementations, those skilled in the art can also choose other parametric face models, such as 3DMM, FaceWarehouse, or a driving system based on BlendShapes (such as ARKit with 52 coefficients).

[0033] Further, in step S102, the parametric face model is deformed according to the geometric parameters to obtain the deformed mesh vertex model, and the initialized 3D Gaussian points are anchored on the surface of the deformed mesh vertex model, specifically including: After obtaining the driving parameters, the standard topology mesh of the FLAME model needs to be transformed to the pose of the current frame. The FLAME model contains... Each vertex is transformed using a Linear Blend Skinning (LBS) function. First, the shape parameter β and expression parameter are used... Linear offset of vertices at zero orientation: , in, As an average face template, and These are the mixed shape bases for shape and expression, respectively, and S represents the identity shape orthogonal basis matrix extracted by principal component analysis, used to describe the inherent differences in facial bone and muscle contours between different individuals. The representation is the orthogonal basis matrix of facial expressions extracted by principal component analysis, used to describe the transient geometric deformation caused by facial muscle movements.

[0034] After linear offset, skin transformation is required using the attitude parameter θ, along with predefined skin weights. This transforms the vertices from standard space to world space. For the i-th vertex... Its transformed coordinates The calculation formula is: , Where k represents the joint index. It is the weight of the i-th vertex affected by the k-th joint. It is a rigid body transformation matrix calculated from the posture parameter θ and the joint position J. Joints is a predefined set of motion joints in the FLAME model, mainly including key joints such as the global head, neck, jaw / chin and eyeballs.

[0035] By following the steps above, we obtain the 3D mesh vertex model after deformation in the current frame. .

[0036] Once the transformation process is determined, Gaussian points can be bound to mesh vertices, allowing the Gaussian points to move along with the mesh vertices. First, the Gaussian points are initialized: N points are uniformly sampled on the UV unfolded map of the FLAME mesh or randomly sampled on the mesh surface as the initial set of 3D Gaussian primitives. ,in, Let N represent the i-th 3D Gaussian element in the set, and let N represent the total number of Gaussian elements generated by sampling.

[0037] After obtaining the initial Gaussian points, assign UV coordinates to each Gaussian point. This coordinate corresponds to the texture space of the FLAME model and remains unchanged throughout training and inference, providing an indexing basis for subsequent UV-based dynamic texture sampling and texture gradient compaction.

[0038] After initializing the Gaussian points, their geometric positions need to be determined. Record the index of the mesh triangle to which each Gaussian point belongs. and the coordinates of the barycenter within the triangle The Gaussian point moves along with the underlying mesh triangles. Before each frame is rendered, the 3D center position of the Gaussian point is determined. Instead of being used directly as optimization parameters, it is based on the deformed mesh vertices of the current frame. Real-time calculation yields: , in , , The coordinates of the three vertices of the triangle to which the Gaussian point belongs in the world coordinates of the current frame.

[0039] In this way, 3D Gaussian points are attached to the surface of the FLAME mesh and move synchronously with the character's nodding, opening mouth, and other actions, achieving physical alignment between geometry and texture, thereby obtaining a set of 3D Gaussian points carrying dynamic geometric positions.

[0040] Further, in step S103: the expression parameters are input into a pre-constructed dynamic texture generation network. The dynamic texture generation network uses a static identity latent code as its content base. Through a feature modulation module, it modulates the texture feature map layer by layer using the expression semantic features obtained by mapping the expression parameters, and outputs a dynamic UV texture map containing high-frequency details. Specifically: To handle detailed dynamic facial textures, this embodiment improves upon traditional texture generation networks by introducing a feature modulation module, constructing a dynamic texture generation network capable of adjusting its output in real time based on facial expression changes. The dynamic texture generation network aims to establish a non-linear mapping relationship between facial expression parameters and texture features, and mainly consists of two parts: an expression mapping module and a texture generation network. Figure 3 As shown.

[0041] (1) Expression Mapping Module: To decouple low-dimensional expression parameters and map them to a latent style space, this embodiment uses a multilayer perceptron (MLP) to construct the mapping network. This MLP contains 3 to 5 fully connected layers, with LeakyReLU activation functions used between layers. The network input is the expression parameters. The expression semantic feature vector is output by propagating forward layer by layer through the mapping network M. The calculation process can be expressed as follows: , in , Let the weights and biases of the i-th layer be denoted as . f The activation function is Leaky ReLU. n This represents the number of connection layers. Through the above mapping process, the original facial expression parameters are mapped into potential facial expression semantic feature vectors. .

[0042] To accommodate the differences in the number of channels at different levels in the texture generation backbone network, it is also necessary to incorporate the facial expression semantic feature vectors. Perform an affine transformation to generate a scaling factor. and offset factor , specifically: , in, A It is an affine transformation matrix. It is a facial expression semantic feature vector. b affine This is the bias vector of the affine transformation layer. The output is the facial expression semantic feature vector. Divided into scaling factors and offset coefficient Its dimension matches the number of channels in the target convolutional layer of the texture generation network. It is applied to the normalized feature map to control the intensity of the feature channels. It is applied to the scaled feature map to control the brightness of the feature channels.

[0043] (2) UV texture generation network: The U-Net architecture is used as the basic generation network. The network input is a learnable static identity latent code. Static identity latent code It is a learnable parameter tensor of a predefined dimension, used to uniquely encode the inherent appearance attributes of a target object independent of transient facial expression changes. The generation of the static identity latent code is independent of the specific image input for each frame. During the initialization phase of model training, this latent code is initialized by random sampling from a standard normal distribution. During end-to-end training of the model, it is used as an optimizable parameter in the network graph structure, iteratively updated synchronously with the weights of the texture generation network using the backpropagation algorithm based on the joint loss function between the final rendered image and the real video frames. When the model training converges, the value of this latent code is fixed, becoming a static content basis uniquely representing the identity of the target object. This identity latent code... After entering the texture generation backbone, a basic feature map is generated. This is used to characterize the basic skin tone and fixed features of a person. Feature modulation is then performed, with an adaptive instance normalization AdaIN module inserted between the convolutional and activation layers during the U-Net upsampling stage.

[0044] Specifically, to generate detailed textures, textures are generated by altering the statistical distribution of the feature maps. (The backbone network...) l The input to the layer is a feature map. After each convolutional operation, the facial expression semantic feature vector is used. Adaptive instance-normalized AdaIN modulation is applied to the feature map to inject high-frequency details such as wrinkles. The input feature map is then spatially modified. Standardize the surface to remove static, smooth texture features. Calculate the standard deviation. and mean : , , in, H Representative feature map Spatial height, W Representative feature map Space width, c For the number of channels, n The batch size is the number of samples processed. In the feature map, the first n The sample, the first c One channel, located at spatial coordinates The specific pixel value at that location.

[0045] Then use scaling factor and offset factor The normalized feature map is rescaled and translated to inject new dynamic texture features:

[0046] in, and Calculate the mean and standard deviation of the feature map in the spatial dimension, and the scaling factor. and offset factor It is composed of facial expression semantic feature vectors The generated affine transformation coefficients, Control the contrast of the texture. Controls the base tone of the texture. This changes when the expression parameters change. , This leads to changes, resulting in The numerical distribution changes in a specific area, thereby generating detailed textures such as wrinkles in subsequent rendering.

[0047] Modulated feature map After passing through the output layer, it is decoded into a diffuse color map. and opacity map To preserve high-frequency details of the texture, the network progressively increases the feature map resolution through multiple levels of upsampling. Upgraded to : , ModBlock is a modulation module containing convolutional layers and AdaIN operations, injecting facial expression information at each resolution level to ensure no loss of texture detail. (Decoded diffuse color map) Includes skin details that change with facial expressions, such as forehead wrinkles, nasolabial folds, and the color and texture of facial flushing; opacity map. It is used to handle the subtle occlusion relationships at the closed area of ​​the lips.

[0048] Optionally, the feature modulation module in this embodiment adopts an adaptive instance normalization AdaIN structure. In other implementations, those skilled in the art can also use a concatenation method (without normalization modulation, but expanding the mapped facial expression semantic feature vector to the same spatial size as the texture feature map, directly concatenating it with the texture feature map in the channel dimension, and then fusing the information through a convolutional layer), a spatial feature transformation (SFT) module (using facial expression parameters to generate spatially adaptive affine transformation parameters, modulating the feature map pixel by pixel to handle texture variations with uneven spatial distribution), or a cross-attention mechanism (using the texture feature map as a query, and the facial expression semantic feature vector as a key and value, dynamically aggregating facial expression information through an attention mechanism). The texture generation network can also be based on the VisionTransformer architecture (using Self-Attention to handle long-distance texture dependencies) or adopt a pure decoder structure based on ResNet. The network output is not limited to diffuse reflection and opacity; it can also output normal maps or displacement maps simultaneously to further enhance the three-dimensionality of fine wrinkles at the geometric level.

[0049] Through the above modulation process, the dynamic texture generation network can dynamically enhance or suppress the feature response of specific areas (such as the corners of the eyes, forehead, etc.) according to the character's facial expression in the current frame, and output the dynamic UV texture map corresponding to the current frame, providing detailed textures such as wrinkles for subsequent rendering.

[0050] Further, in step S104: the gradient magnitude of the dynamic UV texture map is calculated, and the Gaussian points in the complex texture region are densified according to a preset densification scoring standard, specifically: In traditional 3D Gaussian splashing methods, densification strategies are typically based on positional gradients in the 3D view. That is, Gaussian points are split and cloned according to changes in the positional gradient, focusing only on changes in geometric contour edges, such as the outline of a face or the tip of the nose. In geometrically flat areas like the cheeks and corners of the eyes, the gradient is small, and the model considers that few Gaussian points are needed there, thus neglecting densification. However, dynamic textures need to display wrinkles and other textural phenomena caused by facial expressions in these areas with smaller gradients.

[0051] like Figure 4As shown, to achieve better dynamic texture effects, this embodiment proposes an improved densification strategy based on texture gradients. This method allows the model to not only focus on changes in 3D geometric gradients but also consider gradient changes in the UV texture map. If a certain region of the texture map frequently exhibits high-frequency details, the model is guided to perform Gaussian point splitting and cloning in that region, increasing the number of Gaussian points in that region, thereby improving the detail texture. This is specifically achieved through the following method: (1) Texture gradient calculation: In each fixed densification cycle, obtain the dynamic UV texture map generated by the current network. The horizontal gradient is calculated using the Sobel edge detection operator. and vertical gradient : , And calculate the gradient magnitude map of the texture map. : , In the gradient magnitude map, the brighter the area, the deeper the wrinkles or the more details there are, which is where the wrinkles and other detailed textures are located, providing a location basis for subsequent Gaussian point encryption.

[0052] (2) Obtain the texture gradient of the Gaussian point: For each Gaussian point in the scene Using its fixed UV coordinates In the gradient magnitude map Bilinear sampling is performed on the top. , Where N represents the four neighboring pixels surrounding the UV coordinates. The bilinear interpolation weights corresponding to the k-th neighborhood pixel are given by... and The distance between them is calculated inversely. For the k-th neighboring pixel in the gradient magnitude map Discrete grid coordinates on the grid This represents the actual gradient magnitude at the neighboring pixel. The texture gradient intensity corresponding to this Gaussian point is obtained by sampling. In this way, each Gaussian point obtains its own corresponding texture gradient value. Then, based on the magnitude of the texture gradient value and combined with the geometric gradient, a densification strategy of splitting and cloning is applied to the corresponding Gaussian point.

[0053] (3) Densification Strategy: A new densification scoring criterion is constructed, where whether a Gaussian point needs to be split is no longer determined solely by its spatial location, but also by its texture complexity. This embodiment employs a max-pooling strategy, meaning that Gaussian points in areas with large geometric errors or numerous texture details should be split and densified. For Gaussian points... The combined scoring formula is as follows: , in, λ is the texture gradient index; λ is the balancing coefficient used to align the magnitude of the texture gradient to the geometric gradient. This is the view-space position gradient of standard 3DGS, i.e., the geometric index in the model. It is calculated by taking the total loss function L as a function of the view-space center coordinates of the i-th Gaussian point. The partial derivative of the point is used, and its L2 norm is taken as the geometric index of that point: , Then, a preset density threshold is applied. Use a splitting strategy: , Based on the scoring criteria of the splitting strategy, Gaussian points that meet the splitting requirements are cloned and split, thereby encrypting Gaussian points in complex texture areas and making detailed textures clear.

[0054] Optionally, the Sobel operator is used when calculating the texture gradient in this embodiment. In other implementations, those skilled in the art may also use the Laplacian operator, the Canny edge detection operator, or a deep learning-based edge detection network. The densification control can be adjusted more finely by combining the screen space projection area or the viewpoint distance.

[0055] Further, in step S105: based on the UV coordinates of the 3D Gaussian points, the attribute values ​​of each Gaussian point are sampled on the dynamic UV texture map and obtained, and then rendered using a differentiable rasterizer to generate a predicted image, specifically: After generating a 2D UV texture from the network, the 2D UV texture needs to be remapped back into the 3D model to complete the rendering of the entire model.

[0056] Specifically, first, determine the geometric positions of the vertices of the FLAME model mesh, based on the FLAME parameters (β, θ, ... Using the linear blend skin function (LBS), the positions of the mesh vertices in the current pose are calculated: , in The 3D coordinates of the i-th mesh vertex in the current pose. These are the vertex coordinates in the standard pose. The skin weight represents the influence of the k-th bone on the i-th vertex. It is the skeleton transformation matrix, the global rigid body transformation matrix of the k-th bone, which is a... The matrix contains rotation and translation, where θ is the attitude parameter and J is the joint position.

[0057] Based on the vertex positions of the FLAME mesh in the current frame The Gaussian points can be used to calculate their position coordinates. The bound Gaussian points are then used for texture sampling based on the FLAME mesh vertices. Each Gaussian point is bound to a fixed UV coordinate on the FLAME mesh surface. Above, for each Gaussian point Using its UV coordinates and the generated dynamic texture map and The color value of a Gaussian point at the current time is obtained by sampling through bilinear interpolation on the dynamic texture map. and opacity : , .

[0058] After obtaining the color value and opacity of the Gaussian point at the current moment, to simulate the non-Lambertian reflection characteristics of realistic human facial skin under different lighting angles, a viewpoint-dependent specular reflection component is further introduced on top of the basic diffuse color. This is achieved by obtaining the center coordinates of the current rendering camera. Calculate its direction pointing to the center of the i-th Gaussian point. Normalized line-of-sight vector : , A lightweight multilayer perceptron is constructed as the color compensation network, using the gaze direction vector. eigenvectors of Gaussian points As input, predict the color residual. ,in This represents the intensity of specular reflection caused by different viewing angles. The sampled dynamic diffuse color... With specular residual Add them together to get the final rendered color of the i-th Gaussian point from the current viewpoint. : , color It will replace the base color in subsequent α-blending rasterization calculations.

[0059] After obtaining the color, opacity, and other attributes of each Gaussian point, a 3D Gaussian splash renderer is used to project the dynamically attributed 3D Gaussian points onto a 2D screen space to generate a rendered image. The rendering process follows the α-blending formula: , Where N is an ordered set of all 3D Gaussian points covering the pixel, arranged in order of depth from near to far. The color of the i-th Gaussian point after sorting. Let i be the contribution value of the i-th Gaussian point to the opacity of the current pixel. This is a way of representing transmittance, indicating how much light intensity remains before the light reaches the i-th point.

[0060] To ensure the realism of the generated dynamic textures when generating rendered images, a joint loss function is employed. The weights of the dynamic texture generation network (including the facial expression mapping module), the parameters of the color compensation network, the attribute parameters of the 3D Gaussian tuple P, and the static identity latent code are considered. Perform end-to-end joint optimization, where the loss function includes pixel reconstruction loss. L rec Perceived loss L percep and combat losses L adv : , in, For pixel reconstruction loss, a weighted combination of L1 loss and D-SSIM loss is used. This loss is used to ensure color consistency between the rendered image and the real image at the pixel level, taking into account both overall color reproduction and structural similarity. , in, The generated predicted image is rendered for the model. This represents the actual image of the corresponding video frame. The L1 norm is used to constrain the accuracy of pixel colors. It is a structural similarity index used to constrain the local structural consistency of an image. The adjustment coefficient for L1 and SSIM is 0.2 in this embodiment.

[0061] To address the issue of L1 loss leading to image blurring, a perceptual loss based on LPIPS is introduced. This loss extracts features through a pre-trained deep neural network and calculates the distance of the image in the feature space. This loss ensures that the generated image maintains consistency with the real image at the semantic feature level, thereby improving the clarity of facial features. , in, The pre-trained feature extraction network is in the first... l The feature map output of the layer uses the L2 norm.

[0062] To accurately reproduce the micro-texture changes that occur with facial expressions, an adversarial learning mechanism is introduced. A PatchGAN discriminator D, targeting local facial regions, is employed. Discriminator D aims to distinguish whether the input image patch comes from a real video frame. Or the image generated by rendering The goal of the dynamic texture generation network is to "deceive" the discriminator, causing it to misclassify the generated texture as a real texture. The adversarial loss of the generator is defined as: , The discriminator network D outputs a probability value between [0,1], representing the probability that the input image patch is a real image. It is a local facial area randomly cropped from the rendered image. If the generated dynamic wrinkles are blurry, the discriminator will give a low score, resulting in... The increased size forces the generative network to synthesize sharper, more realistic high-frequency details.

[0063] Through the joint constraint of the multi-task loss mentioned above, the model can automatically decouple static identity information and dynamic facial expression features from monocular videos, and learn the nonlinear mapping relationship from facial expression semantics to dynamic textures, effectively recovering high-frequency details and physiological skin color changes driven by facial expressions, and finally generating high-fidelity, drivable 3D digital human head portraits.

[0064] Example 2 This embodiment provides a dynamic face texture generation system based on facial expression semantic feature modulation, including: The data preprocessing module is configured to: acquire video sequences containing target objects, track them using a parametric face model, and extract geometric parameters; wherein, the geometric parameters include shape parameters, pose parameters, and expression parameters; The geometry driving module is configured to: drive the deformation of the parameterized face model according to the geometric parameters to obtain the deformed mesh vertex model, anchor the initialized 3D Gaussian points on the surface of the deformed mesh vertex model, and assign fixed UV coordinates to the 3D Gaussian points. The dynamic texture generation module is configured to: input facial expression parameters into a pre-built dynamic texture generation network, which uses a static identity latent code as the content base, and modulates the texture feature map layer by layer using the facial expression semantic features obtained by mapping the facial expression parameters through the feature modulation module, and outputs a dynamic UV texture map containing high-frequency details. The adaptive densification module is configured to: calculate the gradient magnitude of the dynamic UV texture map and densify the Gaussian points in the complex texture region according to the preset densification scoring criteria; The rendering output module is configured to: sample the attribute values ​​of each Gaussian point on a dynamic UV texture map based on the UV coordinates of the 3D Gaussian point, and render the predicted image through a differentiable rasterizer.

[0065] It should be noted that the above modules correspond to steps S101 to S105 in Embodiment 1, and the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0066] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0067] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0068] Example 3 This embodiment also provides a computer device, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, when the processor executes the program, it completes the method described in Embodiment 1.

[0069] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0070] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0071] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0072] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0073] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0074] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating dynamic facial textures based on facial expression semantic features, characterized in that, include: A video sequence containing the target object is acquired, and a parametric face model is used for tracking to extract geometric parameters; wherein, the geometric parameters include shape parameters, pose parameters, and expression parameters; The parametric face model is deformed according to the geometric parameters to obtain the deformed mesh vertex model. The initialized 3D Gaussian points are anchored on the surface of the deformed mesh vertex model, and fixed UV coordinates are assigned to the 3D Gaussian points. The facial expression parameters are input into a pre-constructed dynamic texture generation network. The dynamic texture generation network uses a static identity latent code as the content base and uses the facial expression semantic features obtained by mapping the facial expression parameters to modulate the texture feature map layer by layer through the feature modulation module, and outputs a dynamic UV texture map containing high-frequency details. Calculate the gradient magnitude of the dynamic UV texture map, and refine the Gaussian points in the complex texture region according to the preset densification scoring standard; Based on the UV coordinates of the 3D Gaussian points, the attribute values ​​of each Gaussian point are sampled on the dynamic UV texture map and rendered using a differentiable rasterizer to generate a predicted image.

2. The dynamic face texture generation method based on facial expression semantic features according to claim 1, characterized in that, The shape parameter is used to describe the geometric features of the target object, the pose parameter is used to describe the spatial orientation of the head and the degree of chin opening and closing, and the expression parameter is used to numerically describe the facial muscle movement state of the current frame.

3. The dynamic face texture generation method based on facial expression semantic features according to claim 1, characterized in that, The parametric face model is deformed based on geometric parameters to obtain a deformed mesh vertex model. Initialized 3D Gaussian points are then anchored onto the surface of the deformed mesh vertex model, specifically as follows: Based on shape and expression parameters, the vertices are linearly offset at zero pose. After the linear offset, the pose parameters are used for skinning transformation. Using the pose parameters and predefined skinning weights, the vertices are transformed from standard space to world space to obtain the deformed mesh vertex model. Sampling is performed on the surface of the mesh vertex model to obtain initial 3D Gaussian points. Based on the sampling position of the 3D Gaussian points on the mesh surface, UV coordinates are assigned to each Gaussian point, and the geometric position of the Gaussian point is determined. Each Gaussian point records the index of the mesh triangle to which it belongs and the centroid coordinates within that triangle. The Gaussian points move with the underlying mesh triangles. Before each frame is rendered, the 3D center position of the Gaussian point is calculated in real time by interpolation based on the deformed mesh vertex coordinates of the current frame and the recorded centroid coordinates.

4. The dynamic face texture generation method based on facial expression semantic features according to claim 1, characterized in that, The facial expression parameters are input into a pre-constructed dynamic texture generation network. This network uses a static identity latent code as its content base and a feature modulation module to modulate the texture feature map layer by layer using the facial expression semantic features mapped from the facial expression parameters. The output is a dynamic UV texture map containing high-frequency details. Specifically: An expression mapping module is constructed, which uses a multilayer perceptron to map the expression parameters into expression semantic feature vectors, and then generates scaling factors and offset factors through affine transformation; A texture generation network is constructed using the U-Net architecture, which generates basic feature maps by taking static identity latent codes as input. In the decoding stage of the texture generation backbone, the normalized feature map is rescaled and translated using the scaling factor and offset factor through the adaptive instance normalization module, so as to realize the layer-by-layer modulation of the texture features by the expression features. After multiple levels of upsampling, a dynamic UV texture map is output, which includes a diffuse color map and an opacity map.

5. The dynamic face texture generation method based on facial expression semantic features according to claim 1, characterized in that, The calculation of the gradient magnitude of the dynamic UV texture map, and the densification of Gaussian points in complex texture regions according to a preset densification scoring standard, specifically involves: The horizontal and vertical gradients of the dynamic UV texture map are calculated using an edge detection operator to obtain a gradient magnitude map. Based on the gradient magnitude map, bilinear sampling is performed to obtain the texture gradient magnitude of the Gaussian point. Based on the texture gradient, a densification score is calculated. Gaussian points with scores exceeding a preset threshold are split or cloned to increase the density of Gaussian points in complex texture regions.

6. The dynamic face texture generation method based on facial expression semantic features according to claim 1, characterized in that, The step involves sampling the attribute values ​​of each Gaussian point on the dynamic UV texture map based on the UV coordinates of the 3D Gaussian points, and then rendering the image using a differentiable rasterizer to generate a predicted image. Specifically: Calculate the 3D center position of each Gaussian point based on the deformed mesh vertex model and centroid coordinates of the current frame; Using the UV coordinates of each Gaussian point, bilinear sampling is performed on the diffuse color map and opacity map of the dynamic UV texture map to obtain the base color value and opacity. Obtain the center coordinates of the current rendering camera, calculate the normalized viewing direction vector pointing to the center of the Gaussian point, input the viewing direction vector and the feature vector of the Gaussian point into the color compensation network, and predict the color residual. The color compensation network is a multilayer perceptron. Add the base color value to the color residual to obtain the final color of the Gaussian point under the current viewpoint; Input a set of 3D Gaussian points carrying dynamic geometric positions and dynamic appearance attributes into a differentiable rasterizable renderer to render a predicted image.

7. The method for generating dynamic facial textures based on facial expression semantic features according to claim 1, characterized in that, The method further includes optimizing the parameters of the dynamic texture generation network and color compensation network, the attribute parameters of the 3D Gaussian points, and the static identity latent code using a joint loss function, specifically: The parameters of the dynamic texture generation network and the color compensation network are optimized based on the self-supervised reconstruction loss between the predicted image and the real image. The self-supervised reconstruction loss includes pixel reconstruction loss, perceptual loss, and adversarial loss; The pixel reconstruction loss is a weighted combination of L1 loss and structural similarity loss; The perceptual loss is calculated based on a pre-trained deep neural network to determine the distance between the predicted image and the real image in the feature space. The adversarial loss employs a PatchGAN discriminator targeting local facial regions, making the generated image more closely resemble the real image in terms of local texture.

8. A dynamic facial texture generation system based on facial expression semantic feature modulation, characterized in that, include: The data preprocessing module is configured to: acquire a video sequence containing the target object, track it using a parametric face model, and extract geometric parameters; wherein the geometric parameters include shape parameters, pose parameters, and expression parameters; The geometry driving module is configured to: drive the deformation of the parameterized face model according to the geometric parameters to obtain the deformed mesh vertex model, anchor the initialized 3D Gaussian points on the surface of the deformed mesh vertex model, and assign fixed UV coordinates to the 3D Gaussian points; The dynamic texture generation module is configured to: input the expression parameters into a pre-constructed dynamic texture generation network, wherein the dynamic texture generation network uses a static identity latent code as the content base, and modulates the texture feature map layer by layer using the expression semantic features obtained by mapping the expression parameters through a feature modulation module, and outputs a dynamic UV texture map containing high-frequency details. The adaptive densification module is configured to: calculate the gradient magnitude of the dynamic UV texture map, and densify the Gaussian points in the complex texture region according to a preset densification scoring standard; The rendering output module is configured to: sample the attribute values ​​of each Gaussian point on the dynamic UV texture map based on the UV coordinates of the 3D Gaussian point, and render the image through a differentiable rasterizer to generate a predicted image.

9. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the dynamic face texture generation method based on facial expression semantic features as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the dynamic face texture generation method based on facial expression semantic features as described in any one of claims 1-7.

Images