A Method and System for Generating 3D Digital Human Heads Based on Temporal Feature Enhancement Audio

Abstract

This invention discloses a method and system for generating 3D digital human avatars based on audio-driven temporal feature enhancement. The method includes: initializing Gaussian units and extracting features based on target video and audio; constructing a dual-branch personalized temporal network, where the second branch uses a temporal feature enhancement module to output temporally smooth deformations; pre-training a general motion field to learn common movements and combining it with identity discrimination loss to promote feature decoupling; freezing the general field during personalized training and combining its basic deformations with the temporally smooth deformations of the personalized temporal network through an adaptive fusion mechanism to generate a dynamic Gaussian representation; and finally, rendering a 3D digital human video with synchronized shape and stable temporal sequence based on the driving audio. This invention achieves high-quality, highly consistent digital human generation with few samples.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method and system for generating 3D digital human heads based on audio driven by temporal feature enhancement. Background Technology

[0002] With the rapid development of virtual reality, metaverse, and human-computer interaction technologies, audio-driven 3D digital human head synthesis technology has become a hot research topic in the industry. This technology aims to generate virtual avatars with high-fidelity visual appearance, accurate lip-sync, and natural facial dynamics based solely on input speech signals.

[0003] Early synthesis methods primarily relied on 3D deformation models. While these maintained basic motion plausibility, their expressive capabilities limited the resulting facial geometry, often resulting in overly smooth surfaces that failed to reproduce details like skin texture. With the advent of neural radiation fields, methods like AD-NeRF and GeneFace incorporated audio features and performed volume rendering within implicit fields, significantly improving visual quality. However, these methods typically suffer from high computational costs, long training cycles, and slow inference speeds, making them unsuitable for real-time interactive applications. Recently, 3D Gaussian sputtering technology, with its explicit discrete primitive representation and efficient rasterization pipeline, has achieved real-time rendering speeds while maintaining visual quality comparable to NeRF, and is beginning to be applied in digital human synthesis. However, existing 3D Gaussian sputtering-based digital human generation methods still have shortcomings in practical applications. For example, mainstream methods like TalkingGaussian and GaussianTalker often require long video training for specific identities to learn robust motion patterns, which severely limits scalability in real-world applications. While some subsequent technologies attempted to incorporate rapid adaptation using a small number of samples, these often came with limitations or costs. For example, InsTag, in an effort to reduce learning difficulty, separated the face and mouth regions, disrupting the integrity of the facial topology and leading to artifacts or structural misalignments at the junctions. Finally, existing technologies lack effective temporal consistency constraints. Because they rely heavily on instantaneous states, synthesized videos are prone to high-frequency flickering between consecutive frames, resulting in blurred renderings and unstable dynamic performance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for generating 3D digital human heads based on audio-driven temporal feature enhancement, comprising the following steps:

[0005] Obtain the video sequence and corresponding audio of the target object, initialize standard three-dimensional Gaussian primitives based on the video sequence, extract expression coefficients, and extract audio feature vectors based on the audio.

[0006] A dual-branch personalized temporal network is constructed, wherein the first branch outputs a first deformation based on the standard three-dimensional Gaussian unit, the audio feature vector, and the expression coefficient; the second branch inputs the standard three-dimensional Gaussian unit and time information into the temporal feature enhancement module, and then outputs a second deformation by combining the audio feature vector and the expression coefficient.

[0007] A general motion field is pre-trained based on a multi-identity dataset to learn common facial motion patterns across identities. During the pre-training process, the dual-branch personalized temporal network is combined with the first deformation to calculate the identity discrimination loss, so that the general motion field focuses on the common facial motion patterns.

[0008] The personalized temporal network is trained based on the video sequence of the target object; in the personalized training, the pre-trained and frozen general motion field is called to generate basic deformation, and the basic deformation is fused with the second deformation through an adaptive fusion mechanism to generate a dynamic three-dimensional Gaussian representation of the target object.

[0009] Based on the input driving audio, a 3D digital human video of the target object is generated and rendered through the frozen general motion field, the trained personalized temporal network, and the adaptive fusion mechanism.

[0010] Optionally, constructing a dual-branch personalized temporal network includes:

[0011] The first branch is constructed by encoding the position of the standard three-dimensional Gaussian unit and fusing it with the audio feature vector and the expression coefficient, and then predicting the first deformation through a multilayer perceptron network.

[0012] The second branch is constructed by inputting the position and time information of the standard three-dimensional Gaussian units into the temporal feature enhancement module. The temporal feature enhancement module extracts temporal smoothing features through a self-attention mechanism and fuses the temporal smoothing features with the audio feature vector and the expression coefficient. The second deformation is predicted through a multilayer perceptron network that shares weights with the first branch.

[0013] Optionally, calculating the identity differentiation loss using the first deformation includes:

[0014] Calculate the similarity between the first deformations predicted by a dual-branch personalized temporal network for different identity data;

[0015] Similarity is penalized to encourage the general motion field to learn common facial motion patterns that are independent of identity, while encouraging the first branch to learn identity-specific features.

[0016] Optionally, fusing the basic deformation with the second deformation through an adaptive fusion mechanism includes:

[0017] Construct an adjustment network to predict the spatial scaling factor based on the position of the standard 3D Gaussian elements;

[0018] The basic deformation is weighted and adjusted using the spatial scaling factor.

[0019] The weighted base deformation, the second deformation, and the position of the standard 3D Gaussian primitive are added together to obtain the dynamic Gaussian primitive position used for rendering.

[0020] Optionally, the personalized training also includes a step of introducing multi-scale geometric consistency constraints, specifically including:

[0021] Based on the video sequence of the target object, obtain the pseudo-true values ​​of the depth map and normal map;

[0022] Based on the dynamic 3D Gaussian representation rendering, a predicted depth map and a predicted normal map are obtained;

[0023] Calculate the normalized consistency loss between the predicted depth map and the pseudo-true value of the depth map at the local block level and the global image level, respectively.

[0024] Calculate the orientation similarity loss between the predicted normal map and the pseudo-true value of the normal map.

[0025] Optionally, personalized training may also include a step of introducing temporal consistency constraints on the second branch, specifically including:

[0026] Based on the second deformation, a series of consecutive frames of images are rendered;

[0027] The difference between consecutive rendered images and the corresponding real video frame sequences is calculated and used as a loss function to optimize the temporal feature enhancement module.

[0028] Optionally, personalized training may also include a step of introducing color consistency constraints, specifically including:

[0029] The predicted image is obtained based on dynamic 3D Gaussian representation rendering;

[0030] Calculate the pixel-level loss between the predicted image and the real video frame;

[0031] Calculate the structural similarity loss between the predicted image and the real video frame.

[0032] Optionally, generating and rendering the 3D digital human video of the target object includes:

[0033] If the input is text, the text is first synthesized into driving audio;

[0034] The driving audio is subjected to feature extraction to obtain a driving audio feature sequence;

[0035] The driving audio feature sequence and corresponding facial expression coefficients are input into the frozen general motion field and the trained personalized temporal network to obtain the basic deformation and the second deformation, respectively.

[0036] The basic deformation is fused with the second deformation through the adaptive fusion mechanism to obtain the dynamic Gaussian elements of each frame;

[0037] Using 3D Gaussian sputtering rendering technology, the dynamic Gaussian primitives are rendered into continuous image frames, which are then synchronized with audio to form a video.

[0038] This invention also provides an audio-driven 3D digital human head generation system based on temporal feature enhancement, used to implement the method described above. The system includes:

[0039] The data preprocessing module is used to acquire and process video, audio and text data, and extract the standard three-dimensional Gaussian primitives, the expression coefficients and the audio feature vectors;

[0040] The model training module is used to construct and train the dual-branch personalized temporal network, the temporal feature enhancement module, and the general motion field, and to implement the adaptive fusion mechanism and various loss constraints.

[0041] The inference rendering module is used to load the frozen general motion field and the trained personalized temporal network obtained by the model training module, generate the dynamic three-dimensional Gaussian representation according to the driving signal and render it to output a 3D digital human video.

[0042] Optionally, the model training module includes:

[0043] A general motion field unit is used to perform pre-training to learn the common facial movement patterns;

[0044] A personalized temporal network unit is used to implement the dual-branch network structure to generate the first deformation and the second deformation;

[0045] The fusion and optimization unit is used to implement the adaptive fusion mechanism and calculate the loss corresponding to the identity differentiation loss, the multi-scale geometric consistency constraint, the temporal consistency constraint, and the color consistency constraint.

[0046] Compared with the prior art, the present invention has the following advantages and technical effects:

[0047] This invention proposes an audio-driven digital human generation technology based on temporal feature enhancement and two-stage adaptive training, which effectively solves the problems of poor model adaptability, temporal jitter in generated images, and blurred details in existing technologies under conditions with few samples. By introducing a temporal feature enhancement module and a dual-branch network structure, the coherence and smoothness of dynamic sequences are significantly improved, avoiding artifacts at the boundary between the mouth and face. Combined with general motion field pre-training and personalized fast adaptive strategies, high-fidelity and high-synchronization digital human customization can be completed with only a small amount of target data, greatly reducing data requirements and training costs. At the same time, through multi-scale geometric regularization constraints, the topological correctness and texture detail clarity of the generated face during motion are effectively guaranteed, thus improving the overall visual quality, temporal stability, and application feasibility of digital human synthesis. Attached Figure Description

[0048] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the system structure according to an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram of the reasoning application process of the present invention in practical application scenarios such as AI news broadcasting and intelligent customer service interaction.

[0052] Figure 4 The image shows a comparison of real video frames from embodiments of the present invention, the rendering results of the present invention, and the generation effects of existing mainstream methods. Detailed Implementation

[0053] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a method for generating 3D digital human heads based on audio-driven temporal feature enhancement, including the following steps:

[0057] Obtain the video sequence and corresponding audio of the target object, initialize standard three-dimensional Gaussian primitives based on the video sequence, extract expression coefficients, and extract audio feature vectors based on the audio.

[0058] A dual-branch personalized temporal network is constructed, wherein the first branch outputs a first deformation based on the standard three-dimensional Gaussian unit, the audio feature vector, and the expression coefficient; the second branch inputs the standard three-dimensional Gaussian unit and time information into the temporal feature enhancement module, and then outputs a second deformation by combining the audio feature vector and the expression coefficient.

[0059] A general motion field is pre-trained based on a multi-identity dataset to learn common facial motion patterns across identities. During the pre-training process, the dual-branch personalized temporal network is combined with the first deformation to calculate the identity discrimination loss, so that the general motion field focuses on the common facial motion patterns.

[0060] The personalized temporal network is trained based on the video sequence of the target object; in the personalized training, the pre-trained and frozen general motion field is called to generate basic deformation, and the basic deformation is fused with the second deformation through an adaptive fusion mechanism to generate a dynamic three-dimensional Gaussian representation of the target object.

[0061] Based on the input driving audio, a 3D digital human video of the target object is generated and rendered through the frozen general motion field, the trained personalized temporal network, and the adaptive fusion mechanism.

[0062] Specifically, this embodiment first performs data preprocessing and initialization. The original video sequence and corresponding audio signal are obtained as input. For the audio, a pre-trained speech recognition model is used to extract the audio feature vector for each frame. For the video frames, head pose parameters, table coefficients, and camera intrinsic parameters are extracted for each frame. Simultaneously, the pre-trained model is used to extract the depth and normals for each frame. Gaussian elements are used as a 3D representation of the speaker's identity. The Gaussian elements used for canonical representation are denoted as... These represent position, rotation, scaling, opacity, and color attributes, respectively.

[0063] Furthermore, constructing a dual-branch personalized temporal network includes:

[0064] The first branch is constructed by encoding the position of the standard three-dimensional Gaussian unit and fusing it with the audio feature vector and the expression coefficient, and then predicting the first deformation through a multilayer perceptron network.

[0065] The second branch is constructed by inputting the position and time information of the standard three-dimensional Gaussian units into the temporal feature enhancement module. The temporal feature enhancement module extracts temporal smoothing features through a self-attention mechanism and fuses the temporal smoothing features with the audio feature vector and the expression coefficient. The second deformation is predicted through a multilayer perceptron network that shares weights with the first branch.

[0066] Specifically, the personalized temporal network constructed in this embodiment differs from the single prediction path of existing technologies. This module designs a dual-branch structure with shared weights. The first branch is the basic deformation branch, which first determines the position of the standard three-dimensional Gaussian unit. Input position encoder Perform high-dimensional mapping to obtain position-encoded features. Simultaneously, obtain audio feature vectors. and expression coefficient Subsequently, a channel attention mechanism was used to encode positional features. With driving characteristics Feature fusion is performed. The fused features are then input into a deformation network based on a three-plane representation, denoted as [network name missing]. . It includes a three-plane encoder and a multilayer perceptron decoder to establish a nonlinear mapping between audio and geometric features, ultimately outputting the first set of fundamental offsets. .

[0067]

[0068] The second branch is the timing correction branch, which introduces a timing feature enhancement module. This unit first adjusts the standard Gaussian position... and time Input to converter In this process, a feature input transformer encoder is introduced. By leveraging its self-attention to capture the temporal correlation between video frames, temporally smoothed coordinate features are calculated; subsequently, these features are encoded and input into a shared-weights three-plane deformation network. In the middle, output the second set of offsets after time-enhanced processing. .

[0069]

[0070] Furthermore, calculating the identity differentiation loss using the first deformation includes:

[0071] Calculate the similarity between the first deformations predicted by a dual-branch personalized temporal network for different identity data;

[0072] Similarity is penalized to encourage the general motion field to learn common facial motion patterns that are independent of identity, while encouraging the first branch to learn identity-specific features.

[0073] Specifically, to address the need for rapid customization with limited sample data, this embodiment employs a two-stage strategy: "general motion field pre-training + personalized rapid adaptation." In the first stage, the cross-identity general training stage, the system trains a general motion field using a large-scale dataset with multiple identities. Specifically, the network receives positional encodings of standard Gaussian locations. Audio feature vectors and expression coefficient As input, an attention mechanism module is introduced to effectively integrate geometric and driving signals. Using audio and expressions Location features Weighted modulation is performed. The modulated features are then input into a deep multilayer perceptron network, denoted as... A nonlinear mapping is performed. The final output is a common Gaussian attribute offset shared across identities, denoted as... The offset Specifically, this includes position offset. Rotational offset and scaling offset ,Right now .

[0074] At this stage, the invention introduces a negative correlation loss function, which forces the decoupling of the general motion field and the personalized module by penalizing the similarity of personalized features between different identities, thus preventing the personalized module from negatively impacting the general motion. Simultaneously, the invention also introduces a temporal feature consistency loss. Specifically, for the image sequence rendered by the Gaussian offset output of the temporal correction path, the sum of the L1 norm distances between the rendered image and the real image is calculated; this loss serves as a specific temporal constraint.

[0075] Furthermore, fusing the basic deformation with the second deformation through an adaptive fusion mechanism includes:

[0076] Construct an adjustment network to predict the spatial scaling factor based on the position of the standard 3D Gaussian elements;

[0077] The basic deformation is weighted and adjusted using the spatial scaling factor.

[0078] The weighted base deformation, the second deformation, and the position of the standard 3D Gaussian primitive are added together to obtain the dynamic Gaussian primitive position used for rendering.

[0079] Specifically, in the second stage, the personalized adaptive training stage, the parameters of the general motion field are frozen, and the personalized module is fine-tuned using only a small amount of data from the target object. To naturally transfer the general basic movements to the unique facial geometry of the target object, this embodiment constructs an adjustment network that predicts an adaptive scaling factor based on the Gaussian position. Weighted blending of general deformations is performed. The final Gaussian position used for rendering is... The calculation is as follows:

[0080]

[0081] Through this "soft projection" fusion method, the system can preserve the unique micro-expressions of the target person while supplementing the missing lip movements due to insufficient data by using general priors.

[0082] Furthermore, the personalized training also includes a step of introducing multi-scale geometric consistency constraints, specifically including:

[0083] Based on the video sequence of the target object, obtain the pseudo-true values ​​of the depth map and normal map;

[0084] Based on the dynamic 3D Gaussian representation rendering, a predicted depth map and a predicted normal map are obtained;

[0085] Calculate the normalized consistency loss between the predicted depth map and the pseudo-true value of the depth map at the local block level and the global image level, respectively.

[0086] Calculate the orientation similarity loss between the predicted normal map and the pseudo-true value of the normal map.

[0087] Specifically, in the rendering optimization stage, this embodiment addresses the common depth ambiguity and structural collapse issues in monocular video training through multi-scale geometric regularization. The system utilizes a monocular depth estimation network to generate pseudo-ground value labels and applies multi-level statistical normalization constraints to the rendered depth map. Specifically, to ensure the fineness of local textures, a block-level depth normalization formula is defined as follows: To maintain the stability of the global facial structure, the image-level depth normalization formula is defined as follows: ;in and These represent the mean and standard deviation of the corresponding region depth, respectively. The cosine similarity loss between the rendered normal map and the predicted normal map is also calculated. These geometric regularization techniques force the generated digital human to maintain the correct facial topology during movement, ultimately outputting a lip-synced and temporally stable digital human video.

[0088] Furthermore, personalized training also includes a step of introducing temporal consistency constraints to the second branch, specifically including:

[0089] Based on the second deformation, a series of consecutive frames of images are rendered;

[0090] The difference between consecutive rendered images and the corresponding real video frame sequences is calculated and used as a loss function to optimize the temporal feature enhancement module.

[0091] Furthermore, personalized training also includes the step of introducing color consistency constraints, specifically including:

[0092] The predicted image is obtained based on dynamic 3D Gaussian representation rendering;

[0093] Calculate the pixel-level loss between the predicted image and the real video frame;

[0094] Calculate the structural similarity loss between the predicted image and the real video frame.

[0095] Furthermore, generating and rendering a 3D digital human video of the target object includes:

[0096] If the input is text, the text is first synthesized into driving audio;

[0097] The driving audio is subjected to feature extraction to obtain a driving audio feature sequence;

[0098] The driving audio feature sequence and corresponding facial expression coefficients are input into the frozen general motion field and the trained personalized temporal network to obtain the basic deformation and the second deformation, respectively.

[0099] The basic deformation is fused with the second deformation through the adaptive fusion mechanism to obtain the dynamic Gaussian elements of each frame;

[0100] Using 3D Gaussian sputtering rendering technology, the dynamic Gaussian primitives are rendered into continuous image frames, which are then synchronized with audio to form a video.

[0101] Specifically, it can be divided into the following five steps:

[0102] S1 Data Preprocessing and Primitive Initialization. Multimodal feature extraction is performed on a small number of video samples of the target object. First, the pre-trained speech recognition model Deepspeech is used to extract the corresponding audio feature vectors frame by frame, and a 3D face fitting model is used to extract expression parameters and camera intrinsics. Simultaneously, a monocular depth estimation network is used to generate pseudo-ground depth maps and normal maps as high-quality labels for subsequent geometric supervision. Based on this, a static 3D Gaussian primitive set in standard space is constructed, and geometric attributes such as position, rotation, and scaling are initialized as the basic framework. These operations are performed by the preprocessing unit.

[0103] S2 Temporal Feature Enhancement Network Construction. A two-branch network is constructed, comprising a basic deformation path and a temporal correction path. The transformer encoder in the temporal feature enhancement module captures long-term temporal dependencies between video frames, generating time-smoothed coordinate features. Combined with the basic deformation network, it predicts basic geometric deformations and personalized micro-expression shifts with temporal consistency in parallel. This step aims to solve the image flickering problem in dynamic sequences caused by traditional methods, ensuring the continuity of high-frequency details such as teeth and tongue.

[0104] S3 General Motion Field Construction and Feature Decoupling. In the pre-training of the general motion field, a cross-identity general motion field is constructed using a large-scale multi-identity dataset. This field employs a deep multilayer perceptron structure, specifically designed to capture common basic motion patterns such as mouth opening and closing. During joint training, identity-negative correlation units are introduced to calculate the similarity between personalized and general features. A negative correlation loss function forces the personalized network to focus only on the unique textures and micro-movements of a specific identity, effectively decoupling general movements from personalized features and laying the foundation for rapid transfer from a small number of samples.

[0105] S4 Adaptive Fusion and Geometric Iterative Optimization. The system enters personalized fast adaptive training, freezing general motion field parameters as prior knowledge. A modulation network predicts the space using an adaptive scaling factor, weighting and fusing general movements into personalized offsets via a "soft projection" mechanism to generate the final dynamic Gaussian attributes. Subsequently, based on a small amount of data from the target object, block-level and image-level depth and normal constraints are introduced using a geometric consistency unit. Combined with a color consistency unit and a temporal constraint unit, the fused dynamic Gaussian is iteratively optimized to prevent structural collapse, ultimately generating a high-fidelity, lip-synced, and temporally stable digital human model.

[0106] S5 Inference-Driven Video Generation. This system constructs a complete inference application workflow for real-world business scenarios. The data input unit receives user-input text or speech signals. For text input, the system first converts it into a speech waveform with the target timbre using a built-in speech synthesis module. Then, it extracts frame-by-frame aligned audio feature sequences using an audio coding model consistent with the training phase. Joint inference is performed using a trained and optimized personalized temporal network and a frozen general motion field, predicting in parallel the target person's unique personalized micro-expression shifts and general basic movements, and achieving adaptive fusion through a modulation network. Finally, the Gaussian rendering unit uses 3D Gaussian sputtering technology to project the fused dynamic Gaussian primitives onto a 2D image plane, quickly generating high-fidelity digital human broadcast videos with accurate lip-sync, stable facial structure, and clear texture, achieving end-to-end automated content production. This system is suitable for scenarios with high real-time and image quality requirements, such as intelligent news broadcasting, online education virtual lecturers, and metaverse instant messaging. It also supports multilingual cross-language driving, demonstrating broad industry applicability and expansion potential.

[0107] Example 2

[0108] like Figures 2-3 As shown, this embodiment also provides an audio-driven 3D digital human head generation system based on temporal feature enhancement, used to implement the method, including:

[0109] The data preprocessing module is used to acquire and process video, audio and text data, and extract the standard three-dimensional Gaussian primitives, the expression coefficients and the audio feature vectors;

[0110] The data preprocessing module includes a preprocessing unit for cleaning and feature extraction of the input video. This unit fits each frame of the video, extracting head pose parameters, camera intrinsic parameters, and facial expression coefficients. Simultaneously, the DeepSpeech audio coding model is used to extract audio feature vectors synchronized with video frames. The Sapiens method is used to generate geometric supervision information for video frames, including depth maps and normal maps.

[0111] The model training module is used to construct and train the dual-branch personalized temporal network, the temporal feature enhancement module, and the general motion field, and to implement the adaptive fusion mechanism and various loss constraints.

[0112] The model training module includes:

[0113] The personalized temporal network unit is used to capture personalized facial movements and micro-expression details of specific target individuals. This network employs a two-branch structure, establishing a mapping from audio features to Gaussian attribute offsets through an internally integrated temporal feature enhancement module and a basic deformation network. This unit can generate a personalized deformation field optimized with temporal context information, addressing the flickering and high-frequency detail loss issues caused by single-frame prediction.

[0114] Temporal feature enhancement unit. This unit first uses a spatiotemporal encoder to transform the standard Gaussian position... With time step The data is mapped to high-dimensional features and then fed into a transform encoder. A self-attention mechanism is used to capture long-term temporal dependencies in the video sequence. Finally, a multilayer perceptron outputs temporally corrected coordinate features. These features are then fed into a basic deformable network to guide the generation of temporally consistent Gaussian shifts.

[0115] The basic deformable network unit uses a three-plane structure to encode Gaussian positions and employs a three-layer perceptron as the base. As the main component for deformation prediction, the network receives standard Gaussian positions encoded with a three-plane structure, or positions corrected for time series, and audio features. and expression coefficient As input, multimodal features are fused using a channel attention mechanism to predict the basic offset of Gaussian points in terms of position, rotation, and scaling attributes. During training, the network shares weights with the temporal feature enhancement module and outputs the basic deformation and temporal enhanced deformation in parallel.

[0116] The General Motion Field Unit is used to learn universal facial motion patterns shared across identities. Unlike the basic deformable network, this unit uses frequency encoding for Gaussian positions, and the network architecture adopts a deep multilayer perceptron architecture. With the encoded position code, audio, and facial expression features as input, it outputs a universal Gaussian attribute offset.

[0117] The fusion and optimization unit is used to implement the adaptive fusion mechanism and calculate the loss corresponding to the identity differentiation loss, the multi-scale geometric consistency constraint, the temporal consistency constraint, and the color consistency constraint.

[0118] Specifically, the fusion and optimization unit includes:

[0119] Color consistency unit. Used to constrain the visual similarity between the rendered image and the real image. The unit calculates the L1 norm loss and structural similarity loss between the rendered image and the reference image, ensuring fidelity in color and local structure. Let be the weighting coefficients, then the color consistency loss is:

[0120]

[0121] Temporal Constraint Unit. Used to supervise the output of the temporal feature enhancement module and prevent timing jitter. This unit calculates the continuous output generated by the temporal enhancement path. The L1 norm loss between the frame-rendered image and the real image sequence forces the Gaussian properties to maintain a smooth transition over time, eliminating high-frequency flicker artifacts in video generation. Its loss function is denoted as:

[0122]

[0123] Identity-negative correlation unit. Used to achieve feature decoupling during the general training phase. This unit calculates the inner product similarity between personalized deformation features of different identities and penalizes high similarity. By maximizing the differences in features across different identities, the general motion field is forced to capture common features, while the personalized network is forced to focus only on the unique details of a specific person. For personalized offsets, the loss function is denoted as:

[0124]

[0125] Geometric Consistency Unit. This unit utilizes the monocular depth estimation network Sapiens to generate pseudo-ground depth maps and normal maps. To balance local details and global structure, this unit designs multi-scale depth constraints: local depth supervision divides the depth map into multiple blocks, calculating block-level normalized depth consistency. Global depth supervision computation achieves normalized depth consistency across the entire graph. Simultaneously, by incorporating cosine similarity loss along the normal direction, the accuracy of facial topology is ensured. The total geometric consistency loss is denoted as: where is the weight.

[0126]

[0127] The inference rendering module is used to load the frozen general motion field and the trained personalized temporal network obtained by the model training module, generate the dynamic three-dimensional Gaussian representation according to the driving signal and render it to output a 3D digital human video.

[0128] Specifically, the inference rendering module includes:

[0129] The joint inference unit loads the trained model parameters and performs forward prediction from audio features to dynamic Gaussian attributes. During the inference phase, this unit calls upon the frozen general motion field and the optimized personalized temporal network to predict the basic lip movement offset across identities and the personalized micro-expression offset of the target person, respectively. It also predicts the spatial scaling factor through a modulation network, and then weights and fuses the two to calculate the final dynamic Gaussian attributes for each frame, achieving real-time combination of general priors and personalized features.

[0130] The rendering and compositing unit is used to convert the dynamic Gaussian units generated by inference into visualized digital human videos. This unit rapidly generates consecutive image frames and then aligns and synthesizes the rendered image sequence with the input audio signal to output a digital human speaking video with accurate lip-sync, stable facial structure, and clear texture details, enabling end-to-end applications that input text / speech and output video instantly.

[0131] This embodiment also included specific experimental verification, and the verification results are as follows: Figure 4 As shown. Figure 4 This figure shows a comparison of the generation results of the embodiments of the present invention and existing mainstream methods on a real test set. The figure illustrates the visual comparison of mouth details in the rendering results of homonyms using real data and various methods, and the peak signal-to-noise ratio (PSNR) is marked. It can be seen that the present invention is more accurate in restoring tooth texture and mouth closure, and the PSNR value is significantly better than the comparison methods, demonstrating the advantage of the present invention in realistically restoring high-frequency details under conditions of a small number of samples.

[0132] The present invention achieves the following technical effects:

[0133] This invention significantly improves the lip-sync accuracy, texture detail fidelity, and temporal stability of audio-driven digital human generation by constructing a dynamic Gaussian representation model based on temporal feature enhancement and combining a two-stage training strategy of general motion field pre-training and personalized adaptation. Experimental results show that the technical solution of this invention outperforms existing methods on multiple public datasets, especially in terms of geometric structure preservation, high-frequency texture restoration, and dynamic coherence under conditions of few samples. It can be widely applied in fields such as virtual anchor generation, instant messaging, metaverse identity construction, and intelligent human-computer interaction, and has good application value and economic benefits. The digital human facial images rendered by the method of this invention show excellent performance in terms of visual clarity and lip-sync accuracy. The generated teeth, tongue, and other oral cavity details are clearly visible without obvious blurring or structural breaks.

[0134] On the one hand, the personalized temporal network proposed in this invention adopts a dual-branch parallel design of basic deformation and temporal correction, effectively solving the problems of image flickering and high-frequency detail loss commonly found in traditional 3D Gaussian methods in dynamic sequences. Qualitative experimental results show that the Gaussian units generated by this invention have smooth transitions in the temporal dimension, effectively reducing inter-frame jitter. Especially in the boundary region between the mouth and face, thanks to the unified modeling of the whole face and the temporal attention mechanism, this invention can accurately reconstruct complex non-rigid deformations, avoiding texture slippage or splicing artifacts caused by "mouth-face" separation common in existing technologies, and ensuring the integrity and naturalness of the facial topology.

[0135] On the other hand, thanks to the two-stage strategy of general motion field pre-training and personalized adaptation, along with multi-scale geometric constraints, this invention achieves fast and high-fidelity model customization with limited sample data. In comparative experiments with existing mainstream methods such as ER-NeRF, MimicTalk, GaussianTalker, and InsTaG, this invention achieves a PSNR of 30.61 dB and a SSIM of 0.925, both at the best levels. Simultaneously, it reduces the LMD (lip-sync accuracy) metric to 3.193, significantly outperforming similar methods. Furthermore, the average training time of this invention is only about 18 minutes, achieving comparable efficiency to lightweight methods while ensuring a significant improvement in image quality. These results demonstrate that this invention achieves the optimal balance between reconstruction accuracy, geometric stability, and training efficiency.

[0136] This invention achieves the following specific technical effects through a temporal feature enhancement and two-stage adaptive training technique:

[0137] Enhanced dynamic expression and temporal consistency: By designing a personalized temporal network, the temporal jitter problem in traditional methods is effectively solved, ensuring the dynamic coherence and smoothness of the generated video.

[0138] Rendering quality and geometry optimization: A multi-scale geometry regularization mechanism is introduced to overcome geometric ambiguity under training with a small number of samples, significantly improve texture fidelity, and effectively prevent facial structure collapse.

[0139] Rapid adaptation capability with a small number of samples: A two-stage strategy of general sports field pre-training and personalized rapid adaptation is used to achieve the ability to quickly transfer and generate digital humans with only a small amount of data, which greatly reduces the data threshold and training cost.

[0140] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating 3D digital human heads based on audio-driven temporal feature enhancement, characterized in that, Includes the following steps: Obtain the video sequence and corresponding audio of the target object, initialize standard three-dimensional Gaussian primitives based on the video sequence, extract expression coefficients, and extract audio feature vectors based on the audio. A dual-branch personalized temporal network is constructed, wherein the first branch outputs a first deformation based on the standard three-dimensional Gaussian unit, the audio feature vector, and the expression coefficient; the second branch inputs the standard three-dimensional Gaussian unit and time information into the temporal feature enhancement module, and then outputs a second deformation by combining the audio feature vector and the expression coefficient. A general motion field is pre-trained based on a multi-identity dataset to learn common facial motion patterns across identities. During the pre-training process, the dual-branch personalized temporal network is combined with the first deformation to calculate the identity discrimination loss, so that the general motion field focuses on the common facial motion patterns. The personalized temporal network is trained based on the video sequences of the target object. In personalized training, the pre-trained and frozen parameters of the general motion field are invoked to generate basic deformation, and the basic deformation is fused with the second deformation through an adaptive fusion mechanism to generate a dynamic three-dimensional Gaussian representation of the target object. Based on the input driving audio, a 3D digital human video of the target object is generated and rendered through the frozen general motion field, the trained personalized temporal network, and the adaptive fusion mechanism.

2. The method according to claim 1, characterized in that, The construction of a dual-branch personalized temporal network includes: The first branch is constructed by encoding the position of the standard three-dimensional Gaussian unit and fusing it with the audio feature vector and the expression coefficient, and then predicting the first deformation through a multilayer perceptron network. The second branch is constructed by inputting the position and time information of the standard three-dimensional Gaussian units into the temporal feature enhancement module. The temporal feature enhancement module extracts temporal smoothing features through a self-attention mechanism and fuses the temporal smoothing features with the audio feature vector and the expression coefficient. The second deformation is predicted through a multilayer perceptron network that shares weights with the first branch.

3. The method according to claim 1, characterized in that, Calculating the identity differentiation loss using the first deformation includes: Calculate the similarity between the first deformations predicted by a dual-branch personalized temporal network for different identity data; Similarity is penalized to encourage the general motion field to learn common facial motion patterns that are independent of identity, while encouraging the first branch to learn identity-specific features.

4. The method according to claim 1, characterized in that, The fusion of the basic deformation and the second deformation through an adaptive fusion mechanism includes: Construct an adjustment network to predict the spatial scaling factor based on the position of the standard 3D Gaussian elements; The basic deformation is weighted and adjusted using the spatial scaling factor. The weighted base deformation, the second deformation, and the position of the standard 3D Gaussian primitive are added together to obtain the dynamic Gaussian primitive position used for rendering.

5. The method according to claim 1, characterized in that, The personalized training also includes a step of introducing multi-scale geometric consistency constraints, specifically including: Based on the video sequence of the target object, obtain the pseudo-true values ​​of the depth map and normal map; Based on the dynamic 3D Gaussian representation rendering, a predicted depth map and a predicted normal map are obtained; Calculate the normalized consistency loss between the predicted depth map and the pseudo-true value of the depth map at the local block level and the global image level, respectively. Calculate the orientation similarity loss between the predicted normal map and the pseudo-true value of the normal map.

6. The method according to claim 2, characterized in that, Personalized training also includes a step of introducing temporal consistency constraints to the second branch, specifically including: Based on the second deformation, a series of consecutive frames of images are rendered; The difference between consecutive rendered images and the corresponding real video frame sequences is calculated and used as a loss function to optimize the temporal feature enhancement module.

7. The method according to claim 1, characterized in that, Personalized training also includes the step of introducing color consistency constraints, specifically including: The predicted image is obtained based on dynamic 3D Gaussian representation rendering; Calculate the pixel-level loss between the predicted image and the real video frame; Calculate the structural similarity loss between the predicted image and the real video frame.

8. The method according to claim 1, characterized in that, The generation and rendering of the 3D digital human video of the target object includes: If the input is text, the text is first synthesized into driving audio; The driving audio is subjected to feature extraction to obtain a driving audio feature sequence; The driving audio feature sequence and corresponding facial expression coefficients are input into the frozen general motion field and the trained personalized temporal network to obtain the basic deformation and the second deformation, respectively. The basic deformation is fused with the second deformation through the adaptive fusion mechanism to obtain the dynamic Gaussian elements of each frame; Using 3D Gaussian sputtering rendering technology, the dynamic Gaussian primitives are rendered into continuous image frames, which are then synchronized with audio to form a video.

9. An audio-driven 3D digital human head generation system based on temporal feature enhancement, used to implement the method of any one of claims 1 to 8, characterized in that, The system includes: The data preprocessing module is used to acquire and process video, audio and text data, and extract the standard three-dimensional Gaussian primitives, the expression coefficients and the audio feature vectors; The model training module is used to construct and train the dual-branch personalized temporal network, the temporal feature enhancement module, and the general motion field, and to implement the adaptive fusion mechanism and various loss constraints. The inference rendering module is used to load the frozen general motion field and the trained personalized temporal network obtained by the model training module, generate the dynamic three-dimensional Gaussian representation according to the driving signal and render it to output a 3D digital human video.

10. The system according to claim 9, characterized in that, The model training module includes: A general motion field unit is used to perform pre-training to learn the common facial movement patterns; A personalized temporal network unit is used to implement the dual-branch network structure to generate the first deformation and the second deformation; The fusion and optimization unit is used to implement the adaptive fusion mechanism and calculate the loss corresponding to the identity differentiation loss, the multi-scale geometric consistency constraint, the temporal consistency constraint, and the color consistency constraint.

Images