A single-video-based personalized digital human generation method

Abstract

The application discloses a personalized digital human generation method based on a single video, relates to the field of virtual digital human modeling and driving, and aims at video and voice data of a target person, realizes high synchronization and natural authenticity of facial expression and voice content of a digital human through a voice-driven synchronization mechanism with a multi-modal alignment constraint and an expression generation model combined with semantic understanding and emotion labels. In model training, a self-supervised style consistency constraint is added to ensure that the generated image style is stable, and a 3D semantic mask is added in the image fusion stage to improve the accuracy and realistic effect of the fusion of the synthesized facial expression and the reference face. The application can quickly clone a digital human and drive it to synthesize expressions with voice based on a single video sample, the generation process is efficient, and the obtained digital human video has excellent realism and interactivity.

Description

Technical Field

[0001] This invention relates to the field of virtual digital human modeling and driving, and in particular to a method for generating personalized digital humans based on a single video. Background Technology

[0002] In the fields of computer vision and artificial intelligence, personalized digital human generation technology is one of the core research directions for constructing virtual characters and enhancing human-computer interaction. Currently, the mainstream technical approaches are mainly divided into two categories: one is the traditional approach based on 3D modeling and hybrid deformation, and the other is the generation approach based on 2D keypoint-driven or end-to-end neural networks. Traditional 3D modeling methods can achieve precise facial expression control by manually constructing high-precision 3D face models and predefining BlendShape parameters. However, these methods have significant drawbacks: firstly, the modeling process relies heavily on manual intervention, requiring the design of individual expression bases for each target character, resulting in long modeling cycles and high costs; secondly, they have poor generalization ability and cannot quickly adapt to new characters or complex expression combinations; thirdly, real-time rendering of 3D models requires extremely high hardware computing power, making it difficult to run smoothly on mobile devices or low-power devices. While end-to-end 2D generation methods have improved real-time performance in recent years by simplifying the modeling process, their limitations are also evident: on the one hand, direct mapping based on 2D keypoints or neural networks struggles to fully preserve the unique facial features of the target person, easily leading to an "average face" effect and loss of personalized details; on the other hand, such methods typically rely on massive amounts of training data to cover diverse facial expression patterns, making it difficult for the model to generate high-fidelity results with only a single video or a small number of samples. Furthermore, the "black box" nature of end-to-end systems results in insufficient controllability and interpretability of expression parameters, preventing developers from precisely adjusting expression details or diagnosing generation anomalies. Therefore, based on these challenges, this invention proposes a personalized digital human generation method based on a single video. Summary of the Invention

[0003] Technical Purpose To address the aforementioned issues, the present invention aims to provide a personalized digital human generation method based on a single video. This method addresses the limitations of digital human applications caused by high modeling costs, low data efficiency, and insufficient real-time performance. It enables the rapid construction of a high-fidelity digital image of a target person from a single video and real-time synchronization of facial expressions and lip movements. While reducing reliance on 3D modeling and massive datasets, it improves generation efficiency and cross-platform compatibility through modular design and lightweight deployment strategies.

[0004] Technical solution To achieve the above objectives, this invention provides a personalized digital human generation method based on a single video. This method pre-acquires a video of a target person and constructs a digital facial model of that person. During model training, a self-supervised style consistency constraint is introduced to ensure stable generation style. Acoustic features of the target person's speech input are extracted, and semantic understanding and emotion recognition are performed on the speech content. Based on the acoustic features, semantic and emotional information, a corresponding target face image is generated under multimodal alignment constraints. Finally, the target face image is fused to the target person's baseline face using a 3D semantic mask to output a personalized digital human video.

[0005] In a first aspect, the present invention provides a method for generating personalized digital humans based on a single video, comprising: Acquire a video or several images of the target person and construct a digital facial model of the target person. In the model training process, introduce a self-supervised style consistency loss to ensure the stability and consistency of the generated style. Acquire the voice input of the target person and extract acoustic feature parameters; The voice input is subjected to semantic understanding and emotion recognition to obtain corresponding semantic information and emotion tags; Under multimodal alignment constraints, the acoustic feature parameters, semantic information, and emotion tags are mapped to corresponding facial expression parameter sequences; wherein a dynamic semantic feature control mechanism is introduced for the semantic stress and emotion intensity of the speech content to adjust the amplitude and details of the generated expressions; Based on the facial expression parameter sequence, a facial image of the target person is generated through the digital human facial model; the generation process includes a differentiable rendering layer, end-to-end expression synthesis of key facial regions, and style transfer constraints to ensure that the generated local expression image is consistent with the overall appearance style of the target person. The facial image of the target person is fused into a pre-stored reference facial image of the target person using a 3D semantic mask. The fusion and overlay of local expression areas with the reference face is completed at the pixel level, and a personalized digital human video synchronized with the voice input content is output.

[0006] Furthermore, a digital facial model of the target person is constructed through an adaptive principal component selection mechanism, and the number of basis vectors required for principal component analysis is dynamically determined according to a preset reconstruction error threshold or variance retention rate. The reconstruction error threshold is adjusted within the range of 3% to 8%, and the variance retention rate is adjusted within the range of 90% to 98%. The specific values ​​are adaptively set according to the richness of facial expression changes in the target video, thereby maximizing the compression of model dimensions to improve modeling efficiency while ensuring the accuracy of facial expression reconstruction.

[0007] Furthermore, the calculation of the style consistency loss adopts a multi-scale style feature comparison strategy. It measures the style difference by acquiring multiple levels of features from the pre-trained convolutional neural network and calculating the style representation difference between the corresponding level features of the generated image and the original image of the target person. The style difference of different facial regions is assigned a weighted coefficient based on the texture complexity of that region.

[0008] Furthermore, the multi-scale style feature comparison strategy specifically includes: The generated image and the original image of the target person are extracted and mapped at multiple levels using a pre-trained convolutional neural network. Calculate the Gram matrix difference of the feature maps at each corresponding level, and use it as the style loss component for that level. Based on the texture complexity of different areas of the face, corresponding weighting coefficients are assigned to the style loss components of each level or region, with regions having richer texture details being given greater weights. Finally, the self-supervised style consistency loss is the sum of all weighted style loss components, thereby ensuring that the generated image is consistent with the target character in both overall style and local texture details, significantly reducing inter-frame style flickering.

[0009] Furthermore, the speech input is preprocessed and divided into consecutive short time frames, and Mel spectral coefficients or MFCC coefficients are extracted as acoustic features. These acoustic features are input into the audio-driven model of a deep neural network to regress and output a sequence of facial expression parameters. The audio-driven model includes a long short-term memory network layer to capture the temporal dependence of the speech signal, and further includes a speaker adaptation mechanism to fine-tune some parameters of the model under the condition of providing only a small amount of speech-expression correspondence data of the target speaker, thereby adjusting the expression parameter mapping to fit the speaker's pronunciation style and expression habits.

[0010] Furthermore, the audio-driven model also includes an adjustable mapping unit for different speakers' speech rates and intonation rhythms. The mapping unit dynamically adjusts the output rate and smoothness of the facial expression parameter sequence according to the phoneme duration, speech rate, and intonation fluctuations of the speech signal. By introducing additional features or modules that reflect speech rhythm, it ensures a rapid response and stable transition in mouth movements generated at fast speech rates, and maintains smooth and natural facial expression changes at slow speech rates.

[0011] Furthermore, the audio-driven modeling process can also employ Transformer networks, temporal convolutional networks, or other equivalent alternative structures.

[0012] Furthermore, the process of mapping acoustic feature parameters, semantic information, and emotion tags into facial expression parameter sequences is implemented by an expression generation network driven by a cross-modal attention mechanism. The network assigns dynamic weights to multimodal information by setting the speech feature sequence as the query and the text semantic feature sequence as the key and value through an attention model.

[0013] Furthermore, the process of using semantic information and emotional tags to drive expression generation includes a contextual semantic linkage mechanism, which dynamically adjusts additional facial expression features such as eyebrows and eyes based on the semantic focus and emotional category in the speech content.

[0014] Furthermore, the process of generating a target facial expression image using a digital human facial model includes using a differentiable rendering layer to perform expression synthesis rendering on local facial regions; the differentiable rendering layer receives the facial expression parameter sequence and combines it with the target person's facial model to generate a corresponding target person's expression image frame.

[0015] Furthermore, the image fusion using a 3D semantic mask specifically includes: Step 1: Based on the pre-constructed 3D face model of the target person, extract the 3D semantic mask of the mouth and eye area; Step 2: Dynamically adjust the geometry and spatial position of the mask according to the facial expression parameters of the current frame to adapt it to the opening and closing state of the target mouth. Step 3: The generated target face local expression image and the pre-stored reference face image are pixel-level blended within the fusion area defined by the three-dimensional semantic mask, and the orientation and scale alignment consistency between the fusion area and the reference face is verified by the spatial transformation matrix.

[0016] Furthermore, when the 3D semantic mask is dynamically adjusted according to the facial expression parameters of the current frame, a trained deformation prediction network is used to replace the predefined fixed deformation model. The deformation prediction network is based on a convolutional neural network structure, which takes the facial expression parameters as input and outputs the corresponding mask deformation field. The mask is geometrically adjusted by the deformation field to obtain a more accurate mask fitting result than a predefined deformation model through learning, thereby reducing the geometric misalignment rate between the mask and the target mouth region. Meanwhile, the pixel-level mixing of the image fusion adopts a multi-resolution cascade fusion strategy, which progressively fuses local expression images and reference face images at different spatial resolutions, specifically: At different spatial resolutions, the generated target facial local expression images are fused with the pre-stored reference facial images step by step; First, coarse alignment and blending are performed at a lower resolution, and then the resolution is gradually increased to refine the texture alignment and edge transition of the blended area. This strategy effectively improves the alignment accuracy and visual smoothness of the blended area in terms of detailed texture.

[0017] Furthermore, the method supports cross-platform deployment, including: Real-time inference is achieved on mobile devices through model compression and quantization. Supports high-concurrency requests through distributed computing in the cloud; Accelerate image synthesis on the web using WebAssembly or WebGL.

[0018] Secondly, the present invention also provides a personalized digital human generation system based on a single video, the system being based on the method described in the first aspect above, comprising: The video modeling module is used to construct a digital facial model of the target person based on a video or several images of the target person, and introduces a self-supervised style consistency loss during the model training process to ensure that the generated style is stable and consistent. The speech feature extraction module is used to extract the acoustic feature parameters of the speech input of the target person; The semantic and sentiment analysis module is used to perform semantic understanding and sentiment recognition on the voice input to obtain corresponding semantic information and sentiment tags; The expression generation module is used to map the acoustic feature parameters, semantic information, and emotion tags to corresponding facial expression parameters under multimodal alignment constraints, and generate a face image of the target person based on the facial expression parameters. The image fusion module is used to fuse the facial image of the target person into a pre-stored baseline facial image of the target person using a 3D semantic mask, and output a personalized digital human video.

[0019] Furthermore, the speech feature extraction module further supports multimodal input, including using audio generated from text-to-speech as a driving signal to achieve end-to-end mapping from text to digital human expressions.

[0020] Furthermore, it also includes an interpretable adjustment module for facial expression parameters, which constrains and refines the sequence of facial expression parameters; when the voice input is interrupted, the adjustment module maintains the current state of the expression parameters or smoothly transitions the expression parameters to a neutral value; when any expression parameter exceeds a preset reasonable range, the adjustment module adjusts it to within a threshold by means of pruning or interpolation correction; the module also provides several interpretable control indicators for real-time adjustment of expression intensity and style by manual or algorithmic means.

[0021] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the aforementioned method for generating personalized digital humans based on a single video.

[0022] This invention constructs a low-data-dependency, high-real-time digital human driving framework through the synergy of three major modules: appearance modeling, audio driving, and image synthesis. The video modeling module uses principal component analysis to extract the geometric and texture features of the target person from a single video segment, constructing a low-dimensional expression base space to replace traditional 3D modeling; the expression generation module uses a long short-term memory network to establish a temporal mapping from the Mel spectrum to expression parameters, achieving interpretable lip movement control; the image fusion module combines non-rigid deformation and generative adversarial networks to generate high-fidelity continuous frames through local geometric adaptation and texture refinement. This invention requires only a single video clip of the target person to clone and generate a digital human image, reducing data requirements and costs. By introducing semantic understanding and emotion-driven mechanisms, the generated digital human's facial expressions can match the speech content and the speaker's emotions, making it more vivid and realistic. Multimodal alignment constraints improve the synchronization accuracy between speech-driven lip movements and audio. Self-supervised style consistency loss ensures the style of the generated digital human remains stable and consistent across different frame sequences, avoiding image flickering and distortion. Furthermore, 3D semantic mask fusion improves the geometric matching degree and naturalness of face synthesis, thus significantly enhancing the realism and interactive experience of the digital human video overall. This solution, through a modular architecture and lightweight deployment strategy, supports cross-platform real-time operation, significantly reducing computing resource requirements and resolving issues of high data dependence, the contradiction between fidelity and efficiency, and weak platform adaptability, providing an efficient and robust solution for virtual interaction scenarios.

[0023] Beneficial effects By implementing the personalized digital human generation method based on a single video provided by the present invention, the following technical effects are achieved: (1) This application extracts the facial expression change patterns of a target person from a single video segment using principal component analysis, and constructs a low-dimensional orthogonal basis space representation to replace traditional 3D modeling or end-to-end black box models. This technology effectively reduces data requirements and computational complexity, while preserving personalized facial details, solving the dependence of traditional methods on multi-view data or large-scale labeled sets, and significantly improving modeling efficiency and generalization ability.

[0024] (2) A temporal regression model for converting Mel spectral features to low-dimensional facial expression parameters is established using a long short-term memory network. The basic combination of facial expression parameters is driven by transparent and controllable intermediate parameters. This design overcomes the limitation of the uninterpretability of end-to-end models, supports precise adjustment of facial expression parameters and avoids the "average face" effect. While ensuring the accuracy of lip movement synchronization, it enhances the controllability and robustness of the generation process.

[0025] (3) Combining expression-based mouth image generation with non-rigid deformation field technology, local geometric adaptation is achieved through keypoint displacement fields or 3D vertex displacement, and generative adversarial networks are used to refine texture fusion. This mechanism balances facial detail fidelity and real-time performance while avoiding high global rendering overhead, resolving the contradiction between the low efficiency of traditional 3D methods and the lack of detail in 2D methods.

[0026] (4) Through speaker adaptive capabilities and speech rate / rhythm adjustment mechanisms, the system automatically adjusts output facial expression parameters for different speakers and speech rates, ensuring the naturalness and accuracy of digital human facial expressions under various audio inputs, significantly improving the system's versatility and robustness. Furthermore, the provided interpretable adjustment module for facial expression parameters allows developers or the system to fine-tune the intensity and style of generated expressions as needed, while automatically constraining overshoot or undershoot of facial expression parameters in abnormal situations to avoid distortion. This interpretable and controllable facial expression adjustment capability makes digital human facial expression generation both accurate and flexible, combining technical rigor with practical application value.

[0027] (5) By decoupling the appearance modeling, audio driving, and image fusion modules, a standardized data interface and parallel pipeline architecture are constructed. Combined with model compression, hardware acceleration, and distributed computing technologies, it is adapted to deployment in multiple scenarios such as mobile, cloud, and web. This framework breaks through the strong dependence of existing solutions on high-performance hardware, realizes high frame rate real-time driving under low-power devices, and significantly expands the application boundaries of digital human technology.

[0028] (6) Only a single video of the target person is needed to clone and generate a digital human image, reducing data requirements and costs; by introducing semantic understanding and emotion-driven mechanisms, the facial expressions of the generated digital human can match the speech content and the speaker's emotions, making them more vivid and realistic; multimodal alignment constraints are used to improve the synchronization accuracy of speech-driven lip movements and audio; self-supervised style consistency loss is used to ensure that the style of the generated digital human is stable and consistent in different frame sequences, avoiding image flickering and distortion; and the combination of 3D semantic mask fusion improves the geometric matching degree and naturalness of face synthesis, thereby significantly improving the realism and interactive experience of digital human videos as a whole. Attached Figure Description

[0029] To make the above-described method for generating personalized digital humans based on single videos more apparent and understandable, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the method described in this application; Figure 2 This is a schematic diagram of the video modeling module structure. Figure 3 This diagram illustrates the structure of the speech feature extraction module. Figure 4 This diagram illustrates the structure of the image fusion module. Detailed Implementation

[0031] Example 1: A method for generating personalized digital humans based on a single video is provided, the method flow is as follows: Figure 1 As shown, the details are as follows.

[0032] 1. System module functional logic and collaborative process This application includes the following three main functional modules, each with the following logical functions, and they are interconnected to form a complete process.

[0033] The video modeling module is used to build the basic model of the digital human's image, with a structure as follows: Figure 2 As shown, this module takes a reference video or several static facial images of the target person as input and extracts facial features for modeling. On one hand, it uses face detection and keypoint extraction algorithms to obtain the geometric structure information of the person's face; on the other hand, it obtains the texture and appearance information of the person's face. Based on the extracted features, a parametric facial model of the person is established, including identity parameters and an expression basis space. The identity parameters represent the inherent facial shape and appearance of the target person; the expression basis space is used to represent the dynamic changes of the face, and can be characterized by a set of pre-trained basis functions. Preferably, a low-dimensional basis for expression changes is obtained through offline training using statistical modeling methods such as principal component analysis. For example, PCA modeling is performed on a large number of image sequences of the mouth region to obtain an average mouth image. and several orthogonal basis characteristics This is used to represent changes in lip shape. In practice, it can be taken as... The module outputs a person appearance model consisting of: neutral-state facial data of the target person, and basis representations of expression / mouth shape changes. It uses several key basis vectors to summarize the majority of mouth deformation features. and mean This data will serve as the basis for subsequent synthesis, enabling the generated digital human to have the same appearance characteristics as the target person.

[0034] A deep learning model is used to train and model multiple facial expression samples of the target person. During the training process, a self-supervised style consistency loss function is introduced to ensure that the model maintains consistency and stability with the image style of the target person when generating the target face image.

[0035] The speech feature extraction module is used to convert the input speech signal into a parameter sequence that drives changes in facial expressions in the digital human, with the structure as follows: Figure 3 As shown, this module first transcribes the speech into text, and then uses natural language processing (NLP) to perform semantic understanding, extracting semantic information that matches the speech content. Simultaneously, it performs emotion recognition on the pitch, energy, and rhythm features of the speech signal to obtain emotion tags reflecting the speaker's emotional state. Then, the extracted acoustic feature parameters, along with the semantic information and emotion tags, are input into the expression-driven model to generate corresponding facial expression parameters to drive the digital human facial model to produce expression changes. It also includes a parallel audio emotion analysis model, which classifies the speaker's emotions in the speech and outputs emotion tags by analyzing features such as pitch, volume changes, and speech rate. The output of the semantic and emotion analysis module is passed to the expression generation module along with the speech features, enabling the expression synthesis process to perceive semantic and emotional factors. Specifically, this includes: performing speech feature processing on the input audio: dividing the original digital audio signal into short frames synchronized with video frames, and calculating acoustic features such as Mel-spectrum coefficients for each frame to form an audio feature sequence. ,in Indicates the first The feature vectors of the audio frames at each time point are then used as inputs to a pre-trained speech-driven model. Through temporal modeling and regression, the corresponding expression parameter sequence is output. This is for Low-dimensional expression parameter vectors are used to control the digital human face in the first dimension. The facial expression states in the video frame. The dimension of the facial expression parameter vector is consistent with the dimension of the facial expression basis space provided by the video modeling module, preferably 6 dimensions. The audio-driven model is implemented using a deep neural network, including long short-term memory network layers to capture phoneme persistence and coarticulation effects, and regresses the facial expression parameters at each moment through several fully connected layers. During model training, a large amount of data with face videos and corresponding speech is used as samples to minimize the error between the predicted mouth parameters and the actual mouth parameters, so that the model learns the mapping relationship between audio and lip movement deformation. The model incorporates a few-sample speaker adaptation mechanism, which can fine-tune some network weights on a small amount of correction data of new speakers to quickly adapt to new voice characteristics and facial expression habits, thereby maintaining the consistency of the driving effect among different people; the model also has a built-in speech rate and pitch adjustment unit, which can dynamically adjust the rate of change of output facial expression parameters according to the real-time rhythm of the input speech, ensuring that lip shape changes are both sensitive and stable under different speech rates. After obtaining the facial expression parameters, the facial expression generation module applies the parameters to the digital human face model through a differentiable rendering layer to generate facial expression image frames. This rendering unit can call graphics shaders or neural networks to update facial textures and maintain consistency in style between frames and overall style through style constraints. At runtime, this module can quickly calculate a synchronized sequence of facial expression parameters based on any input speech; these parameters will drive the movement of the digital human's face in the next module. This audio-driven approach enables direct control of the digital human's lip movements and facial expressions by the speech content, ensuring strict semantic and temporal alignment between the generated digital human and the input speech.

[0036] To ensure strict synchronization between voice-driven facial movements and audio signals, a synchronization mechanism based on multimodal alignment constraints is employed during expression parameter prediction. This mechanism utilizes the correspondence between audio and video frames in the original video to correct the digital human generation process: when generating mouth-related expression parameters, it aligns the phoneme timing of the input speech with the rhythm of the target person's mouth movements, achieving precise audio-visual synchronization. This multimodal alignment synchronization mechanism effectively avoids the phenomenon of lip movements and voices being out of sync in voice-driven digital human creation. Furthermore, a dynamic semantic feature control mechanism is introduced during expression parameter mapping: the value range of some expression parameters is dynamically adjusted based on semantic stress and emotional intensity in the speech content. For example, when a sentence is detected to contain an interrogative tone, expression parameters such as raised eyebrows are appropriately increased; when the emotional label is excitement, the weight of the smile level parameter is increased. Through this semantic / emotional linkage control, the generated expressions not only match the speech timing but also semantically correspond to the spoken content.

[0037] The image fusion module synthesizes continuous video frames based on the static face model provided by the video modeling module and the expression parameter sequence output by the speech feature extraction module, realizing the dynamic changes of digital facial expressions with speech signals. The structure is as follows: Figure 4 As shown. A mask fusion and consistency verification mechanism based on 3D semantic information is preferred. Specifically, a 3D semantic mask for key areas such as the mouth and eye area is obtained using a pre-constructed 3D face model of the target person. When embedding the local facial expression image of the target face generated by speech into the reference face image, the fusion region is defined by this 3D semantic mask, and the consistency of the fusion result is judged in conjunction with the 3D semantic information. This method ensures that the inserted expression image highly matches the original face in terms of edge transition, scale, and orientation, resulting in a natural and coordinated overall facial expression after fusion. Specifically, this module first initializes the reference image of the digital human, for example, using a neutral expression face image output by the video modeling module as the initial base image, or using a neutral face model to render a frontal reference face. Subsequently, for each moment of expression parameters given by the speech feature extraction module, the image fusion module calculates the corresponding facial deformation and applies it to the reference image to generate a new frame. Specifically, in a preferred embodiment, the image of the target mouth region is reconstructed by linearly combining the expression base obtained in the appearance modeling stage: Let... The mean image under neutral expression. For the first The expression coefficient vector of the frame then reconstructs the mouth image. Represented as: (1); In the formula, For the first Image of the mouth region in the frame; The mean image under neutral facial expressions; For the first The first frame One expression parameter; For the first in the expression base space 3 orthogonal basis vectors; The number of expression basis vectors; denoted as the index of the orthogonal basis vector in the expression basis space.

[0038] Each of them and The dimensions are the same, representing an orthogonal basis image in the mouth image space. The mouth region image of the current frame can be quickly generated through the above linear combination. Next, the generated mouth region is synthesized back into the entire face image: based on the facial key points obtained in the appearance modeling stage, the position and contour of the mouth region in the reference face base image are determined, the reconstructed mouth image replaces the corresponding part in the reference face base image, and then they are stitched and fused. For example, this can be achieved using... Image fusion techniques, such as blending or seamless cloning, ensure natural edge transitions and consistent colors. For expressions involving changes in facial geometry, the module also applies appropriate non-rigid deformation to the baseline face image: using keypoint displacement fields or deformation fields calculated from a 3D face model, subtle distortion transformations are applied to the entire face, keeping features like eyebrows and eyes consistent with the baseline image, while the mouth region opens or closes as needed. For example, a local affine transformation can be constructed based on corresponding keypoint pairs, or triangulation interpolation can be used to deform the polygonal mouth region of a neutral face into the target mouth shape of the current frame.

[0039] If a 3D face model is used to represent the expression basis, then for each frame of expression parameters, a new set of 3D face vertex positions is first calculated based on the parameters: (2); In the formula, For the first The set of 3D face vertex positions in a frame; The set of 3D face vertex coordinates when a neutral expression is present; For the first The vertex displacement field corresponding to each expression base. Different base types correspond to 2D and 3D models, respectively. This allows us to obtain the geometric deformation of the face in the current frame. Then, by calculating an appropriate 3D transformation, the face shape is projected onto a 2D screen coordinate system to obtain the new positions of the facial key points for that expression. The 3D rotation transformation is calculated using the Rodriguez formula: if... Let be the unit vector of the rotation axis. Let be the rotation angle, then the rotation matrix is... Represented as: (3); In the formula, It is the identity matrix; For the reason The resulting antisymmetric matrix is ​​used to simulate joint movements of a human face, such as nodding, turning the head, or opening the mouth, by setting appropriate rotation axes and angles. The new keypoint coordinates after projection are used to guide the mapping and interpolation of the 2D texture, deforming the texture of the reference face onto the expression shape of the current frame.

[0040] In either implementation path, the image fusion module ultimately obtains a sequence of frame images. Each frame All of them are target individuals in the first Synthesized facial images at each time step. Playing these frames continuously creates a digital human video animation that is strictly synchronized with the input audio. The module utilizes pre-computed facial basis parameters, linear combination, and efficient deformation processing to quickly generate each frame, ensuring real-time generation.

[0041] Furthermore, depth generation based on differentiable rendering can directly map expression parameters to generate images of the entire face or local expression regions. This differentiable rendering employs a lightweight generative adversarial network or U-shaped network structure. Using a neutral face of the target person as the initial condition, the learned rendering network performs image-domain transformation on the input expression parameters while preserving the person's identity features, outputting a face image with the corresponding expression. In specific implementation, during the training phase, an expression rendering sub-network is trained for the target person. This network uses a baseline neutral face image as reference input and expression parameters as control signals, outputting a rendered face image with expression. This sub-network can adopt an Encoder-Decoder structure, and the expression parameters are fused in the Decoder to deform or modulate the feature layer, thereby generating expression changes corresponding to the input parameters. Due to the end-to-end learning, the rendering network can synthesize realistic images including complex variations such as frown lines and lip details. To ensure the rendering results are both realistic and consistent with the character's identity and style, several constraints are imposed on the rendering network during training: content reconstruction loss is used to ensure that the synthesized expression is similar to the real expression image in pixels or features; style transfer loss is used to ensure that the overall color and texture style of the synthesized image is consistent with the baseline face; and adversarial loss is used to improve the realism of the generated image, etc. Compared with the linear basis method, differentiable rendering layers can depict richer facial texture details and allow end-to-end optimization training. When using differentiable rendering for generation, this invention adds a style transfer constraint term to the loss function. By calculating the difference in style features between the generated image and the original image of the target character, it encourages the rendering network to generate image effects consistent with the overall style of the target character.

[0042] The two generation methods can be used in combination: first, an initial expression synthesis is quickly provided using a linear basis, and then a differentiable rendering network is used to refine and improve realism. The generated facial expression image can be a complete face or just a local expression region, which will be fused with a reference face as needed.

[0043] In summary, this application achieves an efficient audio-driven digital human generation scheme through clear module division and rigorous algorithm design. The input and output data structures of each module are clearly defined, and the processing flow is consistent throughout. It can complete the conversion from speech to facial animation with low computational complexity while ensuring lip-sync accuracy and image realism. This overcomes the dependence on expensive hardware and lengthy training processes, and offers advantages such as high usability, low resource consumption, and strong real-time performance. It can fully meet the application needs of real-time interactive scenarios such as virtual live streaming, intelligent customer service, and online education.

[0044] The modules described above work collaboratively in a pipeline manner: after the video modeling module pre-establishes the static model and facial expression base of the person, the speech feature extraction module maps the input speech into facial expression parameters in real time during runtime and passes them frame by frame to the facial expression generation module; the image fusion module continuously updates the facial image accordingly and outputs a video stream. The data interfaces between the modules are clearly defined: the baseline image and facial expression base parameters provided by the video modeling module are read and used by the latter two modules, and the facial expression parameters output by the speech feature extraction module serve as the input control for image synthesis, ensuring seamless integration between speech-driven and image generation. Through close cooperation between the modules, the system as a whole achieves end-to-end conversion from audio to image with low latency, and the generated digital human facial expressions can accurately follow changes in the audio content. The three modules together ensure the real-time performance and synchronization of the digital human synthesis.

[0045] 2. System Execution Flow The execution process of this method includes the following steps: Step 1: Obtain character footage of the target person as input, such as a 5-10 second video of the person speaking or several photos with different mouth shapes and expressions. Input the footage into the video modeling module to extract facial features and construct a static character model and expression base library. The resulting neutral face images and expression base parameters will be saved in the system for use in subsequent steps.

[0046] Step 2: Receive the speech audio signal to be synthesized as the driving content. After preprocessing the audio, divide it into consecutive short frames and extract the acoustic feature parameters of each frame to form an audio feature sequence arranged in chronological order.

[0047] Step 3: Input the above audio feature sequence into the neural network model of the speech feature extraction module. The model processes the audio sequence step by step and outputs the corresponding expression parameter sequence. Whenever the accumulated audio reaches a duration equivalent to one frame of video, the model generates the expression coefficient for that current moment. In a simple implementation, facial expression parameters can be directly regressed from the audio features of each frame. In the preferred recurrent network implementation, the model integrates audio features from several preceding and following frames to determine the facial expression parameters for the current frame, ensuring smooth and accurate lip movements. The final sequence of facial expression parameters is aligned with the audio timeline and then proceeds to the image synthesis preparation stage.

[0048] Step 4: Initialize the image fusion module, loading the baseline face image provided by the video modeling module as the background image. Then, for each set of expression parameters output by the speech feature extraction module... Perform the following sub-steps to generate the corresponding video frame images: a. Perform a linear combination of the facial expression bases according to the current facial expression parameters, and calculate the synthesized mouth region result for the current frame. ; b. The corresponding parts of the base face image are embedded. If a 2D image is used, the pixels in the mouth area of ​​the base image are replaced and the blending edges are smoothed; if a 3D mesh is used, the complete face image is rendered based on the deformed mesh. Optionally, the head pose can be further adjusted as needed, and the entire face image can be translated or rotated accordingly to obtain a coherent visual effect.

[0049] c. Output the currently synthesized face frame image The video frame sequence is buffered, and the next frame is processed. This process is repeated until the audio stream ends.

[0050] Step 5: After processing all audio frames, the image fusion module encapsulates the generated sequential frame images chronologically to form the output digital human video. The output video frame rate is typically comparable to that of the reference person's video, and the resolution is consistent with the baseline face image. In the final generated video, the digital human's lip movements and facial expressions change naturally with the sound, achieving the desired lip-sync effect.

[0051] Through the above steps, this method achieves a complete automated process from input audio to output digital human video. Among them, appearance modeling is an offline or preprocessing step, which can be reused after completion; the audio driving and image synthesis steps are executed in real time during runtime, processing audio data and image frame data in a parallel pipeline manner to ensure that the overall system can generate digital human animation with low latency and high efficiency.

[0052] As shown in Table 1, compared with traditional schemes that do not employ PCA compression, semantic fusion, and style constraints, this method shows significant advantages in terms of model lightweighting, generation stability, facial expression accuracy, and audio-visual synchronization.

[0053] Table 1. Comparison of Technical Performance Indicators

[0054] To objectively evaluate the performance of this method, the indicators listed in Table 1 are defined as follows: Lip misalignment rate: refers to the percentage of frames in a synthesized video where the average pixel distance between the key points of the digital human's lip contour and the corresponding key points in the real video exceeds 2 pixels, and is used to quantify the geometric accuracy of lip movement synchronization.

[0055] Geometric misalignment rate: refers to the proportion of vertices whose average Euclidean distance between the synthetic facial mesh vertices and the real mesh vertices in the main expression areas exceeds 3mm after registration of the 3D face model, reflecting the accuracy of 3D deformation reconstruction.

[0056] Inter-frame FID reduction rate: Using the traditional method driven by 3D BlendShape as the baseline, the relative reduction rate of inter-frame FID values ​​between this method and the baseline on the same test set is calculated to measure the degree of improvement in inter-frame style consistency and image realism of the generated video.

[0057] 3. Input / output data structure definition To ensure clear module interfaces and consistent system implementation, the data structures of the main inputs and outputs in this application are professionally defined: The audio input is a single-channel digital speech signal, such as a PCM-encoded .wav file or audio stream. A sampling rate of 16kHz and quantization precision of 16 bits are preferred to reduce data volume while ensuring speech clarity. During processing, the audio data is organized into a frame sequence, with each frame corresponding to 20–40 ms of audio samples. A typical frame length is, for example, 512 sampling points. Frames may not overlap or may have some overlap. After feature extraction, each audio frame is represented as a feature vector containing several real-dimensional numbers. These feature vectors are arranged in chronological order to form an audio feature sequence data structure for use by the driving model.

[0058] The input required for human appearance modeling includes at least one video or photo set containing the target person's face. Typically, RGB color image frames are used, with the resolution chosen based on application requirements, such as ensuring the face region size is no less than 128×128 pixels to retain sufficient detail. The input video can be represented as a sequence of image frames or a container format. This data structure provides appearance samples of the target person under different mouth expressions. Facial geometric features obtained through facial landmark extraction can be represented as a two-dimensional coordinate list structure; if further depth / 3D information is included, it can be expanded into a three-dimensional coordinate list. The texture features of the person can be derived from a baseline facial image. The output data of the video modeling module includes: a neutral face model of the target person and expression basis parameters. These data structures will be used as the basis for generating a digital human in subsequent processing.

[0059] Facial expression parameters are key intermediate data output by the speech feature extraction module, and their structure is a list of vectors arranged in a time sequence. Each facial expression parameter vector corresponds to the facial expression state of one frame of video and consists of several consecutive values, each value representing a weight coefficient of a basic facial expression component. For example, each vector... It contains 6 floating-point numbers, representing the weights of the mouth shape in the current frame on the 6 principal variation bases. The data structure of this parameter sequence can be represented as a data structure of size [size missing]. A two-dimensional array, where For video frame rate, This refers to the dimension of facial expression parameters. If a 3D facial parametric model is used, the facial expression parameters can also include rigid motion components and non-rigid motion components, increasing the total dimension accordingly. Facial expression parameter data is linked one-to-one with audio features and video frames via time indexing, serving as the core control data for inter-module transfer and driving synthesis.

[0060] The final output digital human video consists of a series of synthesized face image frames, synchronized with the input audio over time. Each frame uses the same format and resolution as the input reference image. Frames are organized into a video sequence at a fixed frame rate and stored as a standard video file. The output video data structure includes: a list of frame image pixel matrices. Frame count The output includes frame rate information and corresponding audio track data. When focusing solely on visual output, only the image sequence is output. In practical applications, the output is either presented frame-by-frame directly on the display device or encoded and compressed for storage and transmission. Regardless of the format, the output data structure ensures seamless alignment between the digital human's facial movements and the input audio content, providing the final effect of real-time digital human synthesis.

[0061] 4. Cross-platform applicability and exception handling This method and system possess excellent cross-platform applicability, enabling deployment and operation across diverse hardware environments. In mobile environments, the lightweight and efficient algorithm model allows for real-time execution of modules using the mobile device's CPU or GPU. Appearance modeling and audio-driven models are pre-computed on a server or offline, and the resulting model parameters are packaged into the mobile application, allowing the mobile device to perform only real-time inference computation. For deep neural network models, model compression and quantization reduce size, and a mobile neural network acceleration framework accelerates inference, enabling smooth digital human synthesis on ordinary mobile phone chips. In web environments, the system executes key algorithm modules in the browser using technologies such as WebAssembly or WebGL. For example, audio feature extraction and partial facial expression prediction models are deployed as JavaScript / WebAssembly scripts running on the front end, while the facial expression generation module utilizes WebGL shaders or GPU acceleration libraries for image processing. For computationally demanding steps, a front-end / back-end collaborative approach is adopted: the browser acquires user audio and sends it to the server for processing, then returns the generated face frame stream to the webpage for real-time playback. On cloud servers, with ample computing and storage resources, this solution supports higher-resolution video synthesis and more concurrent user requests. Each module is deployed in a distributed manner in the backend cluster. It processes user-requested audio by calling pre-loaded deep learning model instances and quickly generates corresponding video frame sequences. With the help of GPU acceleration in the cloud, this system can further reduce latency while maintaining high fidelity, and provide real-time digital human-driven services to a large number of users simultaneously through load balancing.

[0062] To improve the system's robustness under various input conditions, corresponding abnormal scene handling strategies were designed. When audio is missing or interrupted, the system automatically keeps the digital human's face at the neutral expression of the previous frame or remains still, only resuming lip-syncing upon receiving new audio input, thus avoiding meaningless lip-syncing changes when there is no audio. When audio noise is excessive or unexpected, the audio feature extraction module combines silence detection and noise filtering algorithms. When a long period of silence or invalid audio segment is detected, the corresponding frame's expression parameters are smoothly transitioned or maintained at neutral to ensure the stability of the synthesized result. When facial key point extraction fails, the system activates a backup plan: for example, using an average face model as a substitute, or prompting the user to provide clearer input material. When abnormal values ​​of expression parameters occur during operation, the expression generation module performs constraint clipping and interpolation smoothing on the parameters to prevent the generation of distorted facial images. For streaming scenarios such as real-time camera input, if frame loss or delay occurs, the system ensures synchronization between audio and video processing through buffers and interpolation strategies, so that even the loss of individual frames will not significantly affect the continuity of the digital human's lip-syncing. Through the above measures, this system can maintain stable operation under various abnormal conditions and provide reliable digital human synthesis output.

[0063] 5. Application Scenarios This system can be widely applied in various practical scenarios, significantly lowering the barrier to entry for digital human video production. For example, in the field of virtual live streaming, it can be used to create virtual anchors for real-time broadcasting: simply provide a short video sample of the anchor and real-time audio, and the system can generate virtual anchor lip movements and expressions synchronized with the audio, enabling online live performances or news broadcasts and reducing the cost of real people appearing on camera. In digital human customer service scenarios, it can be used to build an intelligent customer service representative with a human-like appearance. Pre-record the appearance materials of the customer service representative and deploy the model; when the backend customer service AI generates a voice reply, a corresponding video response from the customer service representative is generated simultaneously, providing users with an immersive interactive experience. In online education, it can be used to create virtual digital teachers or training anchors. The teacher only needs to provide a recording and course audio once, and the system can automatically synthesize a video of the teacher lecturing, for large-scale online course content production. In addition, this invention is also applicable to other scenarios that require converting voice content into lifelike facial lip-synced video, such as film and television dubbing, game character dialogue, and social media virtual avatars. In these applications, this system can achieve high-quality digital human-driven effects at a low cost, meeting the needs of real-time interaction or batch content production.

[0064] Example 2: Building upon the aforementioned embodiments, the system is applied to a mobile virtual anchor scenario. Assume a user wants to generate their own digital anchor avatar on their mobile phone and have their lip movements driven by real-time voice. First, the user records a 10-second video using their phone's camera. In the video, the user reads a few sentences with a natural, neutral expression to provide lip movement material. This video is used as reference input to the video modeling module, which automatically detects faces in the video and extracts the coordinate sequence of 68 facial key points and a clear image of a neutral-expression face. Using a pre-trained offline PCA lip base model, the video modeling module establishes a static model of the user's face and a lip expression base library.

[0065] Next, the system loads a pre-deployed audio-driven model and image fusion module onto the phone. The user inputs speech via the phone's microphone or selects text for TTS engine-synthesized speech. The speech feature extraction module extracts Mel-spectral features frame-by-frame from the real-time acquired speech signal and runs a lightweight LSTM network model on the phone's CPU to progressively predict the output expression parameter sequence. With each speech input, the system continuously generates mouth expression parameters for the next frame. The image fusion module uses a pre-acquired neutral face image of the user as a base map and performs mouth region generation and fusion for each frame on the phone's GPU: it generates mouth image fragments through linear combination based on the current expression parameters, fits them onto the user's neutral face base map, and performs smooth transition processing on the edges. When the user speaks quickly, the system automatically predicts expression parameters slightly in advance to compensate for processing delays, ensuring that the generated frames are strictly aligned with the audio signal.

[0066] Ultimately, users see their digital avatar on their phone screen opening and closing its mouth and displaying corresponding facial expressions in sync with the audio content, achieving a real-time virtual anchor effect. Throughout the process, the mobile device's processing is smooth with no noticeable lag. Actual testing on a typical smartphone achieved a composite frame rate of approximately 25 frames per second, validating the system's real-time applicability on mobile devices.

[0067] Example 3: Building upon the aforementioned embodiments, this system is deployed on a cloud server to construct a digital human customer service interaction system. Assume a platform wishes to use a digital human to act as online customer service. First, materials for the customer service digital human need to be prepared: select a friendly-looking employee and record a short video of the employee reading a customer service greeting as a reference for the persona. This video is then input into the video modeling module to extract static model data of the employee's face. This model data is stored on the server side as the basic image of the digital human customer service representative. Subsequently, a pre-trained voice-driven model and image synthesis model are loaded onto the server backend. When a website user initiates a dialogue inquiry, the text-to-speech module of the customer service dialogue system synthesizes speech based on the text content to be replied to and sends the audio to the digital human synthesis server.

[0068] Upon receiving audio, the system's speech feature extraction module immediately extracts features and predicts facial expression parameters, obtaining a complete sequence of expression parameters. Next, the expression generation module uses a pre-stored neutral customer service face image as a base map to generate frame-by-frame facial images of the customer service representative speaking: for each expression parameter in the sequence, a corresponding mouth shape image is generated and superimposed onto the neutral face. If the customer service representative's response is lengthy, the system simultaneously receives the audio stream and outputs image frame streams in real time, allowing the digital human's mouth shape to change continuously. All generated frames are packaged into a video stream at a frame rate of 25 frames per second in the cloud and pushed to the user's browser frontend via WebSocket, playing synchronously with the synthesized speech to present a virtual customer service representative whose lip movements and voices are perfectly matched. Users can see the digital human customer service representative smiling and accurately lip-syncing to the response in their browser, with an effect close to that of a real person. Even with slight network transmission delays, the system's efficient generation process ensures a response time within the hundreds of milliseconds, guaranteeing a good user experience. This demonstrates the feasibility of the system in a cloud service model and its ability to support large-scale real-time interactive digital human applications.

[0069] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media containing computer-usable program code.

[0070] The present invention can provide computer program instructions to a management platform of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to produce a machine, such that the instructions executed by the management platform of the computer or other programmable data processing equipment produce means for implementing the system.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that perform the functions of the system.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions of the system.

[0073] Example 4: Building upon the aforementioned embodiments, a principal component analysis-based expression reconstruction model is employed when constructing the low-dimensional expression base space of the personalized digital human. First, high-dimensional feature data of the target person's face under multiple typical expression states are extracted from a single video segment, forming a training sample set. Principal component analysis is then performed on this dataset to obtain feature vectors and their corresponding feature values ​​arranged in descending order of variance contribution.

[0074] To overcome the limitations of traditional PCA, which has fixed dimensions and cannot optimize compression efficiency for individual video data, this embodiment proposes an adaptive principal component selection mechanism. The core of this mechanism lies in defining a dynamic gating strategy based on reconstruction accuracy requirements. Specifically, a threshold condition reflecting the fidelity requirements of facial expression reconstruction is set. This condition can be a cumulative variance percentage threshold or an upper limit for acceptable reconstruction error.

[0075] Let the original facial expression feature dimension be . The eigenvalue sequence after PCA decomposition is .

[0076] forward The percentage of cumulative variance explained by each principal component is: for;

[0077] Reconstruction error It is proportional to the sum of the eigenvalues ​​corresponding to the discarded principal components, that is... .

[0078] The adaptive selection process uses a gating function. This is achieved by the function continuously determining whether the current cumulative accuracy meets the preset requirements. The decision logic of the gating function can be formally expressed as: when... When the preset threshold is reached for the first time, it is determined that the previous value should be retained. One principal component is sufficient. Equivalently, this process can also be achieved by ensuring the reconstruction error is minimized. This is achieved by keeping the level below the allowed limit.

[0079] Through the aforementioned mechanism, the system can automatically determine the required number of principal components based on the unique facial expression patterns and richness of the target person's video. This method achieves maximum compression of model parameters while strictly ensuring the visual accuracy of facial expression reconstruction, thereby significantly improving the training and inference efficiency of subsequent facial expression generation models and reducing storage overhead.

[0080] Example 5: Building upon the aforementioned embodiments, to ensure visual style consistency between the output image and the original video of the target person during digital human facial expression generation, and to avoid flickering or drifting in tone and texture between frames, this embodiment introduces a self-supervised loss mechanism based on multi-scale style feature comparison. The core of this mechanism lies in utilizing a pre-trained deep convolutional neural network to extract style representations of the image from different semantic levels, and through an innovative weighting strategy, making the model training more focused on key texture regions that significantly affect visual fidelity.

[0081] Specifically, firstly, the original reference image of the target person and the local expression image generated by the model are input into the pre-trained network, and the network is set to... The feature mapping of the layer is ,in, It is the set of real numbers. For the number of channels, This represents the number of spatial locations. The Gram matrix of the features at this layer is calculated. The elements used to characterize its texture style distribution are:

[0082] In the formula, Gram matrix The Middle line, number Column elements; For the first In layer feature mapping, the first The first channel in the Activation value at a spatial location; For the first In layer feature mapping, the first The first channel in the Activation value at a spatial location.

[0083] The Gram matrix captures the correlation between different feature channels, effectively expressing the texture, color, and other style information of an image, and is insensitive to spatial location.

[0084] To achieve finer control over style consistency, this embodiment proposes an adaptive weighting method based on texture complexity. Not all feature layers or facial regions exhibit the same sensitivity to style inconsistencies. Therefore, a texture complexity index is defined for each feature layer participating in the loss calculation. This metric quantifies the richness and intensity of the texture patterns represented by the layer. An effective definition is the normalized energy of the Gram matrix of the reference image:

[0085] In the formula, For reference image in the first Gram matrix elements of the layer.

[0086] Subsequently, appropriate loss weights are assigned to each layer based on its complexity. Ensure that layers with higher complexity account for a larger proportion of the total loss:

[0087] Ultimately, multi-scale weighted style consistency loss Defined as a weighted sum of style differences across all layers:

[0088] In the formula, To generate an image In the The Gram matrix corresponding to the layer feature map; For reference image In the The Gram matrix corresponding to the layer feature map; Let f be the Frobenius norm of the matrix.

[0089] This weighted strategy allows the model to more selectively match the style of feature layers containing rich details during the optimization process, rather than treating all layers equally. Experiments show that after introducing this weighted multi-scale style loss, the inter-frame style abruptness in the generated digital human video is significantly reduced, and it maintains a high degree of consistency with the original person in terms of skin color, lighting, and texture details, effectively improving visual coherence and realism.

[0090] Without style loss, some generated frames show visible differences in local texture compared to the original video; however, after using weighted style loss, these differences are significantly reduced, with the Gram matrix error decreasing from 0.15 to below 0.10, as shown in Table 2.

[0091] Table 2. Quantitative Comparison of Style Loss Effects Before and After

[0092] Example 6: Based on the aforementioned embodiments, in order to achieve efficient fusion of audio features and text semantic information in voice-driven digital human facial expression generation, and to ensure that the generated facial expressions are precisely aligned with the speech content in terms of timing and semantics, this embodiment introduces an facial expression generation network driven by a cross-modal attention mechanism.

[0093] This network employs an encoder-decoder architecture. The encoder processes the input speech signal, converting it into a temporal audio feature sequence. Simultaneously, the semantic and sentiment analysis module recognizes and analyzes the same speech, outputting the corresponding text semantic feature sequence. The decoder generates the... A cross-modal attention module was introduced in the key steps of frame expression parameterization.

[0094] The core of this module lies in establishing a dynamic, content-related information retrieval mechanism. Specifically, the current hidden state of the decoder is used as a query vector to inquire about which modalities and information should be focused on. The audio feature sequence is mapped into a key vector sequence and a value vector sequence through a linear transformation. Attention weights, i.e., the generation of the first... When the expression is displayed in the first frame The level of attention given to moment-in-time audio features is calculated using the following scaled dot product attention formula:

[0095] In the formula, Attention weights; For query vector; The key vector; The dimension of the key vector; To calculate the final value of the summation index variable; For summation index variables.

[0096] Subsequently, these weights are used to perform a weighted summation of the audio value vectors, resulting in a condensed audio context vector that is most relevant to the current generation step:

[0097] In the formula, For context vectors; It is a value vector.

[0098] The context vector This information is fused with current semantic features and other data, and used as input to the decoder to predict the final facial expression parameters.

[0099] Through the aforementioned mechanism, the model can autonomously listen to the entire speech sequence when generating each frame of facial expression, and dynamically extract the most relevant acoustic cues for the current moment. This effectively solves the problems of non-strict alignment between speech and facial expression sequence lengths, as well as the coarse information fusion caused by simple feature splicing. Experiments show that after introducing this cross-modal attention mechanism, the synchronization accuracy between digital human lip movements and speech is significantly improved, and facial expression changes better reflect the semantic emphasis and emotional tone in speech, generating more natural and expressive results.

[0100] Example 7: Based on the aforementioned embodiments, in order to improve the geometric accuracy and visual realism of the fusion of digital human facial expression images and reference faces, this embodiment proposes an optimization strategy that integrates dynamic deformation prediction and multi-resolution fusion.

[0101] In the process of using 3D semantic masks to guide local facial expression image fusion, traditional methods rely on predefined fixed deformation rules to adjust the mask shape, which is difficult to accurately adapt to complex and personalized facial expression geometric changes. This embodiment uses a trained deformation prediction network to replace such a fixed model. This network takes the facial expression parameters of the current frame as input and directly regresses a dense, pixel-level deformation field through an encoder-decoder based convolutional neural network.

[0102] Let the pixel coordinates within the face mask area be in the neutral state. The deformation field output by the deformation prediction network defines the displacement vector for each pixel. Then the target pixel position after adapting to the current expression. It is given by the following formula:

[0103] This formula achieves a nonlinear mapping from the original mask geometry to the target expression geometry. This is achieved by applying a coordinate transformation defined by this deformation field to the reference image. This enables precise geometric alignment between the mask region and the target mouth shape. To encourage a smooth and physically plausible deformation field, regularization constraints are introduced during network training, such as penalizing the spatial gradient norm of the displacement field.

[0104] In the formula, For smoothness constraint loss; The set of pixels for the facial mask area, i.e., the effective area defined by the deformation field; For gradient operators; Gradient vector The norm; Gradient vector The norm of .

[0105] After completing the geometric alignment based on the deformation field, the generated local facial expression image is pixel-wise blended with the corrected reference image. To further optimize the smoothness and detail consistency of the fusion boundary transition, this embodiment adopts a multi-resolution cascaded fusion strategy. This strategy first performs initial fusion at a lower spatial resolution, focusing on global color and structure matching; then, the initial fusion result is upsampled and refined at a higher resolution, focusing on repairing edge details and high-frequency textures. This coarse-to-fine processing effectively avoids seam artifacts or local blurring problems that may occur when fusion is performed at a single resolution, ultimately generating a seamless and realistic synthetic face image.

[0106] Example 8: Based on the aforementioned embodiments, several local video clips and publicly available internet videos were selected to test the complete system and compare it with some alternative solutions, including: 1. The complete method; 2. Instead of using a cross-modal attention mechanism, simply mapping audio features to facial expression parameters through a fully connected layer; 3. Without using style loss, removing the Gram matrix style constraint in the image enhancement module; 4. Instead of using a deformation prediction network, directly generating facial expression images using traditional PCA linear deformation; 5. Without using PCA compression, directly feeding high-dimensional facial expressions into the deformation network and decoder. Evaluation metrics covered image quality (PSNR / SSIM), audio-video synchronization accuracy, facial expression realism, and computational efficiency, as shown in Table 3.

[0107] Table 3. Comparison of Module Ablation Experimental Performance

[0108] In summary, each module plays an important role in the entire system. PCA provides a stable and efficient low-dimensional expression parameter space, the attention mechanism ensures the alignment of audio-visual timing and semantic information, the deformation network ensures the accuracy of geometric deformation, and style loss maintains the consistency of visual texture.

Claims

1. A method for generating personalized digital humans based on a single video, characterized in that, include: Acquire a video or several images of the target person and construct a digital facial model of the target person. In the model training process, introduce a self-supervised style consistency loss to ensure the stability and consistency of the generated style. Acquire the voice input of the target person and extract acoustic feature parameters; The voice input is subjected to semantic understanding and emotion recognition to obtain corresponding semantic information and emotion tags; Under multimodal alignment constraints, the acoustic feature parameters, semantic information, and emotion tags are mapped to corresponding facial expression parameter sequences; wherein a dynamic semantic feature control mechanism is introduced for the semantic stress and emotion intensity of the speech content to adjust the amplitude and details of the generated expressions; Based on the facial expression parameter sequence, a facial image of the target person is generated through the digital human facial model; the generation process includes a differentiable rendering layer, end-to-end expression synthesis of key facial regions, and style transfer constraints to ensure that the generated local expression image is consistent with the overall appearance style of the target person. The facial image of the target person is fused into a pre-stored reference facial image of the target person using a 3D semantic mask. The fusion and overlay of local expression areas with the reference face is completed at the pixel level, and a personalized digital human video synchronized with the voice input content is output.

2. The method according to claim 1, characterized in that: The calculation of style consistency loss adopts a multi-scale style feature comparison strategy. It measures style difference by acquiring multiple levels of features of the pre-trained convolutional neural network and calculating the style representation difference of the corresponding level features between the generated image and the original image of the target person. It assigns a weighted coefficient based on the texture complexity of the region to the style difference of different facial regions.

3. The method according to claim 1, characterized in that: The speech input is preprocessed and divided into consecutive short time frames, and Mel-spectrum coefficients or MFCC coefficients are extracted as acoustic features. These acoustic features are input into the audio-driven model of a deep neural network to regress and output a sequence of facial expression parameters. The audio-driven model includes a long short-term memory network layer to capture the temporal dependence of the speech signal, and further includes a speaker adaptation mechanism to fine-tune some parameters of the model under the condition of providing only a small amount of speech-expression correspondence data of the target speaker, thereby adjusting the expression parameter mapping to fit the speaker's pronunciation style and expression habits.

4. The method according to claim 3, characterized in that: The audio-driven model also includes an adjustable mapping unit for different speakers’ speech rates and intonation rhythms. The mapping unit dynamically adjusts the output rate and smoothness of the facial expression parameter sequence according to the phoneme duration, speech rate and intonation fluctuation of the speech signal. By introducing additional features or modules that reflect the rhythm of speech, we can ensure a rapid response and stable transition of mouth movements at fast speaking speeds, and maintain smooth and natural facial expression changes at slow speaking speeds.

5. The method according to claim 1, characterized in that: The process of mapping acoustic feature parameters, semantic information, and emotion tags into facial expression parameter sequences is implemented by an expression generation network driven by a cross-modal attention mechanism. The network assigns dynamic weights to multimodal information by setting the speech feature sequence as the query and the text semantic feature sequence as the key and value through an attention model.

6. The method according to claim 1, characterized in that: The process of generating a target human facial expression image using a digital human facial model includes using a differentiable rendering layer to perform expression synthesis rendering on local facial regions; the differentiable rendering layer receives the facial expression parameter sequence and combines it with the target human facial model to generate a corresponding target human expression image frame.

7. The method according to claim 1, characterized in that: The image fusion using 3D semantic masks specifically includes: Step 1: Based on the pre-constructed 3D face model of the target person, extract the 3D semantic mask of the mouth and eye area; Step 2: Dynamically adjust the geometry and spatial position of the mask according to the facial expression parameters of the current frame to adapt it to the opening and closing state of the target mouth. Step 3: The generated target face local expression image and the pre-stored reference face image are pixel-level blended within the fusion area defined by the three-dimensional semantic mask, and the orientation and scale alignment consistency between the fusion area and the reference face is verified by the spatial transformation matrix.

8. The method according to claim 7, characterized in that: When the 3D semantic mask is dynamically adjusted according to the expression parameters of the current frame, a trained deformation prediction network is used to replace the predefined fixed deformation model. The deformation prediction network is based on a convolutional neural network structure, which takes the expression parameters as input and outputs the corresponding mask deformation field. At the same time, the pixel-level mixing of the image fusion adopts a multi-resolution cascade fusion strategy, which fuses the local expression image and the reference face image step by step at different spatial resolutions.

9. A personalized digital human generation system based on a single video, characterized in that: The system executes the method according to any one of claims 1-8 during runtime: The system includes: The video modeling module is used to construct a digital facial model of the target person based on a video or several images of the target person, and introduces a self-supervised style consistency loss during the model training process to ensure that the generated style is stable and consistent. The speech feature extraction module is used to extract the acoustic feature parameters of the speech input of the target person; The semantic and sentiment analysis module is used to perform semantic understanding and sentiment recognition on the voice input to obtain corresponding semantic information and sentiment tags; The expression generation module is used to map the acoustic feature parameters, semantic information, and emotion tags to corresponding facial expression parameters under multimodal alignment constraints, and generate a face image of the target person based on the facial expression parameters. The image fusion module is used to fuse the facial image of the target person into a pre-stored baseline facial image of the target person using a 3D semantic mask, and output a personalized digital human video.

10. The system according to claim 9, characterized in that: It also includes an interpretable adjustment module for facial expression parameters, which constrains and refines the sequence of facial expression parameters; when the voice input is interrupted, the adjustment module maintains the current state of the expression parameters or smoothly transitions the expression parameters to a neutral value; when any expression parameter exceeds a preset reasonable range, the adjustment module adjusts it to within a threshold by means of pruning or interpolation correction; the module also provides several interpretable control indicators for real-time adjustment of expression intensity and style by manual or algorithmic means.

Images