Voice-driven digital human generation methods and devices

Abstract

The application discloses a voice-driven digital human generation method and device, and relates to the technical field of computer vision. The method selects original frame images and driving frame images from a video sequence; generates a ground state face model based on the appearance features of a face in the original frame images, and migrates the dynamic features of the face in the driving frame images to the ground state face model to obtain a target face image of a current frame; takes the target face image and audio features as inputs of a pre-trained time sequence denoising model, acquires image noise of a subsequent frame predicted by the time sequence denoising model according to the audio features and the dynamic features of the target face image; determines predicted dynamic features of the target face image of the subsequent frame according to the image noise; and generates a digital human image of the subsequent frame based on the predicted dynamic features and the ground state face model, thereby improving the naturalness of the digital human.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and device for voice-driven digital human generation. Background Technology

[0002] Voice-driven digital human generation technology aims to automatically generate virtual face videos that are synchronized with the input audio signal and possess corresponding lip movements, expressions, and facial micro-movements. Currently, voice-driven digital human generation methods mainly fall into two categories. The first category focuses on precise synchronization between audio and lip movements, using generative adversarial networks to map the input audio to the lip movements of the target video. The core is a pre-trained lip-sync discriminator, which improves synchronization accuracy through multi-frame temporal supervision. However, it only processes the lip region, resulting in weaker reproduction of overall facial details such as teeth and expressions. The second category relies on explicit 3D face model parameters, first driving the face model parameters through audio, and then generating images through a renderer. While this type of method offers greater controllability, its realism heavily depends on the detail and rendering quality of the 3D model, often failing to reproduce facial expression details, leading to stiff digital human images. Therefore, digital humans generated using these methods often lack naturalness.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a voice-driven digital human generation method and device, aiming to solve the technical problem of how to improve the naturalness of digital human generation.

[0005] To achieve the above objectives, this application proposes a voice-driven digital human generation method, which includes the following steps:

[0006] Select the original frame image and the driving frame image from the video sequence;

[0007] A base state face model is generated based on the appearance features of the face in the original frame image, and the dynamic features of the face in the driving frame image are transferred to the base state face model to obtain the target face image of the current frame.

[0008] The target face image and audio features are used as input to a pre-trained temporal denoising model to obtain the image noise of subsequent frames predicted by the temporal denoising model based on the audio features and the dynamic features of the target face image.

[0009] Based on the image noise, the predicted dynamic features of the target face image in subsequent frames are determined;

[0010] Based on the predicted dynamic features and the ground-state face model, digital human images of subsequent frames are generated.

[0011] In one embodiment, the dynamic features include facial expression features and pose features, and the step of obtaining the image noise of subsequent frames predicted by the temporal denoising model based on the audio features and the dynamic features of the target face image includes:

[0012] The facial expression features and pose features of the target face image are concatenated to form a reference latent vector;

[0013] The current frame is expanded to a first preset number of frames based on the reference potential vector;

[0014] The first preset number of frames and the second preset number of noise frames are concatenated as the input latent vector, wherein the noise frames are obtained by adding random noise to the current frame;

[0015] The input latent vector is input into the temporal diffusion Transformer network of the temporal denoising model, and the predicted noise of the second preset number of noise frames output by the temporal diffusion Transformer network is obtained as the image noise.

[0016] In one embodiment, the step of expanding the current frame to a first preset number of frames based on the reference latent vector includes:

[0017] The facial expression features and pose features of the face in the driving frame image are concatenated to form a prediction latent vector;

[0018] An intermediate latent vector is generated between the reference latent vector and the predicted latent vector using a temporal interpolation algorithm, and an intermediate frame is constructed based on the intermediate latent vector.

[0019] The current frame and the intermediate frames are arranged in chronological order to obtain the first preset number of frames.

[0020] In one embodiment, the step of expanding the current frame to a first preset number of frames based on the reference latent vector further includes:

[0021] Background elements other than the face region in the target face image are removed using a pre-trained matting algorithm.

[0022] The current frame, after removing background elements, is extended to the first preset number of frames based on the reference latent vector.

[0023] In one embodiment, the step of determining the predicted dynamic features of the target face image in subsequent frames based on the image noise includes:

[0024] Subtract the image noise from the random noise in the second preset number of noise frames to obtain the denoised image;

[0025] Based on the facial expression and pose features of the denoised image, the predicted dynamic features of the target face image in the subsequent frames are determined.

[0026] In one embodiment, the step of generating a base-state face model based on the facial appearance features in the original frame image includes:

[0027] The face key points detected by the face detection algorithm in the original frame image are obtained, and the face region is obtained by outward cropping based on the face key points;

[0028] Detect the three-dimensional key points of the face in the face region, and map the three-dimensional key points onto the face region through a camera model to obtain a ground-state face image with two-dimensional projection key points;

[0029] The pose features, appearance features, and global appearance descriptors corresponding to the appearance features are obtained from the ground state face image by a pre-trained multi-scale convolutional neural network based on the two-dimensional projection key points.

[0030] Based on the pose features, the appearance features of the face are inversely transformed to obtain the ground state face model.

[0031] In one embodiment, after the step of performing an inverse transformation on the facial appearance features based on the pose features,

[0032] After aligning the ground-state face image with the standard face model, the facial feature mask in the ground-state face image is obtained through a preset face segmentation model;

[0033] The facial features are extracted from the ground-state face image based on the facial feature mask, and the facial expression features in the ground-state face image are determined based on the two-dimensional projection key points in the facial feature image.

[0034] Based on the facial expression features, the appearance features of the face are inversely transformed to obtain the ground state face model.

[0035] In one embodiment, the step of transferring the dynamic features of the face in the driving frame image to the base state face model to obtain the target face image of the current frame includes:

[0036] The multi-scale convolutional neural network obtains driving expression features and driving posture features from the driving frame image based on the two-dimensional projection key points of the face in the driving frame image.

[0037] The driving expression features are transferred to the ground state face model based on the global appearance descriptor, and the pose parameters of the ground state face model are determined according to the driving pose features to obtain the target face image.

[0038] In one embodiment, after the step of determining the predicted dynamic features of the target face image in subsequent frames based on the image noise, the method further includes:

[0039] In response to the received text information, the text information is converted into a semantic feature vector;

[0040] The semantic feature vector is used as the frequency domain modulation parameter of the Fourier convolutional layer in the pre-trained image restoration model. The background elements of the subsequent frames are regenerated through the image restoration model. The frequency domain modulation parameter includes a spectral bias vector and a channel weight vector.

[0041] Furthermore, to achieve the above objectives, this application also proposes a voice-driven digital human generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the voice-driven digital human generation method as described above.

[0042] This application provides a voice-driven digital human generation method, which involves selecting original frame images and driving frame images from a video sequence; generating a base-state face model based on the facial appearance features in the original frame images, and transferring the dynamic features of the face in the driving frame images to the base-state face model to obtain the target face image of the current frame; using the target face image and audio features as input to a pre-trained temporal denoising model to obtain the image noise of subsequent frames predicted by the temporal denoising model based on the audio features and the dynamic features of the target face image; determining the predicted dynamic features of the target face image of subsequent frames based on the image noise; and generating digital human images of subsequent frames based on the predicted dynamic features and the base-state face model.

[0043] The aforementioned method addresses the fundamental issue of naturalness in identity consistency by constructing a base-state face model, avoiding facial distortion or identity drift during the generation process. Subsequently, dynamic feature transfer accurately maps the expressions and poses of the driving frames onto this baseline model, ensuring a natural fusion of facial dynamics and identity appearance within a specific frame. Finally, a temporal denoising model is introduced. By using audio features and the current frame's dynamics as joint conditions, it infers and generates a coherent dynamic sequence of future frames in the latent space, thereby guaranteeing a high degree of synchronization between facial movements and speech rhythm, as well as temporal smoothness of expression changes, thus improving the naturalness of the digital human's performance. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating an embodiment of the voice-driven digital human generation method of this application;

[0047] Figure 2 A flowchart illustrating Embodiment 2 of the voice-driven digital human generation method of this application;

[0048] Figure 3 A flowchart illustrating Embodiment 3 of the voice-driven digital human generation method of this application;

[0049] Figure 4 A flowchart illustrating Embodiment 4 of the voice-driven digital human generation method of this application;

[0050] Figure 5 A flowchart illustrating Embodiment Six of the Voice-Driven Digital Human Generation Method of this Application;

[0051] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the voice-driven digital human generation method in the embodiments of this application.

[0052] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0054] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. It should be noted that all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located, and with authorization from the owner of the corresponding device.

[0055] Voice-driven digital human generation technology aims to automatically generate virtual face videos that are synchronized with the input audio signal and possess corresponding lip movements, expressions, and facial micro-movements. Currently, voice-driven digital human generation methods mainly fall into two categories. The first category focuses on precise synchronization between audio and lip movements, using generative adversarial networks to map the input audio to the lip movements of the target video. The core is a pre-trained lip-sync discriminator, which improves synchronization accuracy through multi-frame temporal supervision. However, it only processes the lip region, resulting in weaker reproduction of overall facial details such as teeth and expressions. The second category relies on explicit 3D face model parameters, first driving the face model parameters through audio, and then generating images through a renderer. While this type of method offers greater controllability, its realism heavily depends on the detail and rendering quality of the 3D model, often failing to reproduce facial expression details, leading to stiff digital human images. Therefore, digital humans generated using these methods often lack naturalness.

[0056] In view of the above problems, this application proposes a voice-driven digital human generation method, which selects original frame images and driving frame images from a video sequence; generates a base state face model based on the facial appearance features in the original frame images, and transfers the dynamic features of the face in the driving frame images to the base state face model to obtain the target face image of the current frame; uses the target face image and audio features as input to a pre-trained temporal denoising model to obtain the image noise of subsequent frames predicted by the temporal denoising model based on the audio features and the dynamic features of the target face image; determines the predicted dynamic features of the target face image of subsequent frames based on the image noise; and generates digital human images of subsequent frames based on the predicted dynamic features and the base state face model.

[0057] The aforementioned method addresses the fundamental issue of naturalness in identity consistency by constructing a base-state face model, avoiding facial distortion or identity drift during the generation process. Subsequently, dynamic feature transfer accurately maps the expressions and poses of the driving frames onto this baseline model, ensuring a natural fusion of facial dynamics and identity appearance within a specific frame. Finally, a temporal denoising model is introduced. By using audio features and the current frame's dynamics as joint conditions, it infers and generates a coherent dynamic sequence of future frames in the latent space, thereby guaranteeing a high degree of synchronization between facial movements and speech rhythm, as well as temporal smoothness of expression changes, thus improving the naturalness of the digital human's performance.

[0058] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or digital human generation system capable of performing the above functions. The following description uses a digital human generation system as an example to illustrate this embodiment and the subsequent embodiments.

[0059] Based on this, the first embodiment of this application provides a voice-driven digital human generation method, referring to... Figure 1 In this embodiment, the voice-driven digital human generation method includes steps S10 to S50:

[0060] Step S10: Select the original frame image and the driving frame image from the video sequence.

[0061] It should be noted that the video sequence is a segment of video data containing the target person, i.e., the digital human to be generated. One or more frames are selected from the video sequence that have neutral facial expressions, clear features, no occlusion, and good lighting as raw frame images. These raw frame images serve as appearance references, providing a source of identity features for digital human generation, such as static attributes like skin texture, hairstyle, face shape, and makeup. One or more frames are selected from different time points within the same video sequence as driving frame images. These driving frame images serve as dynamic references, providing the digital human with the facial dynamic features to imitate, such as specific lip movements, eyebrow movements, and facial muscle movements.

[0062] Optionally, before selecting the original frame image, the digital human generation system processes each frame of the video sequence using face detection algorithms such as MTCNN (Multi-task Cascaded Convolutional Neural Networks) and YOLO to locate the initial face region and perform facial landmark detection on the initial face region. Then, the digital human generation system can calculate a comprehensive quality score for the face image of each frame through multiple evaluation modules, and select the frame with the highest comprehensive quality score as the original frame image.

[0063] For example, a trained facial expression classification neural network is used to analyze each frame of a video sequence, calculating the probability that the frame belongs to a neutral expression; the higher the probability, the higher the overall quality score. Image sharpness is evaluated by calculating the gradient magnitude of each frame, such as the Laplacian operator or frequency domain features; higher sharpness corresponds to a larger gradient variance and a higher overall quality score. A facial landmark model is used to detect whether other objects, such as hands, hair, glasses, or microphones, obscure key facial areas; unobstructed images are assigned an overall quality score. Similarly, by analyzing features such as the brightness distribution of the facial region and the facial rotation angle, images with uniform lighting, moderate brightness, and near-frontal faces are assigned higher overall quality scores.

[0064] Optionally, the user can select the original frame image and the driving frame image, and in response to the user's selection, the original frame image and the driving frame image are selected from the video sequence.

[0065] Step S20: Generate a base state face model based on the appearance features of the face in the original frame image, and transfer the dynamic features of the face in the driving frame image to the base state face model to obtain the target face image of the current frame.

[0066] It should be noted that the ground-state face model is a numerical feature representation extracted and encoded from the original frame image through a deep neural network. It strips away instantaneous dynamic information such as facial expressions and lip movements, recording the static identity and appearance attributes of the target person, such as face shape, facial structure, skin texture, skin color, and hairstyle. In the digital human generation process, the ground-state face model acts as a stable and reusable identity template. Its core function is to ensure that the digital human's identity and appearance features remain consistent throughout the entire generated video sequence, providing a benchmark for subsequent dynamic transfer. The target face image of the current frame is a specific, pixel-level image generated by transferring and fusing the dynamic features extracted from the driving frame image to the ground-state face model. It is a synthetic result at a specific point in time, possessing both the target person's appearance and the driving source's dynamic features.

[0067] It should also be noted that appearance features refer to the static, identity-related attributes extracted from the face in the original frame image. These features describe the inherent, stable visual information of the face, such as facial contours, the precise shape and layout of facial features, skin texture, skin color, pore details, hair patterns, and any permanent markers such as moles. By separating appearance features from transient dynamics and using them as a stable template, the digital human generation system can ensure that the core identity of the digital human remains unchanged regardless of facial movements when generating continuous video sequences, thus avoiding identity drift or facial distortion problems during the generation process. Dynamic features refer to the transient changes related to facial muscle movements extracted from the face in the driving frame image. These features capture changes in facial states, such as the shape of the mouth opening and closing, the raising or lowering of the corners of the mouth, the raising or frowning of the eyebrows, and the opening and closing of the eyelids.

[0068] For example, a base-state face model can be generated by extracting static appearance features from the original frame image based on a deep learning model with an encoder-decoder structure, such as U-Net (U-Net Convolutional Neural Network) or a similar variant, and the dynamic features of the face in the driving frame image can be transferred to the base-state face model.

[0069] For example, the original frame image is first analyzed by an encoder to extract its high-level appearance features. Specifically, the preprocessed original frame image is input into a predefined encoder. This encoder is typically composed of multiple stacked convolutional layers, pooling layers, and activation function layers. Low-level feature analysis is performed on the original frame image in the initial convolutional layer, identifying basic visual elements such as edges, corners, and color blocks. As the original frame image data propagates forward through layers in the neural network, it undergoes progressive nonlinear transformations and feature combinations through deeper convolutional layers. During this process, the spatial dimension of the original frame image data is progressively compressed, while the feature channel dimension is progressively increased, transforming it into higher-level appearance features. These high-level features represent the essential attributes of a face, such as facial contours, skeletal structure, and skin texture, while filtering out specific instantaneous dynamic information, such as facial expressions and subtle lighting changes. These appearance features are encoded into a representation containing all the static identity information of the target person, i.e., the ground-state face model. The ground-state face model can be represented by a feature tensor or a set of feature vectors, defining what the digital person to be generated looks like, but not yet including any specific facial expressions or lip movements.

[0070] Simultaneously, another encoder or a different branch of the same encoder is used to analyze the driving frame image, extracting dynamic features related to facial movement, such as mouth opening and closing, changes in the corners of the mouth, and eye opening and closing. Specifically, if the deep learning model architecture uses an independent dynamic feature encoder, the driving frame image data is directly routed to that independent dynamic feature encoder for processing. If the architecture uses different branches of a shared encoder, the driving frame image data is input into a shared basic feature extraction layer, such as a shallow convolution, and then, at pre-defined network layer points, the intermediate feature maps are branched to dedicated dynamic feature branches. Forward propagation calculations are performed on the selected dynamic feature extraction path to analyze the geometric deformation and motion patterns of the facial region. This analysis process can be implemented during the model training phase by training the network weights to be sensitive to dynamic information such as feature point displacement, muscle contour changes, and mouth opening and closing caused by facial expressions. Finally, spatial information is gradually compressed through convolution and pooling operations, while parameters related to dynamic features are encoded in high-dimensional feature channels. For example, the activation intensity of a specific feature channel may correspond to the degree of mouth corner raising, while another set of channels may correspond to the degree of eyebrow raising. The final output of the dynamic feature extraction path, namely a feature tensor that represents all the key dynamic information of the face in the current driving frame, is standardized and encapsulated to obtain the dynamic features of the face in the driving frame image.

[0071] Next, the decoder fuses the results from the first two steps, adaptively combining the appearance features obtained from the base-state face model with the dynamic features extracted from the driving frame image. Based on the combined features, the decoder reconstructs a new target face image that retains the identity appearance of the original frame image while accurately reproducing the facial dynamics of the driving frame image.

[0072] Step S30: Use the target face image and audio features as input to a pre-trained temporal denoising model to obtain the image noise of subsequent frames predicted by the temporal denoising model based on the audio features and the dynamic features of the target face image.

[0073] Audio features are extracted from video sequences using audio encoders such as Mel spectrograms or Wav2Vec (Waveform-to-Vector). The target face image and audio features of the current frame are input together into a temporal denoising model. In this embodiment, the temporal denoising model does not directly predict the pixel image of the next frame, but performs a denoising process in a compressed, abstract latent space, processing the noisy latent representation of subsequent frames that includes random noise. Specifically, the temporal denoising model predicts the noise to be added to subsequent frames using the dynamic features in the current frame and the audio features of subsequent frames as joint conditions. If the temporal denoising model predicts accurately, subtracting the predicted noise from the noisy latent representation yields a clean latent representation of subsequent frames. This prediction process is essentially a reasoning process by the temporal denoising model, after understanding how lip movements and facial expressions should co-evolve with audio based on the target face image and corresponding audio features of the current frame, to infer the dynamics that subsequent frames should have based on the audio features of the corresponding subsequent frames. After removing the noise predicted by the temporal denoising model from the noisy latent representation, a denoised latent representation is obtained. This denoised latent representation is then converted back to the image space by the decoder, or parameters controlling facial motion, such as facial motion vectors, are parsed from the denoised latent representation. These parameters are the predicted dynamic features for subsequent frames.

[0074] Step S40: Determine the predicted dynamic features of the target face image in subsequent frames based on the image noise.

[0075] Optionally, step S40 includes steps S41 to S42:

[0076] Step S41: Subtract the image noise from the random noise in the second preset number of noise frames to obtain the denoised image.

[0077] Step S42: Based on the facial expression and pose features of the face in the denoised image, determine the predicted dynamic features of the target face image in the subsequent frames.

[0078] For example, subtracting the corresponding noise predicted by the temporal diffusion Transformer network from the latent representation of each noisy frame yields N denoised latent image representations. This subtraction operation is a denoising step in the diffusion model generation process. The initially added random noise represents the uncertainty of various possible future states; while the predicted noise is impurity that the model judges based on the dynamic and audio features of the previous L frames and does not belong to the real future frames. Subtracting this noise is equivalent to extracting the most likely future state from the uncertainty.

[0079] A pre-trained feature encoder, such as the same regression network as in step S31, analyzes each denoised image latent representation, or the image decoded from that latent representation. The feature encoder regresses expression feature vectors describing facial muscle movement and pose feature vectors describing head spatial position from each latent representation. These extracted expression and pose feature vectors are then identified as the predicted dynamic features of the final target face image. The above analysis and extraction process is similar to that in step S31, but here the object of processing is the subsequent frames predicted by the temporal denoising model, rather than the actual current frame.

[0080] Step S50: Based on the predicted dynamic features and the ground-state face model, generate digital human images for subsequent frames.

[0081] Each predicted dynamic feature contains information about the facial expressions and poses of its corresponding predicted subsequent frames. These features can be combined with a ground-state face model and rendered by an image generator such as a decoder to produce specific, highly realistic target face images in subsequent frames, thus forming a dynamic and coherent digital human video sequence synchronized with speech.

[0082] In this embodiment, a staged collaborative processing strategy is used to generate high-fidelity digital human videos synchronized with speech. Specifically, firstly, original frames providing identity and appearance and driving frames providing dynamic references are selected from the video sequence. An encoder-decoder architecture is used to extract static appearance features from the original frames to construct a clean base-state face model, while dynamic features are decoupled from the driving frames. Then, the decoder transfers the dynamic features of the driving frames to the base-state model, generating a target face image of the current frame that combines the target identity with the driving expression / pose. Finally, the current frame image and audio features are input into a temporal denoising model. This model predicts the noise of subsequent frames in the latent space based on audio conditions and current dynamics, and obtains the predicted dynamic features of subsequent frames through denoising operations, thereby synthesizing a digital human video sequence with consistent identity, accurate lip-sync, and coherent dynamics. This method significantly improves the identity consistency, visual realism, and lip-sync accuracy of the generated video through effective decoupling and collaborative control of appearance and dynamics, thus enhancing the naturalness of the generated digital human.

[0083] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description and will not be repeated hereafter. Based on this, the dynamic features include facial expression features and posture features. Please refer to... Figure 2 Step S30 includes steps S31 to S33:

[0084] Step S31: The facial expression features and pose features of the target face image are concatenated to form a reference latent vector.

[0085] Optionally, the temporal denoising model uses a pre-trained regression network, such as a residual neural network or a mobile neural network, to analyze the target face image. The regression network regresses parameters describing facial muscle movement, i.e., expression features, from the target face image. These parameters are, for example, a numerical vector, where each dimension corresponds to a certain expression intensity or a low-dimensional expression embedding. Simultaneously, the regression network regresses parameters describing the spatial position of the head, i.e., pose features. Preferably, the pose features can be represented by a 6-DOF vector, including three rotational angles (yaw, pitch, and roll) and three translational amounts.

[0086] The temporal denoising model concatenates the extracted facial expression feature vector and pose feature vector along the feature dimension to obtain a reference latent vector.

[0087] Step S32: Expand the current frame to a first preset number of frames based on the reference potential vector.

[0088] For example, the temporal denoising model copies the reference latent vector representing the current moment and expands it along the time series dimension to construct a sequence of length L, i.e., a first preset number of frames. In this sequence, the latent representation of each frame is exactly the same as that of the current frame. The purpose of the above steps is to provide a clear temporal starting point and context for the temporal denoising model to perform predictions, indicating that the prediction will begin from the steady state of these L frames.

[0089] Optionally, step S32 includes steps S321 to S322:

[0090] Step S321: Remove background elements other than the face region from the target face image using a pre-trained matting algorithm.

[0091] Step S322: Based on the reference latent vector, the current frame after removing background elements is extended to the first preset number of frames.

[0092] For example, the matting algorithm performs pixel-level classification on the input target face image, dividing each pixel in the image into either a face or background, generating a high-precision alpha mask. This alpha mask is a single-channel grayscale image of the same size as the input target face image, where pixel values ​​in the face region are non-zero preset values, such as 1, and pixel values ​​in the background region are 0, representing complete transparency. Using this alpha mask, the input target face image is composited with a solid-color background, or the image with the alpha channel is directly retained. All background visual information unrelated to the face is removed, generating a target face image with background elements removed. This target face image contains only the face region, eliminating the potential interference of background information on subsequent temporal prediction tasks.

[0093] Then, the reference latent vector is copied and expanded along the time series dimension to construct a sequence of length L (the first preset number) representing the clear current state. Here, the source image used to add random noise is the current frame after background removal, not the original current frame. The generated N noise frames (the second preset number) therefore only contain noisy face regions. The L-frame reference latent vector sequence and the N-frame sequence containing only face noise are concatenated along the time dimension to form a complete sequence of length L+N, which serves as the input latent vector.

[0094] Optionally, step S32 may further include steps S323 to S325:

[0095] Step S323: The facial expression features and pose features of the face in the driving frame image are concatenated to form a prediction latent vector.

[0096] Step S324: Generate an intermediate latent vector between the reference latent vector and the predicted latent vector using a temporal difference algorithm, and construct an intermediate frame based on the intermediate latent vector.

[0097] Step S325: Arrange the current frame and the intermediate frames in chronological order to obtain the first preset number of frames.

[0098] The face in the driving frame image is processed using the same method as in step S31 to obtain the predicted latent vector. A series of intermediate latent vectors are generated between the reference latent vector and the predicted latent vector using temporal interpolation algorithms such as linear interpolation or spherical linear interpolation (SLERP). The interpolation coefficients can be uniformly distributed within a first preset number - 1 (L-1) as needed, or set according to other preset distributions. Each intermediate latent vector is input into the image generator to reconstruct the corresponding intermediate frame image, which represents a reasonable transition from the current state to the driving frame state.

[0099] All frames are arranged chronologically, ensuring the total number of frames equals a first preset number L, thus constructing a dynamically smooth extended sequence of length L. Compared to simple copying, the first preset number of frames generated by interpolation provide realistic motion trajectories, offering rich motion priors for the temporal denoising model. This makes the predicted motion of future frames more consistent with physical laws, significantly reducing abrupt changes in facial expressions and postures.

[0100] Step S33: The first preset number of frames and the second preset number of noise frames are concatenated as the input latent vector. The noise frames are obtained by adding random noise to the current frame.

[0101] Temporal denoising models need to predict subsequent frames, therefore randomness needs to be introduced into the input to represent future uncertainty. For example, a temporal denoising model can add Gaussian random noise of varying intensities to the target face image or its latent representation in the current frame, generating N (a second preset number) different noise frames. These noise frames represent various possible "blurred" or "noisy" initial states for subsequent frames. The temporal denoising model concatenates the expanded L-frame reference latent vector sequence with the generated N-frame noise frame sequence in the time dimension, combining them into a complete sequence of length L+N, i.e., the input latent vector sequence. The first L frames of the input latent vector sequence represent the deterministic current state, providing a solid context for the temporal denoising model; the following N frames represent random noise, which is the target that the temporal denoising model needs to denoise and predict. This combined sequence completely defines a prediction task from a clear present state to a noisy future and is fed into the core denoising network of the temporal denoising model for processing.

[0102] Step S34: Input the input latent vector into the temporal diffusion Transformer network of the temporal denoising model, and obtain the predicted noise of the second preset number of noise frames output by the temporal diffusion Transformer network as the image noise.

[0103] In this embodiment, the temporal diffusion Transformer network is used to understand the inter-frame temporal context and learn how to predict noise from noisy subsequent frames based on conditional information such as audio features.

[0104] For example, the temporal diffusion Transformer network receives the entire input latent vector sequence. Internally, it models the temporal relationships between all frames in the sequence through a multi-head self-attention mechanism. For instance, the temporal diffusion Transformer network analyzes the contextual information established by the clear reference latent vectors of the first L frames, i.e., the current state. Simultaneously, the temporal diffusion Transformer network analyzes the current state of the subsequent N noisy frames. Through the self-attention mechanism, the temporal diffusion Transformer network allows each frame in the sequence to interact with all other frames, enabling the subsequent N noisy frames to fully perceive and understand the dynamic context provided by the first L frames. In this process, the audio features corresponding to the subsequent N noisy frames are also injected into the temporal diffusion Transformer network as conditional information, i.e., the representation of the noisy frame is used as the query, and the corresponding audio features are used as the key and value, so that the temporal diffusion Transformer network can always refer to the speech content that needs to be synchronized when analyzing how to denoise the noisy frames. Based on the above temporal relationship analysis and conditional injection, the core task of the temporal diffusion Transformer network is to predict the random noise added to the subsequent N noisy frames. After layers of forward propagation calculation, the temporal diffusion Transformer network finally generates a predicted noise sequence for each of the subsequent N frames in the sequence at the output end, which is the image noise mentioned above.

[0105] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The step S20, which generates a base-state face model based on the facial appearance features in the original frame image, includes steps S21 to S24:

[0106] Step S21: Obtain the facial key points in the original frame image detected by the face detection algorithm, and perform outward cropping based on the facial key points to obtain the face region.

[0107] The original frame image is processed using a pre-trained face detection algorithm to locate facial key points such as facial contours and facial features. Based on all detected facial key points, their minimum bounding polygon or elliptical bounding boxes are calculated. Subsequently, these polygons or ellipses are expanded by 15%-20% to ensure complete coverage of the entire facial region, including the forehead, hairline, and profile. Finally, the original frame image is cropped based on the expanded polygon or elliptical bounding box to obtain a compact image that fully contains facial information.

[0108] Step S22: Detect the three-dimensional key points of the face in the face region, and map the three-dimensional key points onto the face region through the camera model to obtain a ground-state face image with two-dimensional projection key points.

[0109] Using a regression network based on a 3D Morphable Model (3DMM) or other pre-defined algorithms, the coordinates of 3D key points of the face are estimated from the face region image. A virtual camera model is determined based on the intrinsic and extrinsic parameters of the video sequence. The 3D key points are then accurately projected onto the 2D image plane using this virtual camera model to obtain the 2D projected key points.

[0110] Step S23: Using a pre-trained multi-scale convolutional neural network, the pose features, appearance features, and global appearance descriptors corresponding to the appearance features of the face are obtained from the ground-state face image based on the two-dimensional projection key points.

[0111] By inputting the ground-state face image into a multi-scale convolutional neural network, the multi-scale architecture of the multi-scale convolutional neural network can simultaneously capture different granular information of the ground-state face image, from local details such as texture to global structure such as face shape.

[0112] For example, a single-channel heatmap with the same spatial size as the feature map is first generated for each 2D projected keypoint. On this heatmap, the value at the location of the 2D projected keypoint is the highest, decaying outwards according to a Gaussian distribution. After the base-state face image is input into a multi-scale convolutional neural network, feature maps of different resolutions (scales) are generated. At each scale where keypoint information needs to be utilized, the corresponding keypoint heatmap is fused with the extracted image feature map. For example, the keypoint heatmap can be used as attention weights, multiplied point-by-point with the image feature map. This enhances the feature responses in the region near the keypoint in the feature map, while suppressing the feature responses in the background region. Alternatively, the keypoint heatmap can be used as an additional input channel, concatenated with the image feature map in the channel dimension, and then the network learns how to utilize this positional information through convolutional layers. Guided by the keypoint information, the network decouples the face's pose features, appearance features, and the corresponding global appearance descriptor from the modulated feature map using different task heads.

[0113] The global appearance descriptor compresses spatial information through a global pooling layer, aggregating appearance features into a compact, fixed-length global vector. This global vector is a high-level summary of the appearance features, facilitating rapid identity comparison.

[0114] Step S24: Perform an inverse transformation on the appearance features of the face based on the pose features to obtain the ground state face model.

[0115] The pre-trained mapping network calculates the corresponding appearance deformation field or correction parameters based on the pose features. The inverse operation, i.e., the inverse transformation, is then performed on this appearance deformation field or correction parameters and applied to the original appearance features.

[0116] The appearance deformation field is a grid of the same size as the feature map or image, where each feature point or pixel contains a displacement vector. This displacement vector indicates in which direction and by how much the feature at that point should be moved to counteract the current expression. The correction parameters are a set of transformation parameters that can be directly applied to the appearance feature tensor. For example, a set of affine transformation parameters that can weight and shift feature channels.

[0117] The detected pose feature vectors are input into a mapping network, which, after forward propagation, outputs either an appearance deformation field or a set of correction parameters. To obtain a neutral appearance, the digital human generation system performs an inverse operation on the output of the mapping network: if the output is an appearance deformation field, the inverse operation applies the inverse field of that deformation field to the face. If the output is correction parameters, such as a transformation matrix, the inverse operation applies the inverse of that matrix to the face.

[0118] It's important to note that the mapping network is learned through a large amount of training data. Each set of training data includes a source image of a person in a certain pose and a target image of that person in a standard pose with the same expression. Pose features and original appearance features are extracted from the source image. The pose features are then input into the mapping network to be trained, resulting in a predicted deformation. The inverse transformation of this predicted deformation is applied to the appearance features of the source image to generate a corrected feature. The generated feature is compared with the appearance features extracted from the target image, and the parameters of the mapping network are continuously adjusted through backpropagation to minimize the difference between the two, thus training the mapping network.

[0119] In this embodiment, the key operation of inverse pose transformation actively eliminates the influence of head pose on the identity appearance features in the original frame image, thereby obtaining a base-state face model representing a standard frontal view identity. This lays the foundation for subsequent driving and generation processes, improves the consistency of generated digital human identities, and effectively avoids deformation distortion caused by inconsistent poses of the base template. Furthermore, due to the use of 3D keypoints and pose normalization, this embodiment has more lenient requirements for the original frame image. Even if the original frame image is a side view or a non-frontal angle of the face, the digital human generation system can effectively correct its appearance to a standard pose through inverse transformation, significantly enhancing its adaptability and processing effect on video data.

[0120] Based on the above embodiments, in the fourth embodiment of this application, please refer to Figure 4After step S24, which involves inversely transforming the facial appearance features based on the pose features, the process further includes steps S25-S27:

[0121] Step S25: After aligning the ground state face image with the standard face model, obtain the facial feature mask in the ground state face image through a preset face segmentation model.

[0122] For example, a digital human generation system stores a standard face model, such as an averaged 3D face mesh or a face model with standard key points. By detecting preset key points of the face in the image, and then using affine or perspective transformation, the face in the image is rotated, scaled, and translated to the pose and size corresponding to the standard face model based on the preset key points, thus aligning the base-state face image with the standard face model.

[0123] On the aligned base-state face image, a pre-defined face segmentation model such as U-Net is used for pixel-level classification, assigning a label to each pixel in the image, such as: skin, left eye, right eye, nose, upper and lower lips, eyebrows, etc. Facial feature regions are then determined from these labels, generating a facial feature mask. This mask is a single-channel image of the same size as the base-state face image, where different facial feature regions are assigned different grayscale values ​​to accurately identify the contours and positions of each facial feature component.

[0124] Step S26: Extract facial feature images from the ground-state face image according to the facial feature mask, and determine the facial expression features in the ground-state face image according to the two-dimensional projection key points in the facial feature images.

[0125] For example, based on the aforementioned facial feature mask, individual facial feature regions are cropped from the aligned base-state face image. For each extracted facial feature image, a keypoint detector is used to identify its fine two-dimensional projected keypoints. The number and position of the two-dimensional projected keypoints corresponding to different facial features can be set by the user or determined through pre-training. These two-dimensional projected keypoints are compared with the facial feature keypoints corresponding to a standard face model under neutral expression. By analyzing the changes in geometric relationships such as relative position, distance, and angle between keypoints, a set of parameters representing the degree to which the current image deviates from a neutral expression, i.e., expression features, is regressed to quantify the subtle expressions that may exist in the original frame.

[0126] Step S27: Perform an inverse transformation on the facial appearance features based on the expression features to obtain the ground state facial model.

[0127] The inverse transformation described above is consistent with the steps in Example 3 for inverse transformation of facial appearance features based on pose features. The only difference is that during the training of the mapping network, expression features and original appearance features are extracted from the source image. The expression features are then input into the mapping network to be trained, resulting in a predicted deformation. This inverse transformation of the predicted deformation is applied to the appearance features of the source image to generate a corrected feature. The generated feature is compared with the appearance features extracted from the target image, and the parameters of the mapping network are continuously adjusted through backpropagation to minimize the difference between the two, thus training the mapping network.

[0128] In this embodiment, by actively eliminating the interference of subtle facial expressions in the original frames, the generated base-state face model is more representative of the neutral, static appearance of the person. This ensures the consistency and stability of the digital human's identity features throughout the entire video sequence generation process, effectively avoiding drift or distortion of digital human identity attributes. Lip-syncing and facial expression synthesis based on the base-state face model is more accurate and natural, reducing unreasonable deformations or artifacts, thereby significantly improving the visual realism and overall naturalness of the final generated digital human video.

[0129] Based on the above embodiments, in the fifth embodiment of this application, the step S20 of transferring the dynamic features of the face in the driving frame image to the base state face model to obtain the target face image of the current frame includes steps S28-S29:

[0130] Step S28: The multi-scale convolutional neural network obtains driving expression features and driving posture features from the driving frame image based on the two-dimensional projection key points of the face in the driving frame image.

[0131] Step S29: Based on the global appearance descriptor, the driving expression features are transferred to the base state face model, and the pose parameters of the base state face model are determined according to the driving pose features to obtain the target face image.

[0132] The digital human generation system first performs the same preprocessing procedure on the driving frame image as on the base-state face image, detecting preset key points of the face in the driving frame image and performing outward cropping to obtain the face region image of the driving frame. A 3DMM regression network is used to estimate the 3D key points of this face region image, and then projected through a virtual camera model to obtain precise 2D projected key points on the driving frame image. The processed driving frame image with 2D projected key points is then input into a multi-scale convolutional neural network. The multi-scale convolutional neural network modulates the image features using the same key point heatmap as spatial attention guidance, decoupling the driving expression features and driving pose features.

[0133] Optionally, the digital human generation system uses a conditional generative network based on U-Net or StyleGAN (Style-Based Generative Adversarial Network) architecture to generate target face images.

[0134] For example, the driving expression features and the global appearance descriptor are concatenated or combined into a joint conditional vector through a small fusion network, such as several fully connected layers. The role of the global appearance descriptor is to ensure that during subsequent generation, when the conditional generation network changes its expression, its underlying identity information, such as face shape, bone structure, and skin texture, is always pulled back to the identity defined by the base face model. The driving pose features can be directly used as pose parameters of the target image. If the base face model exists in the form of feature maps, then the feature maps can be subjected to corresponding geometric transformations such as rotation and translation according to the driving pose parameters, placing them from a standard frontal view to the head orientation of the driving frame. If the base model is parameterized, the driving pose parameters can be directly input into the renderer. Then, the joint conditional vector is injected into different layers of the generator and used to calculate the scaling and offset parameters of the normalization layer, thereby controlling the statistical properties of each layer of feature maps and realizing the transfer of expression features. Finally, the feature maps after expression modulation and pose transformation are passed through a series of upsampling layers and convolutional layers to finally output an RGB image, obtaining the target face image.

[0135] In this embodiment, the global appearance descriptor is a highly compact feature vector. Using it as a condition for expression transfer reduces the computational overhead of complex transformations of high-dimensional feature maps, making the transfer process more efficient. Simultaneously, by introducing the global appearance descriptor as an intermediate medium, the identity information of the base-state face model can be locked when transferring the dynamic features of the driving frame. This ensures that the generated target face image is consistent with the driving frame in expression and pose, while its identity appearance is unified with the original frame. Furthermore, this embodiment treats expression transfer and pose determination as two relatively independent sub-processes. This decoupled transfer method allows for more precise and flexible control over the generation results. For example, the pose can be fixed to transfer only the expression, or expressions and poses from different driving sources can be combined, providing greater controllability for digital human generation.

[0136] Based on the above embodiments, in the sixth embodiment of this application, referring to... Figure 5 After step S40, the voice-driven digital human generation method further includes steps S60-S70:

[0137] Step S60: In response to the received text information, the text information is converted into a semantic feature vector.

[0138] Step S70: The semantic feature vector is used as the frequency domain modulation parameter of the Fourier convolutional layer in the pre-trained image restoration model. The background elements of the subsequent frame are regenerated through the image restoration model. The frequency domain modulation parameter includes a spectral bias vector and a channel weight vector.

[0139] For example, a text encoder, such as a Transformer-based pre-trained model, BERT (Bidirectional Encoder Representations from Transformers), or CLIP (Contrastive Language-Image Pre-training), is used to segment the input text information, converting it into a series of word embedding vectors. The text encoder, through its internal attention mechanism and other structures, analyzes the contextual relationships between words, understands the global semantics of the entire text, and outputs semantic feature vectors. These semantic feature vectors are then passed through two different fully connected layers to generate a spectral bias vector and a channel weight vector, respectively. In the Fourier convolutional layer, the Fourier spectrum of the input feature map is transformed. The spectral bias vector is added to the spectrum to shift the overall frequency distribution, similar to adjusting the base hue and macroscopic structure; the channel weight vector is multiplied by the spectrum to scale the importance of different feature channels, similar to adjusting the strength of different visual elements. In this way, the semantic information of the text is directly and efficiently injected into the frequency domain of the image generation process, thereby controlling the overall style and content of the background.

[0140] The image inpainting model takes the background-removed target face image as input, uses the aforementioned frequency domain modulation as a condition to generate an image, and restores a reasonable and coherent background image around the face region based on the text semantics.

[0141] The above embodiments automate the background generation process and seamlessly integrate it with the digital human generation workflow, avoiding tedious manual background editing or shooting work, and realizing end-to-end automated content production.

[0142] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the voice-driven digital human generation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0143] This application provides a voice-driven digital human generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the voice-driven digital human generation method in Embodiment 1 above.

[0144] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing a voice-driven digital human generation device according to embodiments of this application. The voice-driven digital human generation device in these embodiments may include, but is not limited to, mobile terminals such as laptops and tablets (PADs, Portable Application Description) and fixed terminals such as desktop computers. Figure 6 The voice-driven digital human generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0145] like Figure 6 As shown, the voice-driven digital human generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the voice-driven digital human generation device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the voice-driven digital human generation device to communicate wirelessly or wiredly with other devices to exchange data. Although voice-driven digital human generation devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0146] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0147] The voice-driven digital human generation device provided in this application, employing the voice-driven digital human generation method in the above embodiments, can solve the technical problem of how to improve the naturalness of digital human generation. Compared with the prior art, the beneficial effects of the voice-driven digital human generation device provided in this application are the same as those of the voice-driven digital human generation method provided in the above embodiments, and other technical features in this voice-driven digital human generation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0148] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0150] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the voice-driven digital human generation method in the above embodiments.

[0151] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0152] The aforementioned computer-readable storage medium may be included in a voice-driven digital human generation device; or it may exist independently and not be assembled into a voice-driven digital human generation device.

[0153] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a voice-driven digital human generation device, enable the voice-driven digital human generation device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0156] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described voice-driven digital human generation method, thereby solving the technical problem of how to improve the naturalness of digital human generation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the voice-driven digital human generation method provided in the above embodiments, and will not be repeated here.

[0157] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the voice-driven digital human generation method described above.

[0158] The computer program product provided in this application solves the technical problem of how to improve the naturalness of digital human generation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the voice-driven digital human generation method provided in the above embodiments, and will not be repeated here.

[0159] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A voice-driven digital human generation method, characterized by, The voice-driven digital human generation method comprises the following steps: selecting original frame images and driving frame images from a video sequence; obtaining the face key points in the original frame images detected by a face detection algorithm, performing external expansion cropping based on the face key points to obtain a face region; detecting three-dimensional key points of the face in the face region, mapping the three-dimensional key points to the face region through a camera model to obtain a ground state face image with two-dimensional projection key points; obtaining the pose feature, appearance feature and global appearance descriptor corresponding to the appearance feature of the face from the ground state face image according to the two-dimensional projection key points through a pre-trained multi-scale convolutional neural network; performing inverse transformation on the appearance feature of the face based on the pose feature to obtain a ground state face model, and migrating the dynamic feature of the face in the driving frame image to the ground state face model to obtain a target face image of a current frame; taking the target face image and audio feature as the input of a pre-trained time series denoising model, and splicing the expression feature and pose feature of the face in the target face image as a reference latent vector; extending the current frame to a first preset number of frames based on the reference latent vector; splicing the first preset number of frames and a second preset number of noise frames as an input latent vector, wherein the noise frames are obtained by adding random noise to the current frame; inputting the input latent vector into a time series diffusion Transformer network of the time series denoising model to obtain the predicted noise of the second preset number of noise frames output by the time series diffusion Transformer network as image noise; determining the predicted dynamic feature of the target face image of a subsequent frame according to the image noise; generating a digital human image of a subsequent frame based on the predicted dynamic feature and the ground state face model.

2. The voice-driven digital human generation method of claim 1, wherein, The step of extending the current frame to a first preset number of frames based on the reference latent vector comprises: splicing the expression feature and pose feature of the face in the driving frame image as a prediction latent vector; generating an intermediate latent vector between the reference latent vector and the prediction latent vector through a time series difference algorithm, and constructing an intermediate frame according to the intermediate latent vector; arranging the current frame and the intermediate frame in chronological order to obtain the first preset number of frames.

3. The voice-driven digital human generation method of claim 1, wherein, The step of extending the current frame to a first preset number of frames based on the reference latent vector further comprises: removing background elements other than the face region in the target face image through a pre-trained matting algorithm; extending the current frame after removing the background elements to the first preset number of frames based on the reference latent vector.

4. The voice-driven digital human generation method of claim 1, wherein, The step of determining the predicted dynamic feature of the target face image of a subsequent frame according to the image noise comprises: subtracting the image noise from the random noise in the second preset number of noise frames to obtain a denoised image; determining the predicted dynamic feature of the target face image of the subsequent frame according to the expression feature and pose feature of the face in the denoised image.

5. The voice-driven digital human generation method of claim 1, wherein, The step of performing inverse transformation on the appearance feature of the face based on the pose feature further comprises: After aligning the base-state face image with a standard face model, a preset face segmentation model is used to obtain a facial feature mask in the base-state face image; According to the facial feature mask, facial feature images are extracted from the base-state face image, and expression features of the face in the base-state face image are determined according to two-dimensional projection key points in the facial feature images; Based on the expression features, the appearance features of the face are inversely transformed to obtain the base-state face model.

6. The voice-driven digital human generation method of claim 5, wherein, The step of migrating the dynamic features of the face in the driving frame image to the base-state face model to obtain the target face image of the current frame includes: According to the two-dimensional projection key points of the face in the driving frame image, driving expression features and driving pose features are obtained from the driving frame image by using the multi-scale convolutional neural network; Based on the global appearance descriptor, the driving expression features are migrated to the base-state face model, and pose parameters of the base-state face model are determined according to the driving pose features to obtain the target face image.

7. The speech-driven digital human generation method of any one of claims 1-6, wherein, After the step of determining the predicted dynamic features of the target face image of the subsequent frame according to the image noise, the method further includes: In response to the received text information, the text information is converted into a semantic feature vector; The semantic feature vector is used as a frequency domain modulation parameter of a Fourier convolution layer in a pre-trained image restoration model, and background elements of the subsequent frame are regenerated by using the image restoration model, wherein the frequency domain modulation parameter includes a frequency spectrum bias vector and a channel weight vector.

8. A speech-driven digital human generation device, characterized by, The device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the voice-driven digital human generation method according to any one of claims 1 to 7.

Images