Digital human video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a digital human video generation method and device, electronic equipment and a storage medium, and relates to the technical field of digital human video generation, an initial predictor and an initial prediction model in a digital human video generation model adopted by the method are both constructed based on an SANA framework, and by means of the SANA framework, the initial predictor and the initial prediction model can be effectively predicted. According to the method, the generated digital human video can be more stable and rich in details, computing resources needed for generating the digital human video are reduced, the generation speed of the digital human video is improved, it is guaranteed that the method can be operated in real time on consumption-level hardware, the deployment cost and the application threshold are reduced, and the application scene is expanded. In addition, a target style code can be predicted from the reference audio by adopting a style predictor, and diversified emotion expressions can be generated, so that a digital human video with rich emotions is generated.

Description

Technical Field

[0001] This invention relates to the field of digital human video generation technology, and in particular to a digital human video generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of digital human video generation, existing technologies are mainly divided into two categories: one is based on traditional generation models, and the other is based on diffusion models.

[0003] Traditional generative model-based methods, such as Generative Adversarial Networks (GANs), generate high-quality images through adversarial training. However, when generating digital humans with complex emotional expressions and diverse movements, they suffer from unstable results and insufficient detail. While diffusion-based methods can generate high-quality digital human videos, they are computationally complex, consume significant computational resources, and are slow, making them difficult to meet the application requirements for real-time generation and deployment.

[0004] Moreover, existing digital human video generation methods have limited emotional expression capabilities, failing to accurately capture and express the emotions input by users, and lacking vividness and diversity. Furthermore, because existing digital human video generation methods consume significant computing resources, they are difficult to run on ordinary consumer-grade hardware, increasing deployment costs and application barriers. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for generating digital human videos, in order to address the deficiencies existing in related technologies.

[0006] This invention provides a method for generating digital human videos, comprising: Obtain reference audio and images of the individuals; The reference audio and the image of the person are input into the digital human video generation model to obtain the digital human video output by the digital human video generation model. The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer; the style predictor is used to predict a target style code from the reference audio; the facial motion prediction model is used to apply motion noise and the reference audio, and generate facial motion parameters based on the target style code; the renderer is used to generate a digital human video based on the person image and the facial motion parameters. The style predictor is obtained by training an initial predictor based on noisy style code samples and audio and video style codes in video samples. The facial motion prediction model is obtained by training an initial prediction model based on a second audio sample, a style reference video, and facial motion parameter labels in the style reference video. Both the initial predictor and the initial prediction model are built on the SANA framework.

[0007] According to a digital human video generation method provided by the present invention, the style predictor is specifically used for: Based on the image of the person, the target style code is predicted from the reference audio.

[0008] According to a digital human video generation method provided by the present invention, the style predictor is further specifically used for: The target style code is determined based on the first audio feature, the second audio feature, and the image of the person. The first audio feature is extracted from the reference audio based on a speech pre-trained model; The second audio feature is extracted from the reference audio based on a speech emotion base model.

[0009] According to the present invention, a digital human video generation method is provided, wherein the facial motion prediction model includes an audio encoder and a linear DiT module; The audio encoder is used to extract the audio encoding of the reference audio; The linear DiT module is used to obtain the facial motion parameters based on the motion noise, the audio encoding, and the target style code.

[0010] According to a digital human video generation method provided by the present invention, the renderer is specifically used for: Extract the spatial features and latent feature representations of the image of the person; Based on the facial motion parameters, the spatial features are deformed to obtain deformed features; The deformed features are fused with the latent feature representation to form an enhanced feature representation; The enhanced feature representation is input into the temporal attention module to obtain the temporal attention feature output by the temporal attention module; The temporal attention features are decoded to gradually reconstruct each video frame in the digital human video.

[0011] According to a digital human video generation method provided by the present invention, the initial prediction model includes an initial audio encoder, an initial style encoder, and an initial linear DiT module; The facial motion prediction model is trained on the initial prediction model based on the following steps: The second audio sample is input into the initial audio encoder to obtain the audio encoded sample output by the initial audio encoder; The style reference video is input into the initial style encoder to obtain the reference style code output by the initial style encoder; The audio encoded sample, the reference style code, and the noise sample are input into the initial linear DiT module to obtain the predicted facial motion parameters output by the initial linear DiT module. Based on the predicted facial motion parameters and the facial motion parameter labels, the initial prediction model is iteratively trained to obtain the trained prediction model, and the style encoder in the trained prediction model is removed to obtain the facial motion prediction model.

[0012] According to a digital human video generation method provided by the present invention, the digital human video generation model is deployed on a cloud server, or the digital human video generation model is deployed on an edge device after optimization and cropping; The step of inputting the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model includes: Based on the application programming interface, the model deployed on the cloud server or the edge device is called, and the reference audio and the image of the person are input into the called model to obtain the digital human video.

[0013] The present invention also provides a digital human video generation apparatus, comprising: The acquisition module is used to acquire reference audio and images of people; The generation module is used to input the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model; The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer; the style predictor is used to predict a target style code from the reference audio; the facial motion prediction model is used to apply motion noise and the reference audio, and generate facial motion parameters based on the target style code; the renderer is used to generate a digital human video based on the person image and the facial motion parameters. The style predictor is obtained by training an initial predictor based on noisy style code samples and audio and video style codes in video samples. The facial motion prediction model is obtained by training an initial prediction model based on a second audio sample, a style reference video, and facial motion parameter labels in the style reference video. Both the initial predictor and the initial prediction model are built on the SANA framework.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the digital human video generation method as described above.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the digital human video generation method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the digital human video generation method as described above.

[0017] The present invention provides a digital human video generation method, apparatus, electronic device, and storage medium. First, reference audio and a human image are acquired. Then, the reference audio and human image are input into a digital human video generation model to obtain a digital human video output by the model. The initial predictor and initial prediction model in this method are both built on the SANA framework. Leveraging the SANA framework, the generated digital human video is more stable and detailed, while reducing the computational resources required for generation, increasing the generation speed, and ensuring real-time operation on consumer-grade hardware. This reduces deployment costs and application barriers, and expands application scenarios. Furthermore, the use of a style predictor allows for the prediction of target style codes from the reference audio, enabling the generation of diverse emotional expressions and thus generating digital human videos with rich emotional content. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the digital human video generation method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the renderer's workflow in the digital human video generation method provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the workflow of the style predictor in the digital human video generation method provided by the present invention.

[0022] Figure 4This is a schematic diagram of the training process of the initial prediction model in the digital human video generation method provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the digital human video generation device provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Existing digital human video generation methods suffer from several drawbacks, including unstable generation results, insufficient detail, high computational complexity, high computational resource consumption, slow generation speed, difficulty in meeting real-time generation and deployment requirements, limited emotional expression capabilities of generated digital human videos, inability to accurately capture and express user-input emotions, lack of vividness and diversity, high computational resource consumption, difficulty in running on ordinary consumer hardware, and increased deployment costs and application barriers. Therefore, this invention provides a digital human video generation method.

[0027] Figure 1 This is a flowchart illustrating a digital human video generation method provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes: S1, obtain reference audio and character images; S2, input the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model; The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer; the style predictor is used to predict a target style code from the reference audio; the facial motion prediction model is used to apply motion noise and the reference audio, and generate facial motion parameters based on the target style code; the renderer is used to generate a digital human video based on the person image and the facial motion parameters. The style predictor is obtained by training an initial predictor based on noisy style code samples and audio and video style codes in video samples. The facial motion prediction model is obtained by training an initial prediction model based on a second audio sample, a style reference video, and facial motion parameter labels in the style reference video. Both the initial predictor and the initial prediction model are built on the SANA framework.

[0028] Specifically, the digital human video generation method provided in this embodiment of the invention is executed by a digital human video generation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0029] First, perform step S1 to obtain reference audio and a person image. The digital human image in the generated digital human video is identical to the person image, and the timbre and speech content of the digital human video are identical to those in the reference audio.

[0030] Then, step S2 is executed to introduce the digital human video generation model. Reference audio and a person image are input into the digital human video generation model to obtain the digital human video output by the model.

[0031] The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer. Reference audio is input to the style predictor, which predicts the target style code from the reference audio. Alternatively, the reference audio and a noise style code can be input to the style predictor to obtain the target style code output by the style predictor. The noise style code can be obtained by adding Gaussian noise to any real style code. The target style code is used to characterize the speaking style in the reference audio and to control the style of the digital human image in the generated digital human video.

[0032] The facial motion prediction model is fed with motion noise, reference audio, and target style code. The model then applies the motion noise and reference audio, conditioned on the target style code, to generate facial motion parameters. These parameters characterize the speaker's facial expressions in the reference audio.

[0033] like Figure 2 As shown, by inputting the image of the person and facial motion parameters into the renderer, a digital human video output by the renderer can be obtained.

[0034] The style predictor in the digital human video generation model is obtained by training the initial predictor using noisy style code samples, audio and video style codes from video samples.

[0035] First, video samples are obtained. These video samples can come from the MEAD dataset, HDTF dataset, and Voxceleb2 dataset. The videos in the MEAD dataset have diverse speaking styles and emotional expressions, which are suitable for training emotional models. The HDTF dataset provides high-quality facial motion capture data, which helps improve the model's detail performance. The Voxceleb2 dataset provides rich speaker identity information, which helps the model learn the features of different speakers.

[0036] After obtaining the video samples, preprocessing operations can be performed, including cropping, cleaning, and annotation. Here, the video samples can be cropped into fixed-length video segments, and segments that do not meet the requirements, such as those with occlusion, insufficient lighting, or poor video quality, can be cleaned out. Furthermore, each video segment of the video samples can be annotated, including sentiment analysis.

[0037] Subsequently, audio and video style codes are extracted from video samples as training data. Video style codes can be extracted from video samples using a style encoder. The style encoder can use 3DDMorphable models (3DMMs) to extract 3DMM expression parameter sequences from video samples, process them through a transformer encoder, and aggregate them through a self-attention pooling layer to obtain the video style codes. Noisy style code samples can be obtained by adding Gaussian noise to the video style codes.

[0038] Then, the first audio feature samples in the audio of the video samples can be extracted using a pre-trained speech model, which can be a pre-trained HuBERT model.

[0039] To enhance the emotional expression of the speech, a second audio feature sample is extracted from the audio of the video samples using a speech emotion pedestal model. This speech emotion pedestal model can be an emotion2vec model.

[0040] The first audio feature sample and the second audio feature sample are concatenated to obtain the sample concatenation result. The sample concatenation result and the noise style code sample are then input into the initial style predictor to obtain the prediction result.

[0041] Based on the prediction results and the video style code, a first training loss can be calculated. The initial style predictor is then iteratively trained using this first training loss. When the first training loss converges or reaches a first preset number of iterations, the style predictor is obtained. The first preset number of iterations can be set as needed and is not specifically limited here.

[0042] The facial motion prediction model in the digital human video generation model can be trained on the initial prediction model using a second audio sample, a style reference video, and facial motion parameter labels from the style reference video. The facial motion parameter labels can be extracted from the style reference video using a 3DMM.

[0043] The initial prediction model takes as input a second audio sample and a style reference video, and outputs predicted facial motion parameters extracted from the style reference video. By inputting the second audio sample and the style reference video into the initial prediction model, the predicted facial motion parameters output by the initial prediction model can be obtained.

[0044] A second training loss can be calculated based on the predicted facial motion parameters and their labels. The initial prediction model is then iteratively trained using this second training loss. When the second training loss converges or reaches a second preset number of iterations, the facial motion prediction model is obtained. The second preset number of iterations can be set as needed and is not specifically limited here.

[0045] It is understood that the initial predictor and initial prediction model used in the embodiments of this invention can both be built based on the SANA framework. The SANA framework includes a flow matching model and a flow matching model decoder (Flow-DPM-Solver). The flow matching model mainly fits the data distribution through flow matching and learns the mapping relationship between input and output through the objective function of flow matching, which can reduce computational complexity and the number of parameters while ensuring the generation quality of digital human videos. The flow matching model decoder can reduce generation steps, speed up inference, and achieve more efficient sampling. By optimizing the sampling process, the model can significantly shorten the generation time while ensuring generation quality.

[0046] The digital human video generation method provided in this embodiment of the invention first acquires reference audio and a person image; then, it inputs the reference audio and person image into a digital human video generation model to obtain a digital human video output by the model. The initial predictor and initial prediction model in this method's digital human video generation model are both built on the SANA framework. Leveraging the SANA framework, the generated digital human video can be more stable and detailed, while reducing the computational resources required for generation, thus increasing the generation speed. This ensures the method can run in real-time on consumer-grade hardware, thereby reducing deployment costs and application barriers, and expanding application scenarios. Furthermore, the style predictor can predict the target style code from the reference audio, enabling the generation of diverse emotional expressions, thereby generating digital human videos with rich emotions.

[0047] Based on the above embodiments, the style predictor is specifically used for: Based on the image of the person, the target style code is predicted from the reference audio.

[0048] Specifically, research has found that style codes in audio are related to speaker identity. Therefore, target style codes can be predicted from reference audio using person images. For example, 3DMM identity parameters can be extracted from person images, and then these 3DMM identity parameters can be used as input to a style predictor. In other words, the 3DMM identity parameters of the person image, the noise style code, and the reference audio are all input into the style predictor to enhance the accuracy of the style predictor in predicting the target style code.

[0049] Based on the above embodiments, the style predictor is further specifically used for: The target style code is determined based on the first audio feature, the second audio feature, and the image of the person. The first audio feature is extracted from the reference audio based on a speech pre-trained model; The second audio feature is extracted from the reference audio based on a speech emotion base model.

[0050] Specifically, in the embodiments of the present invention, such as Figure 3 As shown, the input to the style predictor can include the concatenation result of the first audio feature and the second audio feature, the 3DMM identity parameters of the person image, and the noise style code, and the output is the target style code.

[0051] It is understandable that the first audio feature can be obtained and output by the speech pre-training model after inputting the reference audio into the speech pre-training model, and the second audio feature can be obtained and output by the speech emotion base model after inputting the reference audio into the speech emotion base model.

[0052] In this embodiment of the invention, the target style code is determined by the first audio feature, the second audio feature, and the image of the person, which can reduce the data processing volume of the style predictor.

[0053] Based on the above embodiments, the facial motion prediction model includes an audio encoder and a linear DiT module; The audio encoder is used to extract the audio encoding of the reference audio; The linear DiT module is used to obtain the facial motion parameters based on the motion noise, the audio encoding, and the target style code.

[0054] Specifically, the facial motion prediction model used in this embodiment of the invention may include an audio encoder and a linear DiT (Diffusion Transformer) module. The audio encoder can extract the audio code of the reference audio; that is, by inputting the reference audio into the audio encoder, the audio code output by the audio encoder can be obtained. The linear DiT module can use motion noise, audio code, and target style code to obtain facial motion parameters; that is, by inputting motion noise, audio code, and target style code into the linear DiT module, the facial motion parameters output by the linear DiT module can be obtained.

[0055] Understandably, the linear DiT module is a core component of the flow matching model in the SANA framework. It effectively reduces computational complexity by replacing the traditional self-attention mechanism with linear attention. The DiT module is a module combining a diffusion model and the Transformer architecture, primarily used for generation tasks. It utilizes the Transformer's self-attention mechanism to globally model the data, capturing complex dependencies. Within the diffusion model framework, the DiT module generates high-quality samples through a progressive denoising process. Traditional DiT modules use the conventional self-attention mechanism. In self-attention, computational complexity increases quadratically with sequence length, leading to significant computational overhead when processing high-resolution images. The linear DiT module, based on linear attention, reduces computational complexity to a linear level through clever mathematical transformations, enabling the model to process long sequence data more efficiently.

[0056] In this embodiment of the invention, the SANA framework can achieve efficient and high-quality digital human video generation through the linear DiT module, providing a solid technical foundation for digital human video generation tasks.

[0057] Based on the above embodiments, the renderer is specifically used for: Extract the spatial features and latent feature representations of the image of the person; Based on the facial motion parameters, the spatial features are deformed to obtain deformed features; The deformed features are fused with the latent feature representation to form an enhanced feature representation; The enhanced feature representation is input into the temporal attention module to obtain the temporal attention feature output by the temporal attention module; The temporal attention features are decoded to gradually reconstruct each video frame in the digital human video.

[0058] Specifically, the renderer used in this embodiment of the invention can be a Motion-I2V framework to achieve consistent and controllable image-to-video (I2V) generation. In the process of predicting from image to video, motion-enhanced temporal attention is introduced: a motion-enhanced temporal attention module is proposed to enhance 1-D temporal attention in the video latent diffusion model. The renderer expands the temporal receptive field and improves video consistency by distorting the spatial features of the person image according to the predicted motion trajectory and adaptively injecting them into the synthesized frame at multiple scales.

[0059] The process of a renderer generating a digital human video can include: A) Feature extraction and encoding: The image of the person is encoded into a latent feature representation by an encoder, and the spatial features of the image of the person are extracted for subsequent processing.

[0060] B) Feature Warping: Based on facial motion parameters, the spatial features of the image are warped to obtain warped features. This operation aims to propagate the spatial features of the image to subsequent frames, thereby maintaining temporal consistency. Specifically, for each pixel, its position in subsequent frames is calculated based on facial motion parameters, and the corresponding spatial features are assigned to these positions.

[0061] C) Feature Enhancement and Fusion: The deformed features are fused with the latent feature representation of the person's image to form an enhanced feature representation. Here, a cross-attention mechanism can be used to interact with the latent feature representation of the person's image, using the deformed features as keys and values, thereby achieving effective utilization of facial motion parameters.

[0062] D) Temporal Attention Enhancement: The enhanced feature representation is input into the temporal attention module, which captures dependencies along the temporal dimension, resulting in the temporal attention features output by the module. This temporal attention module, through a self-attention mechanism, can focus on important connections between different frames, further enhancing temporal consistency. Furthermore, to perceive temporal order, sinusoidal positional encoding can be added to the temporal attention features to distinguish features at different time steps, improving the accuracy of temporal modeling.

[0063] E) Feature Decoding and Reconstruction: Decode the temporal attention features to gradually reconstruct each video frame in the digital human video. During the decoding process, the features are gradually refined to generate high-quality video frames.

[0064] In this embodiment of the invention, a renderer is used to reconstruct the digital human video, which can further improve the generation efficiency and quality of the digital human video.

[0065] Based on the above embodiments, such as Figure 4 As shown, the initial prediction model includes an initial audio encoder, an initial style encoder, and an initial linear DiT module, and the number of each of the initial audio encoder, initial style encoder, and initial linear DiT module can be n.

[0066] The facial motion prediction model is trained on the initial prediction model based on the following steps: The second audio sample is input into the initial audio encoder to obtain the audio encoded sample output by the initial audio encoder; The style reference video is input into the initial style encoder to obtain the reference style code output by the initial style encoder; The audio encoded sample, the reference style code, and the noise sample are input into the initial linear DiT module to obtain the predicted facial motion parameters output by the initial linear DiT module. Based on the predicted facial motion parameters and the facial motion parameter labels, the initial prediction model is iteratively trained to obtain the trained prediction model, and the style encoder in the trained prediction model is removed to obtain the facial motion prediction model.

[0067] Specifically, the initial prediction model includes an initial audio encoder, an initial style encoder, and an initial linear DiT module, while the facial motion prediction model may only include the audio encoder and linear DiT module from the trained prediction model.

[0068] When training the initial prediction model, the second audio sample can be segmented into frames to align with the style reference video. The prediction process of the initial prediction model can be improved by adding a window [tw, t+w] to consider the preceding and following audio content, thereby increasing the coherence of the prediction. Here, t is the time step, and w is the window size.

[0069] After alignment, the second audio sample is input to the initial audio encoder, which then outputs the encoded audio sample.

[0070] Input the style reference video into the initial style encoder, and the initial style encoder will output the reference style code.

[0071] The audio encoded sample, reference style code, and noise sample are input into the initial linear DiT module, which outputs the predicted facial motion parameters.

[0072] Using predicted facial motion parameters and their labels, a third training loss is calculated. This third training loss is then used to iteratively train the initial prediction model. When the third training loss converges or reaches a third preset number of iterations, the trained prediction model is obtained. The third preset number of iterations can be set as needed and is not specifically limited here.

[0073] After obtaining the trained prediction model, the style encoder is removed from it to obtain the facial motion prediction model. The style encoder in the trained prediction model can then be used to extract video style codes from video samples.

[0074] In this embodiment of the invention, the initial audio encoder, the initial style encoder, and the initial linear DiT module are jointly trained to obtain a style encoder for extracting video style codes and to build a facial motion prediction model, thereby improving the overall training efficiency of the digital human video generation model.

[0075] Based on the above embodiments, the digital human video generation model is deployed on a cloud server, or the digital human video generation model is deployed on an edge device after optimization and cropping; The step of inputting the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model includes: Based on the application programming interface, the model deployed on the cloud server or the edge device is called, and the reference audio and the image of the person are input into the called model to obtain the digital human video.

[0076] Specifically, in this embodiment of the invention, the digital human video generation model can be deployed on a cloud server, or it can be optimized and trimmed before being deployed on an edge device. This edge device can be a mobile device, an embedded system, etc., thus enabling the digital human video generation model to be applied to application scenarios with high real-time requirements.

[0077] Furthermore, when generating digital human videos, an Application Programming Interface (API) can be used to call models deployed on cloud servers or edge devices, that is, to optimize the digital human video generation model before or after cropping. By inputting reference audio and a person's image into the called model, the digital human video output by the model can be obtained.

[0078] When optimizing and cropping digital human video generation models, quantization and compression can be performed to reduce computational resource consumption. Quantization converts high-precision parameters in the model into low-precision representations, reducing memory usage and computational load. Furthermore, code-level optimization improves operational efficiency, including but not limited to using efficient matrix operation libraries, optimizing memory management, and reducing unnecessary computations and data transfers.

[0079] In this embodiment of the invention, two deployment locations are provided for the model, which can further reduce costs and expand application scenarios. When the model is deployed on a cloud server, the local installation requirements can be reduced, avoiding the need to install complex software and hardware environments locally.

[0080] Based on the above embodiments, such as Figure 5 As shown, this embodiment of the invention provides a digital human video generation device, comprising: Module 51 is used to acquire reference audio and character images; The generation module 52 is used to input the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model; The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer; the style predictor is used to predict a target style code from the reference audio; the facial motion prediction model is used to apply motion noise and the reference audio, and generate facial motion parameters based on the target style code; the renderer is used to generate a digital human video based on the person image and the facial motion parameters. The style predictor is obtained by training an initial predictor based on noisy style code samples and audio and video style codes in video samples. The facial motion prediction model is obtained by training an initial prediction model based on a second audio sample, a style reference video, and facial motion parameter labels in the style reference video. Both the initial predictor and the initial prediction model are built on the SANA framework.

[0081] Based on the above embodiments, the style predictor is specifically used for: Based on the image of the person, the target style code is predicted from the reference audio.

[0082] Based on the above embodiments, the style predictor is further specifically used for: The target style code is determined based on the first audio feature, the second audio feature, and the image of the person. The first audio feature is extracted from the reference audio based on a speech pre-trained model; The second audio feature is extracted from the reference audio based on a speech emotion base model.

[0083] Based on the above embodiments, the facial motion prediction model includes an audio encoder and a linear DiT module; The audio encoder is used to extract the audio encoding of the reference audio; The linear DiT module is used to obtain the facial motion parameters based on the motion noise, the audio encoding, and the target style code.

[0084] Based on the above embodiments, the renderer is specifically used for: Extract the spatial features and latent feature representations of the image of the person; Based on the facial motion parameters, the spatial features are deformed to obtain deformed features; The deformed features are fused with the latent feature representation to form an enhanced feature representation; The enhanced feature representation is input into the temporal attention module to obtain the temporal attention feature output by the temporal attention module; The temporal attention features are decoded to gradually reconstruct each video frame in the digital human video.

[0085] Based on the above embodiments, the initial prediction model includes an initial audio encoder, an initial style encoder, and an initial linear DiT module; The facial motion prediction model is trained on the initial prediction model based on the following steps: The second audio sample is input into the initial audio encoder to obtain the audio encoded sample output by the initial audio encoder; The style reference video is input into the initial style encoder to obtain the reference style code output by the initial style encoder; The audio encoded sample, the reference style code, and the noise sample are input into the initial linear DiT module to obtain the predicted facial motion parameters output by the initial linear DiT module. Based on the predicted facial motion parameters and the facial motion parameter labels, the initial prediction model is iteratively trained to obtain the trained prediction model, and the style encoder in the trained prediction model is removed to obtain the facial motion prediction model.

[0086] Based on the above embodiments, the digital human video generation model is deployed on a cloud server, or the digital human video generation model is deployed on an edge device after optimization and cropping; The step of inputting the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model includes: Based on the application programming interface, the model deployed on the cloud server or the edge device is called, and the reference audio and the image of the person are input into the called model to obtain the digital human video.

[0087] Specifically, the functions of each module in the digital human video generation device provided in this embodiment correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment.

[0088] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the digital human video generation method provided in the above embodiments.

[0089] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the digital human video generation method provided in the above embodiments.

[0091] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the digital human video generation method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating digital human videos, characterized in that, include: Obtain reference audio and images of the individuals; The reference audio and the image of the person are input into the digital human video generation model to obtain the digital human video output by the digital human video generation model. The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer; the style predictor is used to predict a target style code from the reference audio; the facial motion prediction model is used to apply motion noise and the reference audio, and generate facial motion parameters based on the target style code; the renderer is used to generate a digital human video based on the person image and the facial motion parameters. The style predictor is obtained by training an initial predictor based on noisy style code samples and audio and video style codes in video samples. The facial motion prediction model is obtained by training an initial prediction model based on a second audio sample, a style reference video, and facial motion parameter labels in the style reference video. Both the initial predictor and the initial prediction model are built on the SANA framework.

2. The digital human video generation method according to claim 1, characterized in that, The style predictor is specifically used for: Based on the image of the person, the target style code is predicted from the reference audio.

3. The digital human video generation method according to claim 2, characterized in that, The style predictor is also specifically used for: The target style code is determined based on the first audio feature, the second audio feature, and the image of the person. The first audio feature is extracted from the reference audio based on a speech pre-trained model; The second audio feature is extracted from the reference audio based on a speech emotion base model.

4. The digital human video generation method according to claim 1, characterized in that, The facial motion prediction model includes an audio encoder and a linear DiT module; The audio encoder is used to extract the audio encoding of the reference audio; The linear DiT module is used to obtain the facial motion parameters based on the motion noise, the audio encoding, and the target style code.

5. The digital human video generation method according to any one of claims 1-4, characterized in that, The renderer is specifically used for: Extract the spatial features and latent feature representations of the image of the person; Based on the facial motion parameters, the spatial features are deformed to obtain deformed features; The deformed features are fused with the latent feature representation to form an enhanced feature representation; The enhanced feature representation is input into the temporal attention module to obtain the temporal attention feature output by the temporal attention module; The temporal attention features are decoded to gradually reconstruct each video frame in the digital human video.

6. The digital human video generation method according to any one of claims 1-4, characterized in that, The initial prediction model includes an initial audio encoder, an initial style encoder, and an initial linear DiT module; The facial motion prediction model is trained on the initial prediction model based on the following steps: The second audio sample is input into the initial audio encoder to obtain the audio encoded sample output by the initial audio encoder; The style reference video is input into the initial style encoder to obtain the reference style code output by the initial style encoder; The audio encoded sample, the reference style code, and the noise sample are input into the initial linear DiT module to obtain the predicted facial motion parameters output by the initial linear DiT module. Based on the predicted facial motion parameters and the facial motion parameter labels, the initial prediction model is iteratively trained to obtain the trained prediction model, and the style encoder in the trained prediction model is removed to obtain the facial motion prediction model.

7. The digital human video generation method according to any one of claims 1-4, characterized in that, The digital human video generation model is deployed on a cloud server, or the digital human video generation model is deployed on an edge device after optimization and cropping; The step of inputting the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model includes: Based on the application programming interface, the model deployed on the cloud server or the edge device is called, and the reference audio and the image of the person are input into the called model to obtain the digital human video.

8. A digital human video generation device, characterized in that, include: The acquisition module is used to acquire reference audio and images of people; The generation module is used to input the reference audio and the image of the person into the digital human video generation model to obtain the digital human video output by the digital human video generation model; The digital human video generation model includes a style predictor, a facial motion prediction model, and a renderer; the style predictor is used to predict a target style code from the reference audio; the facial motion prediction model is used to apply motion noise and the reference audio, and generate facial motion parameters based on the target style code; the renderer is used to generate a digital human video based on the person image and the facial motion parameters. The style predictor is obtained by training an initial predictor based on noisy style code samples and audio and video style codes in video samples. The facial motion prediction model is obtained by training an initial prediction model based on a second audio sample, a style reference video, and facial motion parameter labels in the style reference video. Both the initial predictor and the initial prediction model are built on the SANA framework.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the digital human video generation method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the digital human video generation method as described in any one of claims 1-7.

Images