Video generation method and device of digital human and electronic equipment

Abstract

The invention provides a video generation method and device of a digital human and electronic equipment, and relates to the technical fields of artificial intelligence, voice technologies, natural language processing and the like. Comprising the following steps: acquiring enhanced prompt information, and encoding the enhanced prompt information to obtain time sequence constraint information; acquiring a reference image sequence including a digital human image, and encoding the reference image sequence to obtain an identity style vector of the digital human; obtaining a time domain audio according to the enhancement prompt information and the reference image sequence; and generating a target digital human video according to the time sequence constraint information, the identity style vector and the time domain audio. According to the method and the device, coherent digital human actions, expressions and mouth shapes can be quickly generated according to voice, texts and instructions coming in real time, the actions and expressions of the digital human can change in real time along with line content, real dialogue interaction is realized, high-cost video shooting is not depended on, and deployment in a real-time scene can be realized. The method is suitable for the application fields of intelligent e-commerce, agents and the like.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of speech technology, natural language processing and the like. BACKGROUND

[0002] Digital human (Digital Human, Meta Human) is a digitalized figure close to human image created by using digital technology, covering the penetration of digital technology in various levels and stages of human physical, physiological and intelligent. Its core value lies in breaking the physical limit to provide personification services through hyper-realism, toolization and strong interaction, and the system framework is composed of five modules of figure image, voice generation, animation generation, audio and video synthesis display and interaction.

[0003] In the related art, digital human products are widely used in live streaming, online customer service, education and training, virtual hosting, brand operation and the like. However, the digital human generation scheme relying on pre-shooting or pre-generated materials is difficult to generate new visual content according to real-time user input. SUMMARY

[0004] The present disclosure provides a digital human video generation method and device and electronic equipment.

[0005] According to an aspect of the present disclosure, a digital human video generation method is provided, wherein the method comprises: obtaining enhancement prompt information and encoding the enhancement prompt information to obtain time sequence constraint information; obtaining a reference image sequence including a digital human image and encoding the reference image sequence to obtain an identity style vector of the digital human; obtaining time domain audio according to the enhancement prompt information and the reference image sequence; generating a target digital human video according to the time sequence constraint information, the identity style vector and the time domain audio.

[0006] According to another aspect of the present disclosure, a digital human video generation device is provided, wherein the device comprises: a first encoding module configured to obtain enhancement prompt information and encode the enhancement prompt information to obtain time sequence constraint information; a second encoding module configured to obtain a reference image sequence including a digital human image and encode the reference image sequence to obtain an identity style vector of the digital human; an obtaining module configured to obtain time domain audio according to the enhancement prompt information and the reference image sequence; a generation module configured to generate a target digital human video according to the time sequence constraint information, the identity style vector and the time domain audio.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the digital human video generation method described above.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the digital human video generation method described above.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the digital human video generation method described above.

[0010] The present disclosure can realize the generation of coherent digital human actions, expressions and lip movements according to real-time incoming voice, text and instructions, wherein the actions and expressions of the digital human can change in real time with the content of the script, realizing real conversational interaction. The digital human video generated by the present disclosure does not rely on high-cost video shooting and can be deployed in real-time scenarios. In live streaming, customer service, virtual tour and other business scenarios, the digital human video can express, act and feedback in real time according to user questions.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them: Figure 1 is a schematic diagram of an exemplary embodiment of a digital human video generation method shown by the present disclosure; Figure 2 is a schematic diagram of an exemplary embodiment of a digital human video generation method shown by the present disclosure; Figure 3 is a schematic diagram of an exemplary embodiment of a reference voice instance acquisition method shown by the present disclosure; Figure 4 is a schematic diagram of an exemplary embodiment of a first voice token sequence acquisition method shown by the present disclosure; Figure 5 is a schematic diagram of an exemplary embodiment of a second voice token sequence acquisition method shown by the present disclosure; Figure 6is a schematic diagram of an exemplary embodiment of a digital human video generation method shown by the present disclosure; Figure 7 is a schematic diagram of an exemplary embodiment of a digital human video generation method shown by the present disclosure; Figure 8 is a schematic diagram of an exemplary embodiment of a digital human video generation method shown by the present disclosure; Figure 9 is a structural schematic diagram of a DiT model shown by the present disclosure; Figure 10 is a schematic diagram of a video clip shown by the present disclosure; Figure 11 is a schematic diagram of an exemplary embodiment of a digital human video generation method shown by the present disclosure; Figure 12 is a schematic diagram of an exemplary embodiment of a target digital human generation model training process shown by the present disclosure; Figure 13 is a schematic diagram of a digital human video generation apparatus according to an exemplary embodiment of the present disclosure; Figure 14 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only. It will thus be recognized by those of ordinary skill that various changes and modifications can be made to the embodiments described and illustrated herein without departing from the scope and spirit of the present disclosure. Also, for the purpose of clarity and the brevity, the description below omits the description of well-known functions and structures.

[0014] Natural Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a scientific field that integrates linguistics, computer science, and mathematics. Natural language processing is mainly applied to machine translation, public opinion monitoring, automatic abstract, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, Chinese OCR, etc.

[0015] Artificial Intelligence (AI) is a key driving force behind the new round of technological revolution and industrial transformation. It is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. AI is an important component of the discipline of intelligence; it attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. AI is a very broad science, encompassing robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, and more.

[0016] Speech technology is a key human-computer interaction technology centered on Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). Speech recognition systems convert speech to text through audio acquisition, feature extraction, and deep learning models, with applications covering smart home control, in-vehicle systems, and speech translation. Speech synthesis systems generate natural-speech from text using text preprocessing, acoustic model training, and vocoders, and are applied in intelligent customer service, audiobooks, and assistive devices for the visually impaired. Both technologies rely on deep learning algorithms for optimization and face technical challenges such as pronunciation differences and environmental noise.

[0017] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0018] Figure 1 This is a schematic diagram of an exemplary embodiment of a video generation method for digital humans disclosed herein, such as... Figure 1 As shown, the video generation method for this digital human includes the following steps: S101, obtain the enhanced prompt information and encode the enhanced prompt information to obtain the timing constraint information.

[0019] In some implementations, enhanced prompts include descriptions of facial expressions and actions, or instructions.

[0020] In some implementations, at least one of the following enhancement information is acquired: emotional tendency, gaze command, and action cue; and enhanced cue information is generated based on the enhancement information and the original cue information.

[0021] In this embodiment of the disclosure, the original prompt information may be the prompt instructions used when generating the digital human's video. This prompt information may be basic information provided by the user to guide the generation and interaction of the digital human's video. The original prompt information may include, for example, a text prompt of the target lines corresponding to the digital human. In other embodiments, it may also include other prompt information such as the language style and language type used by the digital human. This disclosure does not limit this.

[0022] In this embodiment of the disclosure, the enhanced prompt information is encoded according to the text encoder UMT5 to obtain the timing constraint information, which can be understood as the fusionable timing conditions.

[0023] In this embodiment of the disclosure, instruction compliance and dialogue state control are achieved through timing constraint information to obtain immediate action responses. For example, explicit actions in enhanced prompts (waving, liking, wiping sweat, making a heart shape, etc.) and image description text captions in enhanced videos are used to achieve the effect of timing-aligned modeling of facial expressions and emotions. Furthermore, rhythm and lip-sync constraints are applied between actions and speech to ensure that actions and speech rhythms are "matched," so that actions do not overshadow or interrupt speech, appearing coherent and natural.

[0024] The caption (image description text) is a natural language description of the corresponding image content generated by a vision-language model.

[0025] The interaction between dialogue semantics and emotional inclination drives the subtle movements of the digital human's eyebrows, eyes, head, and upper body, avoiding a stiff performance and achieving fine-grained performance driven by semantics.

[0026] S102, obtain a reference image sequence including the digital human image, and encode the reference image sequence to obtain the digital human's identity style vector.

[0027] In some implementations, the first frame of the reference image sequence or the starting frame of the continuation is extracted into an identity style vector by a 3D variational autoencoder (3D VAE Encoder) to ensure the stability of appearance and color across segments.

[0028] Among them, the 3D VAE Encoder is an encoder module used to compress multi-view renderings of 3D objects (corresponding to the reference image sequence of this disclosure) into 3D latent space.

[0029] S103, acquire temporal audio based on enhanced prompt information and reference image sequence.

[0030] Based on the enhanced prompts and reference image sequences, information extraction and timbre consistency comparison are performed to obtain the voice instance that best matches the digital human image in terms of timbre and emotion. Then, based on the voice instance, the temporal audio corresponding to the target lines is generated.

[0031] S104: Generate the target digital human video based on temporal constraint information, identity style vector, and temporal audio.

[0032] In this embodiment of the disclosure, the overall architecture of the real-time digital human voice-driven system uses the speech signal of the temporal domain audio as the temporal dominant signal, which runs through the generation process of the target digital human video. In the implementation, temporal constraint information, identity style vector and temporal domain audio are simultaneously fused in the same spatiotemporal latent space to generate the target digital human video, realizing the integrated generation of lip movements, facial expressions and upper body movements, and maintaining natural alignment between sound and image.

[0033] In this embodiment, enhanced prompts are encoded to obtain temporal constraint information, and a reference image sequence is encoded to obtain a digital human identity style vector. Temporal audio is obtained based on the enhanced prompts and the reference image sequence. A target digital human video is generated based on the temporal constraint information, the identity style vector, and the temporal audio. This allows for the rapid generation of coherent digital human actions, expressions, and lip movements based on real-time incoming voice, text, and commands. The digital human's actions and expressions can change in real-time according to the dialogue content, achieving true conversational interaction. The digital human video generated by this disclosure does not rely on high-cost video shooting and can be deployed in real-time scenarios. In business scenarios such as live streaming, customer service, and virtual tours, the digital human video can express, act, and respond in real-time based on user questions.

[0034] Figure 2 This is a schematic diagram of an exemplary embodiment of a video generation method for digital humans disclosed herein, such as... Figure 2 As shown, the video generation method for this digital human includes the following steps: S201, Obtain enhanced prompt information and encode the enhanced prompt information to obtain timing constraint information.

[0035] S202, obtain a reference image sequence including the digital human image, and encode the reference image sequence to obtain the digital human's identity style vector.

[0036] For a description of steps S201 to S202, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0037] S203, based on the reference image sequence, search in the preset timbre vector library to obtain candidate timbres that match the digital human image.

[0038] In some implementations, image features are extracted from the reference image sequence to obtain global image features of the reference image sequence. Based on the global image features, a nearest neighbor search is performed in a preset timbre vector library to obtain candidate timbres similar to the digital human image.

[0039] Optionally, the global image features of the reference image sequence may be, for example, global image features that include the subject and its environment.

[0040] Optionally, the timbre vector library contains a variety of timbres of different ages, styles, tones, and genders. In order to improve the consistency between the digital human image and the timbre, in this embodiment of the disclosure, after extracting the global image features of the reference image sequence, a nearest neighbor search is performed in the pre-built timbre vector library to recall several candidate timbres in the timbre vector library that are similar to the digital human image.

[0041] Nearest neighbor retrieval, also known as Approximate nearest neighbor (ANN) retrieval, essentially involves quickly finding the nearest points of a target point among many dense vectors and considering them as the most similar nodes. It is mainly used for image retrieval and high-dimensional retrieval.

[0042] S204. Based on the enhanced prompts and candidate timbres, obtain the reference voice instance that best matches the digital human image in terms of timbre and emotion.

[0043] In some implementations, global image features and local human features of the reference image sequence, as well as text features corresponding to the enhanced prompt information, are fused to obtain image fusion features; candidate speech instances corresponding to candidate timbres are obtained, and audio features, audio emotion vectors, and speaker embedding vectors of the candidate speech instances are fused to obtain audio fusion features; the candidate speech instances are sorted according to the image fusion features and audio fusion features to obtain reference speech instances.

[0044] The Rank model is a machine learning model used for ranking or re-ranking tasks. It is commonly used in information retrieval, natural language processing and other fields. Its main function is to rank the results based on the relevance of the query to the document.

[0045] In this embodiment, a Rank model can be invoked to rank candidate speech instances based on enhanced prompts and candidate timbres. From these candidate speech instances, the reference speech instance that best matches the digital human character in terms of timbre and emotion is obtained. It should be noted that the reference speech instance here is a matched speech instance with a timbre and emotion that is similar to the digital human, but the reference lines and target lines corresponding to the reference speech instance may not be the same.

[0046] The local human body features in this embodiment of the present disclosure may be, for example, local human body features extracted from the first frame image in the reference image sequence, including the movement and posture features of various local key points of the human body (such as the head, shoulders, elbows, knees, etc.).

[0047] For example, such as Figure 3As shown, in the ranking stage, the Rank model adopts a dual-tower structure. The left tower receives and concatenates global image features, local human features, and text features corresponding to enhanced prompts, and obtains image fusion features through a projection layer. The right tower receives audio features, audio emotion vectors, and speaker embedding vectors, and obtains audio fusion features through a projection layer. Based on the image fusion features and audio fusion features, contrastive learning is performed, and candidate speech instances are ranked to obtain the candidate speech instance with the highest number as the reference speech instance.

[0048] S205, Generate time-domain audio based on the reference speech instance.

[0049] In some embodiments, the reference audio of a reference speech instance and the corresponding reference dialogue are fed into a text-to-speech (TTS) cloning module to generate temporal audio consistent with the digital human's image, semantics, timbre, and emotion based on the target dialogue. The TTS voice cloning module is a system that converts text into speech using deep learning technology, and it has the ability to clone and simulate specific voices.

[0050] In some implementations, the reference speech instance is discretized to obtain a first speech semantic unit token sequence; the reference text corresponding to the reference speech instance is obtained, and a first text token sequence of the reference text is determined; the target line in the enhanced prompt information is obtained, and a second text token sequence of the target line is determined; based on the first speech token sequence, the first text token sequence, and the second text token sequence, a second speech token sequence corresponding to the target line is predicted; and time-domain audio is generated based on the second speech token sequence and the reference speech instance.

[0051] In this context, Token represents the semantic units into which text or speech is segmented in natural language processing, serving as the basic elements for model processing.

[0052] Optionally, time-domain audio is generated based on the second speech token sequence and the reference speech instance, including: obtaining the timbre features corresponding to the reference speech instance; using the timbre features, the second speech token sequence, and the generation progress as constraints, performing stream matching iteration on the masked Mel spectrum to obtain the target Mel spectrum; and processing the target Mel spectrum and the second speech token sequence based on the vocoder to generate time-domain audio.

[0053] For example, such as Figure 4 As shown, the Audio Tokenizer is first invoked to discretize the reference audio of the reference speech instance into a first speech token sequence; as... Figure 5As shown, the LLM (Large Language Model) is then invoked to generate a large language model based on the first voice token sequence, the first text token sequence, and the second text token sequence. This model predicts and generates the second voice token sequence corresponding to the target dialogue. Figure 6 As shown, the final stage is the conditional flow matching decoding stage. The second speech token sequence, global speaker embedding (timbre features), and the masked Mel spectrum are used to complete the denoising and reconstruction under the time step drive to obtain a new Mel spectrum. This new Mel spectrum is then synthesized into temporal audio by a vocoder, thereby outputting temporal audio that matches the target dialogue pronunciation and is close to the reference audio in timbre and emotion.

[0054] Among them, the tokenizer is a tool in Natural Language Processing (NLP) used to break down a piece of text into smaller units. Flow Matching is a generative modeling technique that designs generation trajectories by directly learning the velocity field, achieving a continuously controllable generation process and overcoming the limitations of traditional diffusion models. A vocoder is a synthesis system that converts speech feature parameters (such as Mel-frequency spectra) into waveform signals. It utilizes only model parameters during transmission, and combines model parameter estimation and speech synthesis techniques during encoding and decoding.

[0055] S206. Generate the target digital human video based on temporal constraint information, identity style vector, and temporal audio.

[0056] For a description of step S206, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0057] This disclosure uses a reference image sequence to search a pre-defined timbre vector library to obtain candidate timbres that match the digital human image. Based on enhanced prompts and candidate timbres, it obtains the reference speech instance that best matches the digital human image in terms of timbre and emotion. Based on the reference speech instance, it generates temporal audio, which can generate the audio corresponding to the target lines in real time with a fast response, resulting in a voice that not only matches the pronunciation of the lines but also closely resembles the reference audio in timbre and emotion. This supports the synchronous change of the digital human image's speech content and enables real-time expression feedback.

[0058] Figure 7 This is a schematic diagram of an exemplary embodiment of a video generation method for digital humans disclosed herein, such as... Figure 7 As shown, the video generation method for this digital human includes the following steps: S701, obtain enhanced prompt information.

[0059] For a description of step S701, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0060] S702, based on the enhanced prompt information, determine the subtitles or script corresponding to the target digital human video.

[0061] For example, the subtitles corresponding to a target digital human video can be determined based on the target lines in the enhanced prompts.

[0062] For example, the script corresponding to the target digital human video can be determined based on at least one of the enhanced information, such as emotional tendency, eye gaze instructions, and action cues in the enhanced prompts, like "smile and talk".

[0063] S703, based on subtitles or scripts, performs time-series alignment of the digital human's actions, expressions, and emotions to obtain video constraint information.

[0064] In this embodiment, using subtitles or scripts as a baseline, the actions, expressions, and emotions of the digital human are time-aligned. The start and end points of actions, expression intensity curves, gaze direction, stressed words, and pause positions are automatically supplemented and aligned to form multi-layered captions with a timeline (word-level, phoneme-level, and action / expression keyframe-level), serving as video constraint information. For example, specifying the second of speech, such as looking to the right and smiling.

[0065] S704, based on the subtitles or script, determines the prosodic information of the temporal audio and constrains it to obtain audio constraint information.

[0066] The audio constraint information disclosed herein includes at least one of the following: silent intervals, pause locations, and / or weak aerophone subsets of time-domain audio; and posture information of the relevant organs corresponding to the silent intervals, pause locations, and weak aerophone subsets.

[0067] In some implementations, the prosodic information of the temporal audio is determined and constrained based on the subtitles or script to achieve silent state optimization, making the digital human image more natural in silent state. The prosodic information is used to indicate the activities in silent state, including fine-grained descriptions of silent intervals, such as looking forward, slightly moving the shoulders, blinking, etc.

[0068] S705: Based on the video constraint information and audio constraint information, the target prompt information is obtained.

[0069] To improve the effectiveness of instruction compliance, the original sparse subtitles or script descriptions are expanded into dense time-series annotations based on video and audio constraint information to obtain target prompt information.

[0070] S706 encodes the target prompt information to obtain timing constraint information.

[0071] The text encoder UMT5 is invoked to encode the target prompt information and obtain the timing constraint information.

[0072] S707, acquire a reference image sequence including the digital human image, and encode the reference image sequence to obtain the digital human's identity style vector.

[0073] S708 acquires temporal audio based on enhanced prompt information and a reference image sequence.

[0074] S709 generates a target digital human video based on temporal constraint information, identity style vector, and temporal audio.

[0075] For a description of steps S707 to S709, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0076] In this embodiment, enhancements can be made through data, conditions, and generation behavior to construct silent intervals, pause positions, and / or weak aerobic sound subsets. Fine-grained pose information of the corresponding organs, such as upper limit constraints on mouth closure, lip and tongue amplitude, facial expression change rate, and head and shoulder angular velocity, is annotated in dense temporal captions to clarify silent behavior specifications. This invention, by strengthening the injection conditions of the audio modality and refining the granularity of the caption description in datasets containing paused speech, ensures that the lips remain still and exhibit natural micro-expressions during silent periods, achieving controllable silence / speaking and resulting in consistent and vivid digital human image generation.

[0077] Figure 8 This is a schematic diagram of an exemplary embodiment of a video generation method for digital humans disclosed herein, such as... Figure 8 As shown, the video generation method for this digital human includes the following steps: S801, acquire enhanced prompt information, encode the enhanced prompt information, and obtain timing constraint information.

[0078] S802, acquire a reference image sequence including the digital human image, and encode the reference image sequence to obtain the digital human's identity style vector.

[0079] S803 acquires temporal audio based on enhanced prompt information and a reference image sequence.

[0080] For a description of steps S801 to S803, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0081] S804 inputs temporal constraint information, identity style vector, and temporal audio into the target digital human generation model.

[0082] In this embodiment of the disclosure, the target digital human generation model can be a diffusion model, such as the DiT (DiffusionTransformer) model. The DiT model is a diffusion model based on the Transformer architecture with a self-attention mechanism, mainly used for high-quality image and video generation.

[0083] S805 uses a target digital human generation model to perform video diffusion based on temporal constraint information, identity style vectors, and temporal audio to obtain the target digital human video.

[0084] Temporal audio is extracted from the speech side and injected into the DiT model as an acoustic embedding. Temporal audio serves as the main driving cue for lip movements, rhythm, and emotion.

[0085] like Figure 9 As shown, the DiT model includes a feedforward neural network (FNN), a cross-attention layer, and a self-attention layer. The DiT model jointly models temporal constraint information, identity style vectors, and temporal audio within a 3D spatiotemporal latent space, progressively denoising to generate latent video. This is then reconstructed frame-by-frame by a decoder (3D VAE Decoder) into the target digital human video. Finally, temporal audio is directly synthesized with the video for playback, ensuring strict audio-visual alignment throughout. The 3D VAE Decoder's role is to decode the latent vectors generated by the DiT model into video frames.

[0086] Introducing fine-grained labels into the temporal constraint information helps DiT learn the joint distribution of speech-semantics-actions, making lip movements, facial expressions and body movements more coordinated and natural, thus enhancing the vividness of the digital human image.

[0087] 3D spatiotemporal latent can reduce computational load, thereby supporting streaming inference and fragmented continuation to ensure stable output for long-term services such as live streaming and customer service.

[0088] In this embodiment, lip movements, facial expressions, and upper body movements are jointly generated within the same latent space, achieving temporal and lighting consistency between the entire face and upper body. To further enhance controllability, on the one hand, sparse captions are expanded into dense temporal captions, refined to the strength range of words / pauses / movements and facial expressions, serving as steady-state timeline constraints; on the other hand, prompt design and parsing are enhanced, explicitly incorporating emotional tendencies, gaze instructions, and action cues, and weighting the conditional channels to enable the model to generate coordinated lip-sync, facial expression, and posture linkage at emphasized words, transitions, and pauses. This strategy fundamentally eliminates the patch boundary and temporal misalignment problems caused by "lip-syncing," resulting in a more natural and coherent generation effect.

[0089] In some implementations, during the generation of the target digital human video, in response to the current generated video length exceeding a set length, the currently generated video is segmented to obtain a video segment; such as... Figure 10 As shown, the last few frames of a video segment are determined as reference images for continuing the next video segment. The reference images for continuing the next video segment are used as the starting constraints for the next video segment, and the next video segment is generated.

[0090] In this embodiment, a two-layer mechanism of reference frame anchoring and continuation frame alignment ensures image consistency. When the length of the currently generated video is less than or equal to a set length, the image embedded from the first frame (reference frame) is used as a global identity / style condition and continuously injected throughout the generation process to lock facial shape, texture, and color distribution. When the length of the currently generated video is greater than the set length, the currently generated video is segmented. When a long sequence of videos is segmented into multiple video chunks, the last few segments of the previous video chunk are used as reference images for the continuation of the next video chunk to drive the initial state and motion inertia of the video chunk. The appearance, lighting, and camera pose at the boundary are aligned. The reference image for the continuation of the next video chunk is used as the starting constraint for the next video chunk, and the next video chunk is generated. This maintains the continuity of appearance and timing between segments and avoids "face-changing color distortion" and detail drift.

[0091] A chunk refers to a fixed-size processing unit into which a large dataset or continuous stream is divided, for use in parallel computing or batch loading.

[0092] In this disclosure, the first frame reference uses 3DVAE latent variables as an identity anchor point. Combined with cross-fragment reference frame synchronization, color consistency regularization, and identity contrast loss, it prevents positive distortion, streaking, and color cast, achieving stability in identity and chromaticity. The latent variables of the last frame of the previous segment serve as the initial state of the next segment, ensuring the continuity of appearance and lighting over long periods, and enabling seamless segment continuation.

[0093] like Figure 11As shown in this embodiment, an end-to-end pipeline of "text encoder (UMT5), 3D variational autoencoder 3DVAE, and DiT" is used to encode the enhanced prompt information to obtain temporal constraint information and to encode the reference image sequence to obtain the digital human's identity style vector. Based on the enhanced prompt information and the reference image sequence, temporal audio is obtained. In this embodiment, the temporal audio is processed through an adapter, which allows DiT to adapt to the structure of the audio modality. The DiT model (i.e., the target digital human generation model) jointly models the temporal constraint information, identity style vector, and temporal audio in the 3D spatiotemporal latent space and gradually denoises to generate a latent video. Subsequently, the decoder (3D VAE Decoder) corresponding to the 3D variational autoencoder restores the target digital human video frame by frame. Finally, the temporal audio is directly synthesized and played with the video to ensure strict audio-visual alignment throughout the process.

[0094] This disclosure enables the construction of a real-time generation system capable of millisecond-level response by end-to-end modeling the mapping between speech, semantics, intent, and action flow. This system allows for the synchronous changes in the actions, lip movements, expressions, and speech content of a digital human avatar, meeting the needs of various scenarios such as live streaming, interactive Q&A, and intelligent agent display.

[0095] Figure 12 This is a schematic diagram illustrating an exemplary implementation of the training process for a target digital human generation model as disclosed in this disclosure, such as... Figure 12 As shown, the training process of the target digital human generation model includes: S1201, Obtain sample video and determine video description information based on sample video.

[0096] In some implementations, computer vision and natural language processing technologies are used to automatically analyze the video content of sample videos and generate video description information.

[0097] S1202, determine the sample reference image and sample audio based on the sample video.

[0098] In some implementations, the sample video is parsed and information is extracted to obtain a sample reference image and sample audio.

[0099] S1203, using video description information, sample reference images and sample audio as training samples, the first teacher model is trained based on the training samples to obtain the second teacher model, wherein the teacher model is used to predict digital human videos.

[0100] In some implementations, video description information, sample reference images, and sample audio are used as training samples and input into the first teacher model. Based on the same concept as the video generation method for digital humans provided in this disclosure, the digital human video predicted by the first teacher model is obtained. Then, the first teacher model is adjusted in reverse according to the predicted digital human video to achieve training of the first teacher model.

[0101] S1204, Distill the second teacher model based on dynamic pattern decomposition to obtain the target digital human generation model.

[0102] In some implementations, the second teacher model is distilled based on dynamic pattern decomposition to compress its denoised trajectory, resulting in a student model that includes only the key prediction steps of the second teacher model. The student model is then trained using training samples, regressing the label information of the teacher model at the key prediction steps during training to adjust the student model's parameters until training is complete and the target digital human generation model is obtained.

[0103] The student model is trained through several steps, including: training the student model based on trajectory generation using training samples. During trajectory generation training, the denoised trajectory of the second teacher model is simulated using a sliding window continuation method. The actual denoised trajectory of the second teacher model is determined as the supervision information for the student model. Based on the supervision information and the simulated denoised trajectory, the model parameters of the student model are adjusted until training is complete, resulting in the target digital human generation model.

[0104] Optionally, a contrastive learning loss is used during training to make matching image-audio pairs closer in the embedding space and mismatched pairs farther apart, thereby effectively scoring and ranking the retrieved candidate timbres during inference and selecting the sample whose timbre and emotion best match the semantics of the current image and prompt.

[0105] Optionally, during training, the model uses the dense caption as a condition to impose time alignment constraints on attention, so that instructions such as "look to the right - smile - emphasize a word - pause briefly" are stably triggered within the corresponding time window, achieving synchronous execution of text, speech and video.

[0106] In this embodiment of the disclosure, the training target, combined with the classifier-free guidance of the audio channel, increases the dependence on silent conditions at the training end and suppresses irregular mouth movements. This can significantly improve the lip closure rate and cross-frame consistency in silent intervals, eliminating the incongruity of "talking even in silence".

[0107] Classifier-Free Guidance is a generative control technique for diffusion models. By jointly training conditional and unconditional generative models, it adjusts the balance between the quality and diversity of the generated results during the inference phase through a guiding scale.

[0108] When using diffusion backbones such as DiT as teacher models, conventional sampling often requires tens to hundreds of time steps, each triggering large-scale forward computation, resulting in end-to-end latency far exceeding the real-time threshold and uneconomical resource consumption. This disclosure employs DMD distillation technology to address the technical problems of excessive denoising steps, high latency, and high energy consumption in existing diffusion-based video generation during the inference stage. This enables high-speed generation for interactive and real-time scenarios without significantly sacrificing image quality and temporal consistency.

[0109] Specifically, this invention proposes to compress the multi-step denoising trajectory of the teacher model into a few steps or even a one-step approximation of the student model through DMD distillation, and to equivalently transform the original "long chain integral sampling process" into "few updates". Under the same hardware conditions, it significantly improves the frame generation speed and reduces the computational cost. After compressing the number of steps, the corresponding timeline will become shorter.

[0110] DMD stands for Dynamic Mode Decomposition, a data-driven fluid dynamics analysis method that extracts the dynamic characteristics and evolution patterns of flow structures from time-series observation data.

[0111] This disclosure proposes a "step-to-step" teacher-student matching method. Under full noise scheduling, the teacher model outputs conditional predictions (including but not limited to noise terms, velocity terms, or residual terms) at each sampling time step, forming a high-fidelity denoised trajectory and update direction. The student model makes predictions only at a small number of sampling nodes and learns denoised updates across large steps by regressing the teacher's soft labels at corresponding time points or after step mapping, thus achieving a generation effect similar to the teacher with fewer iterations. The student model can maintain isomorphism with the teacher to align the feature space, or combine lightweight structures and parameter efficiency techniques to further reduce computation. Through the above trajectory approximation and step update learning, the student model can reproduce the teacher's convergence endpoint or key intermediate state under conditions of fewer sampling steps, achieving generation efficiency with fewer steps.

[0112] In terms of time-based compression, this disclosure maps the multi-step timeline of the teacher model to a small number of "key steps" in the student model. Preferably, a nonlinear mapping encrypted at both high-noise and low-noise ends is used to reduce step errors while maintaining detail and structural restoration capabilities. For example, in experiments and implementations, when the student model has 2-4 steps, it achieves subjective quality and temporal stability close to that of the teacher model. Furthermore, in a single-step distillation configuration, the student directly learns the end-to-end mapping from noise to data to achieve extremely low latency. Although a slight decrease in image quality may occur in complex shots and scenes with rich details, it can meet the strict latency constraints of highly real-time interactive services. Based on the above design, this disclosure approximately linearly reduces the number of inference floating-point operations (FLOPs) per step under the same resolution and conditional input, significantly improving the end-to-end frame generation speed and fundamentally solving the speed bottleneck problem of diffusion-based video generation in real-time scenarios.

[0113] FLOPs represent the number of floating-point operations, a benchmark metric used to measure the computational efficiency of algorithms or hardware, reflecting the computational complexity required to process a task.

[0114] To address the issue of image and temporal decay caused by fewer sampling steps in DMD distillation, which speeds up the process, this invention introduces a policy evaluation roll-out mechanism during the distillation stage to achieve training-inference consistency. Specifically, after the student model completes its fewer-step distillation, it not only uses the teacher's offline trajectory as supervision but also continuously generates multiple time windows (autoregressive or sliding window continuation) based on its own predictions. These student-generated sequences are then subject to teacher supervision and consistency constraints (perceptual consistency, identity similarity, color / lighting matching, cross-frame motion smoothing, and region-weighted constraints). This "generate-re-supervised" closed loop... Loop training allows students to learn error correction and drift suppression capabilities within the same distribution as inference, significantly mitigating facial drift, loss of detail, and rhythm misalignment in long sequences. Through roll-out, the model becomes more robust to accumulated errors during actual inference, automatically correcting minor deviations in subsequent frames and maintaining the stability of character appearance, style, and temporal structure. This effectively suppresses image decay and improves long-term consistency while maintaining the real-time speed provided by DMD. Roll-out refers to a policy evaluation technique that simulates the multi-step execution trajectory of the current policy in the environment to predict long-term rewards and optimize decisions.

[0115] This disclosure achieves training-inference consistency through roll-out. During training, the student model makes predictions, and training is performed based on the predicted output. This simulates the errors that students may make during inference, allowing the model to learn to generate a high-quality digital human video based on poor input. During inference, the model continues writing using the last frame, reducing attenuation issues and improving the quality of the generated video. The training phase simulates inference continuation, using the generated last frame instead of a clean frame as the starting point, and superimposing segment consistency and drift penalties to eliminate long-term attenuation and "becoming blurrier as it's written".

[0116] DMD real-time distillation can compress multi-step diffusion into fewer-step inference, significantly reducing end-to-end latency while maintaining image quality and lip-sync. It achieves frame-by-frame online generation with low latency, enabling long-term stable real-time video generation.

[0117] This disclosure applies to applications such as smart e-commerce and smart agents.

[0118] Figure 13 This is a schematic diagram of a video generation apparatus for a digital human according to an exemplary embodiment of the present disclosure, such as... Figure 13 As shown, the video generation device 1300 for the digital human includes: The first encoding module 1310 is used to acquire enhanced prompt information and encode the enhanced prompt information to obtain timing constraint information; The second encoding module 1320 is used to acquire a reference image sequence including the digital human image and encode the reference image sequence to obtain the digital human's identity style vector. The acquisition module 1330 is used to acquire temporal audio based on the enhanced prompt information and the reference image sequence; The generation module 1340 is used to generate a target digital human video based on temporal constraint information, identity style vector, and temporal audio.

[0119] In some implementations, the acquisition module 1330 is further configured to: Based on the reference image sequence, a search is performed in the preset timbre vector library to obtain candidate timbres that are compatible with the digital human image; Based on enhanced prompts and candidate timbres, obtain reference voice instances that best match the digital human image in terms of timbres and emotions; Generate time-domain audio based on a reference speech instance.

[0120] In some implementations, the acquisition module 1330 is further configured to: Image features are extracted from the reference image sequence to obtain the global image features of the reference image sequence; Based on global image features, a nearest neighbor search is performed in a pre-set timbre vector library to obtain candidate timbres similar to the digital human image.

[0121] In some implementations, the acquisition module 1330 is further configured to: The global image features and local human body features of the reference image sequence, as well as the text features corresponding to the enhanced prompt information, are fused to obtain image fusion features; Obtain candidate speech instances corresponding to candidate timbres, and fuse the audio features, audio emotion vectors, and speaker embedding vectors of the candidate speech instances to obtain audio fusion features; Candidate speech instances are sorted based on image fusion features and audio fusion features to obtain reference speech instances.

[0122] In some implementations, the acquisition module 1330 is further configured to: Discretize the reference speech instance to obtain the first speech semantic unit token sequence; Obtain the reference text corresponding to the reference speech instance, and determine the first text token sequence of the reference text; Obtain the target dialogue from the enhanced prompt information and determine the second text token sequence of the target dialogue; Based on the first voice token sequence, the first text token sequence, and the second text token sequence, predict the second voice token sequence corresponding to the target line; Time-domain audio is generated based on the second speech token sequence and the reference speech instance.

[0123] In some implementations, the acquisition module 1330 is further configured to: Obtain the timbre features corresponding to the reference speech instance; Using timbre features, second speech token sequence, and generation progress as constraints, the masked Mel spectrum is iteratively matched with the stream to obtain the target Mel spectrum; The target Mel spectrum and the second speech token sequence are processed by a vocoder to generate time-domain audio.

[0124] In some embodiments, the first encoding module 1310 is further configured to: Based on the enhanced prompts, determine the subtitles or script corresponding to the target digital human video; Based on subtitles or scripts, the actions, expressions, and emotions of the digital human are time-aligned to obtain video constraint information; Based on the subtitles or script, the prosodic information of the temporal audio is determined and constrained to obtain audio constraint information; Based on the video constraint information and audio constraint information, the target prompt information is obtained; The target prompt information is encoded to obtain the timing constraint information.

[0125] In some implementations, the audio constraint information includes at least one of the following: The silent intervals, pause locations, and / or weak aerobic subsets of temporal audio; The posture information of the relevant organs corresponding to the silent intervals, pause positions, and weak aerodynamic subsets.

[0126] In some embodiments, the first encoding module 1310 is further configured to: Obtain at least one piece of enhanced information from emotional tendency, eye gaze instructions, and motor cues; Enhanced prompts are generated based on the enhanced information and the original prompts.

[0127] In some implementations, the generation module 1340 is further configured to: If the length of the currently generated video exceeds the set length, the currently generated video is segmented to obtain a video clip; Identify the last few frames of a video segment as reference images for continuing the next video segment; Using the reference image for continuing the next video segment as the starting constraint, the next video segment is generated.

[0128] In some implementations, the generation module 1340 is further configured to: Input the temporal constraint information, identity style vector, and temporal audio into the target digital human generation model; By using the target digital human generation model, video diffusion is performed based on temporal constraint information, identity style vector, and temporal audio to obtain the target digital human video.

[0129] In some embodiments, the apparatus further includes a training module 1350 for: Acquire sample videos and determine video description information based on the sample videos; Based on the sample video, determine the sample reference image and sample audio; Using video description information, sample reference images, and sample audio as training samples, the first teacher model is trained based on the training samples to obtain the second teacher model, which is used to predict digital human videos. The second teacher model is distilled based on dynamic pattern decomposition to obtain the target digital human generation model.

[0130] In some implementations, the training module 1350 is also used for: Distillation based on dynamic mode decomposition is performed on the second teacher model to compress the denoised trajectory of the second teacher model, so as to obtain a student model that includes only the key prediction steps of the second teacher model. Based on the training samples, the student model is trained, and the label information of the teacher model at key prediction steps is regressed during the training process to adjust the model parameters of the student model until the training is completed and the target digital human generation model is obtained.

[0131] In some implementations, the training module 1350 is also used for: The student model is trained based on trajectory generation, using training samples. During the training process based on trajectory generation, the denoised trajectory of the second teacher model is simulated by sliding window continuation method; The actual denoised trajectory of the second teacher model is determined as the supervision information of the student model; The model parameters of the student model are adjusted based on supervised information and simulated denoised trajectories until the training ends, resulting in the target digital human generation model.

[0132] This disclosure enables the rapid generation of coherent digital human movements, expressions, and lip movements based on real-time voice, text, and commands. The digital human's movements and expressions can change in real-time according to the dialogue, achieving true conversational interaction. The digital human videos generated by this disclosure do not rely on high-cost video shooting and can be deployed in real-time scenarios. In business scenarios such as live streaming, customer service, and virtual tours, the digital human videos can express, act, and respond in real-time based on user questions.

[0133] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0134] Figure 14 A schematic block diagram of an example electronic device 1400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0135] like Figure 14 As shown, device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1402 or a computer program loaded from storage unit 1408 into random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0136] Multiple components in device 1400 are connected to I / O interface 1405, including: input unit 1406, such as a keyboard, mouse, etc.; output unit 1407, such as various types of displays, speakers, etc.; storage unit 1408, such as a disk, optical disk, etc.; and communication unit 1409, such as a network card, modem, wireless transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] The computing unit 1401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as the video generation method for digital humans. For example, in some embodiments, the video generation method for digital humans can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the video generation method for digital humans described above can be performed. Alternatively, in other embodiments, computing unit 1401 may be configured to perform a video generation method for digital humans by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0143] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a video of a digital human, wherein, The method includes: Obtain enhanced prompt information and encode the enhanced prompt information to obtain timing constraint information; A reference image sequence including a digital human image is obtained, and the reference image sequence is encoded to obtain the identity style vector of the digital human; Based on the enhanced prompt information and the reference image sequence, obtain the temporal audio; Based on the temporal constraint information, the identity style vector, and the temporal audio, a target digital human video is generated.

2. The method according to claim 1, wherein, The step of obtaining temporal audio based on the enhanced prompt information and the reference image sequence includes: Based on the reference image sequence, a search is performed in a preset timbre vector library to obtain candidate timbres that match the digital human image; Based on the enhanced prompt information and the candidate timbre, obtain the reference voice instance that best matches the digital human image in terms of timbre and emotion. The time-domain audio is generated based on the reference speech instance.

3. The method according to claim 2, wherein, The step of searching a preset timbre vector library based on the reference image to obtain candidate timbres that match the digital human image includes: Image features are extracted from the reference image sequence to obtain the global image features of the reference image sequence; Based on the global image features, a nearest neighbor search is performed in a preset timbre vector library to obtain candidate timbres similar to the digital human image.

4. The method according to claim 2, wherein, The step of obtaining the reference voice instance that best matches the digital human image in terms of timbre and emotion dimension based on the enhanced prompt information and the candidate timbre includes: The global image features and local human body features of the reference image sequence, as well as the text features corresponding to the enhanced prompt information, are fused to obtain image fusion features; Obtain candidate speech instances corresponding to the candidate timbre, and fuse the audio features, audio emotion vectors, and speaker embedding vectors of the candidate speech instances to obtain audio fusion features; The candidate speech instances are sorted according to the image fusion features and audio fusion features to obtain the reference speech instances.

5. The method according to claim 2, wherein, The step of generating the time-domain audio based on the reference speech instance includes: The reference speech instance is discretized to obtain the first speech semantic unit token sequence; Obtain the reference text corresponding to the reference speech instance, and determine the first text token sequence of the reference text; Obtain the target dialogue from the enhanced prompt information and determine the second text token sequence of the target dialogue; Based on the first voice token sequence, the first text token sequence, and the second text token sequence, predict the second voice token sequence corresponding to the target line; The time-domain audio is generated based on the second speech token sequence and the reference speech instance.

6. The method according to claim 5, wherein, The step of generating the time-domain audio based on the second speech token sequence and the reference speech instance includes: Obtain the timbre features corresponding to the reference speech instance; Using the timbre features, the second speech token sequence, and the generation progress as constraints, the masked Mel spectrum is iterated through flow matching to obtain the target Mel spectrum; The target Mel spectrum and the second speech token sequence are processed using a vocoder to generate the time-domain audio.

7. The method according to any one of claims 1-6, wherein, The process of encoding the enhanced prompt information to obtain timing constraint information includes: Based on the enhanced prompt information, determine the subtitles or script corresponding to the target digital human video; Based on the subtitles or script, the actions, expressions and emotions of the digital human are time-aligned to obtain video constraint information; Based on the subtitles or script, the prosodic information of the temporal audio is determined and constrained to obtain audio constraint information; Based on the video constraint information and the audio constraint information, target prompt information is obtained; The target prompt information is encoded to obtain the timing constraint information.

8. The method according to claim 7, wherein, The audio constraint information includes at least one of the following: The silent intervals, pause locations, and / or weak aerobic sound subsets of the time-domain audio; The posture information of the relevant organs corresponding to the silent interval, pause position, and weak aerodynamic subset.

9. The method according to claim 7, wherein, The method further includes: Obtain at least one piece of enhanced information from emotional tendency, eye gaze instructions, and motor cues; The enhanced prompt information is generated based on the enhanced information and the original prompt information.

10. The method according to any one of claims 1-6, wherein, In the process of generating the target digital human video, the method further includes: In response to the current generated video length being greater than a set length, the current generated video is segmented to obtain a video segment; The last few frames of the video segment are determined as reference images for continuing the next video segment; Using the reference image for continuing the next video segment as the starting constraint for the next video segment, the next video segment is generated.

11. The method according to any one of claims 1-6, wherein, The step of generating a target digital human video based on the temporal constraint information, the identity style vector, and the temporal audio includes: The temporal constraint information, the identity style vector, and the temporal audio are input into the target digital human generation model; The target digital human video is obtained by performing video diffusion based on the temporal constraint information, the identity style vector, and the temporal audio through the target digital human generation model.

12. The method according to claim 11, wherein, The training process of the target digital human generation model includes: Acquire sample videos and determine video description information based on the sample videos; Based on the sample video, a sample reference image and sample audio are determined; Using the video description information, the sample reference image, and the sample audio as training samples, the first teacher model is trained based on the training samples to obtain the second teacher model, wherein the teacher model is used to predict digital human videos; The second teacher model is distilled based on dynamic pattern decomposition to obtain the target digital human generation model.

13. The method according to claim 12, wherein, The distillation of the second teacher model based on dynamic pattern decomposition to obtain the target digital human generation model includes: The second teacher model is distilled based on dynamic mode decomposition to compress the denoised trajectory of the second teacher model, resulting in a student model that includes only the key prediction steps of the second teacher model. Based on the training samples, the student model is trained, and during the training process, the label information of the teacher model at the key prediction step is regressed to adjust the model parameters of the student model until the training ends and the target digital human generation model is obtained.

14. The method according to claim 13, wherein, The training of the student model includes: Based on the training samples, the student model is trained using trajectory generation. During the trajectory generation-based training process, the denoised trajectory of the second teacher model is simulated by a sliding window continuation method; The actual denoised trajectory of the second teacher model is determined as the supervision information of the student model; Based on the supervised information and the simulated denoised trajectory, the model parameters of the student model are adjusted until the training ends, thus obtaining the target digital human generation model.

15. A video generation device for a digital human, wherein, The device includes: The first encoding module is used to acquire enhanced prompt information and encode the enhanced prompt information to obtain timing constraint information; The second encoding module is used to acquire a reference image sequence including the digital human image, and to encode the reference image sequence to obtain the digital human's identity style vector. The acquisition module is used to acquire temporal audio based on the enhanced prompt information and the reference image sequence; The generation module is used to generate a target digital human video based on the temporal constraint information, the identity style vector, and the temporal audio.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

18. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-14.

Images