Video generation and model training methods, devices, storage media, and computer program products

By splitting the text in the intelligent customer service system into multiple text fragments and processing them in parallel to generate facial and body driving data from voice, the latency problem in the digital human image driving process is solved, enabling more efficient video generation and improving the user interaction experience.

CN123123655APending Publication Date: 2026-11-17MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610975697.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In intelligent customer service systems, the combined time spent on text generation and image driving during the digital human image driving process increases system response latency and affects the user's real-time interactive experience.

Method used

The target text is split into multiple text segments in the time domain, and the speech segments are processed in parallel to generate facial and limb driving data. The digital human dynamic video is generated by processing the facial driving model and limb driving model in parallel.

Benefits of technology

It reduces the waiting latency between text generation and image driving, as well as the serial computation latency between face and limb driving, thereby improving video generation efficiency and meeting real-time requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123655A_ABST
    Figure CN123123655A_ABST
Patent Text Reader

Abstract

This application provides a video generation and model training method, apparatus, storage medium, and computer program product. The video generation method includes: splitting the target text into multiple text segments in the time domain, and performing speech conversion on each text segment to obtain multiple speech segments corresponding to the multiple text segments; processing the multiple speech segments in parallel to generate facial driving data and limb driving data for each speech segment; wherein, for each speech segment, calling the facial driving model to generate facial driving data, and simultaneously calling the limb driving model to generate limb driving data; and driving a digital human standard image based on the facial driving data and limb driving data of each speech segment to generate a digital human dynamic video corresponding to the target text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation and model training method, apparatus, storage medium, and computer program product. Background Technology

[0002] With the development of large language model technology, intelligent customer service systems have been widely applied in various service scenarios. Driven by the need to improve the interactive experience, visual interaction methods based on image-based digital humans are gradually becoming an important development direction for intelligent customer service.

[0003] Currently, the response process of intelligent customer service typically includes: receiving user questions, generating text replies, and using the text replies to drive digital human images for visualization. The driving of the digital human images usually employs a serial processing method. This method causes the time spent on text generation and image driving to accumulate, leading to an increase in overall system response latency and impacting the user's real-time interactive experience. Summary of the Invention

[0004] This application provides a video generation and model training method, apparatus, storage medium, and computer program product.

[0005] This application provides a video generation method, the method comprising: The target text is split into multiple text segments in the time domain, and each text segment is converted into speech to obtain multiple speech segments corresponding to the multiple text segments; The multiple speech segments are processed in parallel to generate facial driving data and limb driving data for each speech segment; wherein, for each speech segment, the facial driving model is called to generate facial driving data, and the limb driving model is called simultaneously to generate limb driving data. Based on facial and limb driving data from each segment of speech, a standard digital human image is generated to produce a dynamic video of the digital human corresponding to the target text.

[0006] This application provides a model training method, the method comprising: Collect sample speech and annotate the sample speech with first facial driving data and first limb driving data; For the sample speech, the first model is invoked to predict facial driving data to obtain second facial driving data, and the second model is invoked simultaneously to predict limb driving data to obtain second limb driving data. The first model is trained based on the loss function between the first facial driving data and the second facial driving data to obtain the facial driving model as described in claim 1. The second model is trained based on the loss function between the first limb driving data and the second limb driving data to obtain the limb driving model as described in claim 1. The first model includes a first encoder, a temporal modeling layer, and a first decoder. The first encoder is used to encode the temporal embedding vector of speech, the temporal modeling layer is used to process the temporal dependencies of the encoded features, and the first decoder is used to predict face driving data. The second model includes a second encoder, a mapper, and a second decoder. The second encoder is used to encode temporal embedding vectors of speech, the mapper is used to map limb movement parameters based on encoded features, and the decoder is used to predict limb drive data.

[0007] This application provides an electronic device, including: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement a video generation method or a model training method.

[0008] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a video generation method or a model training method.

[0009] This application provides a computer program product, including a computer program that, when executed by a processor, implements a video generation method or a model training method.

[0010] This application provides a video generation and model training method, apparatus, storage medium, and computer program product. The video generation method includes: splitting target text into multiple text segments in the temporal domain, and performing speech conversion on each text segment to obtain multiple speech segments corresponding to the multiple text segments; processing the multiple speech segments in parallel to generate facial driving data and limb driving data for each speech segment; wherein, for each speech segment, calling a facial driving model to generate facial driving data, and simultaneously calling a limb driving model to generate limb driving data; and driving a standard digital human image based on the facial driving data and limb driving data of each speech segment to generate a dynamic digital human video corresponding to the target text. The technical solution provided in this application segments the target text into speech in the temporal domain, and then generates driving data for the digital human image in parallel. Furthermore, for each speech segment, the driving task for the digital human image is split in the spatial and temporal domains, decoupling facial and limb driving for parallel processing, and finally fusing to obtain a digital human video image. This reduces the waiting latency between text generation and image driving, as well as the serial computation latency between facial and limb driving, improving video generation efficiency to meet real-time requirements. Attached Figure Description

[0011] Figure 1 A flowchart illustrating a video generation method provided in an embodiment of this application; Figure 2 A schematic diagram of an exemplary architecture for generating dynamic videos of digital humans is provided for embodiments of this application; Figure 3 A schematic flowchart illustrating a model training method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. The embodiments below can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0014] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0015] This application provides a video generation method, which is applied to a video generation device, and the video generation device can be any electronic device.

[0016] This video generation method is applicable to applications such as large-scale intelligent customer service. Addressing the issue of significant latency and unmet real-time performance in intelligent customer service question-and-answer sessions, which negatively impacts user experience, the method splits the driving task into spatial and temporal domains. In the spatial domain, facial and limb drivers are decoupled and processed in parallel, with cross-modal feature fusion resolving consistency issues. In the temporal domain, text is segmented and processed in parallel, with cross-segment frame alignment and jitter reduction optimization addressing alignment problems. This achieves real-time question-and-answer performance and improves user experience. The core points are: When users are talking to the intelligent customer service, the response text generated by the large model, that is, the target text, is split into multiple text segments in the time domain, and each text segment is converted into a corresponding audio segment. By processing each audio segment in parallel, driving data corresponding to each audio segment is generated to drive the standard image of the digital human (e.g., the image of the intelligent customer service). In other words, by splitting the task in the time domain, the latency of question and answer is reduced. For single-segment speech, in the process of processing it to generate driving data for driving standard images of digital humans, the process of generating driving data is split into facial driving tasks and limb driving tasks. By processing facial driving tasks and limb driving tasks in parallel, the corresponding driving data is generated. That is, by splitting the tasks in the spatial domain, the latency of question and answer is reduced.

[0017] The video generation method of this application mainly includes the following: The target text is split into multiple text segments, for example, k text segments [T1,T2,...,Tk], where k is a natural number greater than 1; Convert [T1,T2,...,Tk] into speech [voice1,voice2,...,voicek]; Prosodic features, phonemic features, and emotional features are extracted from the speech [voice1, voice2, ..., voicek] to form temporal embedding vectors for each speech segment, which are then transformed (generated) into facial-driven data and limb-driven data. The system completes the spatial and temporal migration of actions and expressions and drives the generation of digital human video clips, which are then combined into a video stream and pushed to the user for playback.

[0018] Based on the above, the video generation method of this application is described in detail below.

[0019] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the video generation method mainly includes the following steps: S101. The target text is split into multiple text segments in the time domain, and each text segment is converted into speech to obtain multiple speech segments corresponding to the multiple text segments.

[0020] In the embodiments of this application, the video generation device can split the target text into multiple text segments in the time domain and perform speech conversion on each text segment to obtain multiple audio segments corresponding to the multiple text segments.

[0021] It should be noted that, in the embodiments of this application, in the intelligent customer service dialogue scenario, the target text can be the reply text generated by the intelligent customer service based on the context when the user asks a question to the intelligent customer service. Of course, the target text can also be other text formed by information that needs to be provided to the user, and this embodiment of the application does not limit it.

[0022] In the embodiments of this application, the video generation device splits the target text into multiple text segments in the time domain, including: splitting the target text in the time domain according to semantics and / or rhythm, based on the principles of complete semantic units, standardized punctuation and pauses, and reasonable breathing pauses, to obtain multiple text segments.

[0023] It should be noted that, in the embodiments of this application, the video generation device equally divides the target text into k text segments (multiple text segments) according to semantics and / or rhythm; it is necessary to ensure that punctuation pauses are standardized, breathing pauses are reasonable, and semantic units are complete, where k is a natural number greater than 1.

[0024] It should be noted that in the embodiments of this application, a combination of natural language processing and deep learning is used to implement the segmentation algorithm. Natural Language Processing (NLP) is the "foundation" and "intelligence" of text segmentation, providing linguistic basis, feature sources, rule-based heuristics, and evaluation criteria. Deep Learning (DL) provides fundamental capabilities and knowledge, serving as the "intelligent engine" for text segmentation. Through powerful neural networks (especially Transformer models / Bidirectional Encoder Representations from Transformers (BERT)), it automatically learns the optimal boundary determination strategy from data, achieving high-precision and adaptive segmentation. The framework can employ the classic unsupervised text segmentation algorithm (TextTiling) + BERT fine-tuning, thereby calculating the text segmentation: [T1,T2,...,Tk] = ((TextTiling + BERT)(Text)), where T1,T2,...,Tk represent k text segments, and Text represents the target text.

[0025] In the embodiments of this application, the video generation device can convert each text segment of the target text into a piece of speech. Specifically, the text-to-speech (TTS) framework OpenVoice can be used to convert k text segments [T1, T2, ..., Tk] into corresponding speech: [voice1, voice2, each voicek] = OpenVoice([T1, T2, ..., Tk]).

[0026] S102. Parallel processing of multiple speech segments to generate facial driving data and limb driving data for each speech segment; specifically, for each speech segment, the facial driving model is called to generate facial driving data, and the limb driving model is called simultaneously to generate limb driving data.

[0027] In the embodiments of this application, after obtaining multiple audio segments corresponding to multiple text fragments, the video generation device further processes the multiple audio segments in parallel to generate facial driving data and limb driving data for each audio segment; wherein, for each audio segment, the facial driving model is called to generate facial driving data, and the limb driving model is called simultaneously to generate limb driving data.

[0028] It is understood that in the embodiments of this application, the video generation device generates facial driving data and limb driving data in parallel for each segment of speech. Furthermore, when generating driving data for a single segment of speech, facial driving data and limb driving data are also generated in parallel. That is, the generation of driving data between each segment of speech is parallel, and the generation of different types of driving data for a single segment of speech is also parallel, which is beneficial for timely subsequent video generation and ensures the real-time nature of video generation.

[0029] In the embodiments of this application, for each segment of speech, a face-driven model is invoked to generate face-driven data, and a limb-driven model is invoked simultaneously to generate limb-driven data. This includes: for each segment of speech, extracting speech features and temporal features to form corresponding temporal embedding vectors; wherein, the speech features of each segment of speech include at least one or more of the following types of features: prosodic features, phoneme features, and emotional features; for each segment of speech, a face-driven model is invoked to generate face-driven data based on the corresponding temporal embedding vectors, and a limb-driven model is invoked simultaneously to generate limb-driven data based on the corresponding temporal embedding vectors.

[0030] It should be noted that, in the embodiments of this application, the video generation device can use the open-source framework Whisper-tiny to obtain temporal embedding vectors from each segment of speech, containing information such as prosodic features, phoneme features, and emotional features of the speech. Specifically, the implementation is as follows: Whisper-tiny is used to convert the audio signal of the input speech into an 80-channel Mel spectrogram, capturing the prosodic features, phoneme features, and emotional features of the speech segment, thereby generating a temporal embedding vector Shape=[t, prosodic, phoneme, emotional, ...] containing the prosodic features, phoneme features, and emotional features of the speech segment, where t represents the temporal features of the speech segment, such as the duration of the speech segment, which is not limited here. The temporal embedding vectors corresponding to each segment of speech are integrated together to obtain a set of temporal embedding vectors [Shape1, Shape2, ..., Shapek], where Shape1, Shape2, ..., Shapek represent the temporal embedding vectors of k segments of speech, respectively.

[0031] It should be noted that in the embodiments of this application, Whisper-tin is a lightweight multilingual speech recognition model developed by OpenAI. It is the smallest version in the Whisper series and is designed for scenarios with limited hardware resources and requiring fast processing.

[0032] In the embodiments of this application, for the same audio segment, the video generation device can simultaneously invoke a face-driven model and a limb-driven model. The face-driven model generates face-driven data based on the temporal embedding vector corresponding to the audio segment, and the limb-driven model generates face-driven data based on the temporal embedding vector corresponding to the audio segment. For detailed explanations of the face-driven model and the limb-driven model, please refer to the model training section below.

[0033] For example, in an embodiment of this application, the target text is split into k text segments and converted into k speech segments. The k speech segments correspond to k temporal embedding vectors, which are integrated into a temporal embedding vector set [Shape1, Shape2, ..., Shapek]. Facial driving data ms, including lip-sync and facial expression data, are extracted from each speech segment [Shape1, Shape2, ShaShapek] using a facial driving model, resulting in [ms1, ms2, ..., msk], where MS represents temporal lip-sync and facial expression driving data (facial driving data). Limb driving data lb are extracted from each speech segment [Shape1, Shape2, ShaShapek] using a limb driving model, resulting in [lb1, lb2, Slbk].

[0034] S103. Based on facial and limb driving data from each segment of speech, drive the standard digital human image to generate a dynamic video of the digital human corresponding to the target text.

[0035] In the embodiments of this application, the video generation device can drive the standard digital human image to generate a dynamic video of the digital human corresponding to the target text by obtaining the driving data of each segment of speech.

[0036] In the embodiments of this application, the video generation device drives digital human standard images based on facial driving data and limb driving data of each speech segment to generate a digital human dynamic video corresponding to the target text. The process includes: parallel traversal of facial driving data and limb driving data, calling an image-driven model, driving digital human standard images based on facial driving data of each speech segment to generate a digital human facial video, and driving digital human standard images based on limb driving data of each speech segment to generate a digital human limb video; aligning and fusing the digital human facial video and digital human limb video of the same speech segment into a single digital human video segment; and splicing the digital human video segments of each speech segment in time sequence to generate a digital human dynamic video.

[0037] In the embodiments of this application, when the image-driven model generates digital human face videos and digital human limb videos for two adjacent speech segments, it controls that the facial information of the ending frame of the digital human face video of the preceding speech segment is consistent with the facial information of the starting frame of the digital human face video of the following speech segment, and that the limb information of the ending frame of the digital human limb video of the preceding speech segment is consistent with the limb information of the starting frame of the digital human limb video of the following speech segment. Through the above constraints, the temporal continuity of adjacent video segments can be guaranteed.

[0038] For example, in an embodiment of this application, see Figure 2 The generation of digital human dynamic videos based on facial and limb driving data from various speech segments mainly includes the following parts: 1. Parallel traversal of the face-driven dataset MSDATA (including face-driven data ms1, ms2, ..., msk for each speech segment) and the limb-driven dataset LBDATA (including limb-driven data lb1, lb2, ..., lbk for each speech segment).

[0039] 2. Call the image-driven model to generate lip movements and facial expressions, and generate digital human facial videos (constraining the last action point of the previous segment). The image-driven model is called to generate a digital human body video (the constraint is the last action point of the previous segment).

[0040] 3. Align and merge the digital human facial video and digital human body video of the same audio segment to obtain a video clip.

[0041] 4. Splice the video clips together in chronological order; 5. Complete the generation of dynamic videos of digital humans.

[0042] It should be noted that, in the embodiments of this application, after the video generating device generates a digital human dynamic video, it can push the video to the client, which can then display and play the video.

[0043] In the embodiments of this application, as described above, the video generation device uses multiple models when generating digital human dynamic videos based on target text. These models can be trained by a model training device, which can be any electronic device. The model training device can be the same as the video generation device, or it can be independent and different devices. This application embodiment does not limit this. The model training method is explained below.

[0044] Figure 3 This is a flowchart illustrating a model training method provided in an embodiment of this application. Figure 3 As shown in the embodiments of this application, the model training method mainly includes the following steps: S201. Collect sample speech and label the sample speech with the first facial driving data and the first limb driving data.

[0045] S202. For the sample speech, call the first model to predict facial driving data to obtain the second facial driving data, and simultaneously call the second model to obtain the second limb driving data based on the predicted limb driving data. S203. Train a first model based on the loss function between the first face driving data and the predicted face driving data to obtain a face driving model. The first model includes a first encoder, a temporal modeling layer and a first decoder. The first encoder is used to encode the temporal embedding vector of speech, the temporal modeling layer is used to process the temporal dependency of the encoded features, and the first decoder is used to predict the face driving data. S204. Train the second model based on the loss function between the first limb driving data and the predicted limb driving data to obtain the limb driving model. The second model includes a second encoder, a mapper and a second decoder. The second encoder is used to encode the temporal embedding vector of speech, the mapper is used to map limb action parameters based on the encoded features and the decoder is used to predict limb driving data.

[0046] Based on the above steps S201-S204, the following explains how the first model trains to obtain the facial driving model and the second model trains to obtain the limb model.

[0047] The following explains how the face-driven model is obtained by training the first model.

[0048] In embodiments of this application, the model training device can collect sample speech and annotate the sample speech with first facial driving data. The first facial driving data can be understood as facial driving data that conforms to the characteristics of the sample speech and is intended to be used to drive standard digital human images.

[0049] It should be noted that, in the embodiments of this application, the sample speech can be any segment of speech, and the number of sample speech can be multiple. The specific number of sample speech and the duration of speech can be set according to actual needs and application scenarios, and this application embodiment does not limit them.

[0050] It should be noted that, in the embodiments of this application, the first facial driving data for annotating sample speech includes, but is not limited to: Data acquisition: Simultaneously acquire sample speech (including different emotions / speech rates), lip-sync video (fixed frontal view), and phoneme / prosodic annotation.

[0051] Key annotations: phoneme alignment (e.g., International Phonetic Alphabet (IPA)), prosodic parameters (fundamental frequency F0, energy), and sentiment labels (category + intensity).

[0052] The first facial driving data includes, but is not limited to: mouth shape parameters, such as lip opening and closing, corner of the mouth position, jaw rotation angle, and expression parameters, such as eyebrow movement, eye opening and closing, and cheek movement.

[0053] In the embodiments of this application, when the model training device obtains the first facial driving data of the sample speech, it calls the first model to predict the facial driving data for the sample speech to obtain the second facial driving data.

[0054] It should be noted that, in the embodiments of this application, the implementation of the model training device predicting facial driving data may include: extracting speech features and time features to form a corresponding temporal embedding vector for sample speech; wherein, the speech features of the sample speech include at least one or more of the following types of features: prosodic features, phoneme features, and emotion features; for the sample speech, calling the first model to predict facial driving data based on the corresponding temporal embedding vector to obtain the second facial driving data.

[0055] It should be noted that, in the embodiments of this application, feature extraction includes, but is not limited to: Prosodic features: extract fundamental frequency (F0), energy, and speech rate, and capture temporal changes after normalization.

[0056] Phoneme features: The sequence is generated by the phoneme recognition model and converted into an embedding vector.

[0057] Emotional features: Use pre-trained models to extract acoustic features such as Mel-Frequency Cepstral Coefficients (MFCC) and speech rate, or directly learn the emotion vector.

[0058] It should be noted that, in the embodiments of this application, the first model includes a first encoder, a temporal modeling layer and a first decoder. The first encoder is used to encode the temporal embedding vector of speech, the temporal modeling layer is used to process the temporal dependencies of encoded features, and the first decoder is used to predict facial driving data.

[0059] It should be noted that, in the embodiments of this application, the first model architecture is as follows: Encoder-decoder structure: Encodes prosody, phonemes, and emotional features respectively, and decodes to generate facial driving data such as lip-sync parameters (e.g., 3D vertex coordinates or AU intensity).

[0060] Temporal modeling: Long Short-Term Memory (LSTM) / Transformer is used to handle long-term dependencies, while sliding window or Connectionist Temporal Classification (CTC) loss is used to align phonemes and lip movements.

[0061] It should be noted that, in the embodiments of this application, the first model can realize the fusion of multiple features involved in the temporal embedding vector, for example, by dynamically integrating prosodic features, phoneme features, and emotional features by embedding cross attention or gating mechanisms in the first model.

[0062] In the embodiments of this application, when the model training device calls the first model to predict facial driving data for sample speech and obtains second facial driving data, it can train the first model based on the loss function between the first facial driving data pre-annotated for the sample speech and the second facial driving data predicted by the model. The trained first model is the aforementioned facial driving model.

[0063] It should be noted that, in the embodiments of this application, the model training of the first model can combine the following strategies: Multi-task loss: Jointly optimize lip shape (mean squared error (MSE)), phoneme alignment (CTC), prosodic consistency (Pearson correlation), and sentiment expression (cosine similarity).

[0064] Data augmentation: Adding noise and adjusting speech rate / pitch to improve model robustness.

[0065] Course learning: Gradually transitioning from simple data (neutral speech) to complex scenarios (mixed emotions).

[0066] It should be noted that, in the embodiments of this application, the evaluation and improvement of the first model may include the following: Objective evaluation: lip-sync (MSE), phoneme accuracy (word error rate (WER)), and sentiment classification accuracy.

[0067] Subjective evaluation: naturalness (Mean Opinion Score, MOS) and synchronicity (user feedback).

[0068] Areas for improvement: few-shot learning (meta-learning), speaker-independent feature separation, and lightweight deployment.

[0069] It should be noted that, in the embodiments of this application, the training strategy, evaluation and improvement of the first model described above are only exemplary solutions, and can be set in combination with actual needs and application scenarios. The embodiments of this application are not limited.

[0070] The following explains how the limb-driven model is obtained by training the second model.

[0071] In embodiments of this application, the model training device can collect sample speech and annotate the sample speech with first limb driving data. The first limb driving data can be understood as limb driving data that conforms to the characteristics of the sample speech and is expected to be used to drive standard images of digital humans.

[0072] It should be noted that, in the embodiments of this application, the sample speech can be any segment of speech, and the number of sample speech can be multiple. The specific number of sample speech and the duration of speech can be set according to actual needs and application scenarios, and this application embodiment does not limit them.

[0073] It should be noted that, in the embodiments of this application, the first limb driving data for annotating sample speech includes, but is not limited to: Data acquisition: Simultaneous acquisition of body movements (motion capture (MoCap) / sensors) and voice / text emotion tags.

[0074] Key markings: Action division area (head / trunk / limbs) + Emotion classification (6 categories: joy / anger / sorrow / fear / disgust / surprise) and intensity.

[0075] First-order limb drive data includes, but is not limited to, limb characteristics such as joint angles, range of motion, and frequency (e.g., arm swing cycle).

[0076] In the embodiments of this application, when the model training device obtains the first limb driving data of the sample speech, it calls the second model to predict the limb driving data for the sample speech to obtain the second limb driving data.

[0077] It should be noted that, in the embodiments of this application, the method of the model training device predicting limb driving data may include: extracting speech features and time features to form a corresponding temporal embedding vector for sample speech; wherein, the speech features of the sample speech include at least one or more of the following types of features: prosodic features, phoneme features, and emotion features; for the sample speech, calling the second model to predict limb driving data based on the corresponding temporal embedding vector to obtain the second limb driving data.

[0078] It should be noted that in the embodiments of this application, temporal embedding vectors are used to predict both limb driving data and facial driving data. Therefore, in actual execution, the temporal embedding vector only needs to be extracted once for the sample speech in the manner described above, and it can be used to predict the driving data in both model training stages. For details on the extraction of various features in the temporal embedding vector, please refer to the aforementioned related content, which will not be repeated here.

[0079] It should be noted that, in the embodiments of this application, the second model includes a second encoder, a mapper, and a second decoder. The second encoder is used to encode the temporal embedding vector of speech, the mapper is used to map limb movement parameters based on the encoded features, and the decoder is used to predict limb driving data. It should be noted that, in the embodiments of this application, the second model architecture is as follows: Encoder: Bidirectional LSTM processes temporal actions, and Convolutional Neural Network (CNN) extracts speech features.

[0080] Mapper: Fully connected layer + attention mechanism, which maps emotions to action parameters (e.g., "anger" → leaning forward).

[0081] Decoder: Recurrent Neural Network (RNN) generates action sequences, and Generative Adversarial Network (GAN) enhances naturalness.

[0082] In the embodiments of this application, when the model training device calls the second model to predict limb driving data for sample speech and obtains the second limb driving data, it can train the second model based on the loss function between the first limb driving data pre-annotated for the sample speech and the second limb driving data predicted by the model. The trained second model is the aforementioned limb driving model.

[0083] It should be noted that, in the embodiments of this application, the training optimization of the second model includes, but is not limited to: Loss function calculation: Action reconstruction (MSE) + sentiment alignment (cross-entropy).

[0084] Model training strategy: transfer learning (pre-trained sentiment classification model) + curriculum learning (from simple to complex sentiment).

[0085] In the embodiments of this application, referring to the aforementioned video generation method, not only are face-driven models and limb-driven models used, but also an image-driven model is used when generating digital human dynamic videos based on driving data. The training method of this model is explained below.

[0086] In the embodiments of this application, the model training device may further perform the following steps: acquiring digital human sample images; training the third model based on the digital human sample images to obtain an image-driven model by performing the following dual-capability training: generating masks for the facial and limb regions of the digital human sample images, freezing the backbone network of the third model, performing local predictions of the face and limbs based on the masks, and training the local image editing capability of the third model through mask loss; introducing the pose key points and motion vector field of the digital human sample images as input conditions into the third model to train the ability of the third model to generate coherent action sequences; wherein, the third model is a diffusion model based on a neural network architecture, used to generate dynamic videos of digital humans.

[0087] It should be noted that in the embodiments of this application, the image-driven approach is solved through a diffusion model. To achieve parallel processing in the spatial and temporal domains, two problems need to be solved: first, the generation and modification of parts of the image can be realized; second, the actions of segmented connection parts can be seamlessly connected.

[0088] It should be noted that, in the embodiments of this application, the diffusion model as the third model can be a stable diffusion (SD) generation model.

[0089] In the embodiments of this application, on the one hand, the DreamBooth technique combined with Low-Rank Adaptation (LoRA) can be used to fine-tune the SD generation model (the third model) to achieve the purpose of modifying the image portion. The specific implementation of model fine-tuning is as follows: Core Methodology: DreamBooth+Region Identifiers: This approach uses rare markers (such as [V]) to bind target regions (facial / limb regions), combined with class hints (such as "a [V]bag") to achieve local feature learning, avoiding global parameter updates. It can also be understood as using markers to specify a region of the image, allowing the model to learn only the features of that region.

[0090] LoRA local adaptation: Inject a low-rank matrix into the model's U-Net or Variational Autoencoder (VAE) to freeze the backbone network. The encoder only needs to fine-tune a small number of parameters (preferably rank r = 4-8).

[0091] Key constraint technologies: Region mask loss: Only the target region loss is calculated (formula L=MSE(generated region, target region)×mask_weight), which improves training efficiency.

[0092] Class prior preservation: Generate a regularized dataset using general hints (such as "a dog"), jointly optimize instance loss and class loss, and prevent semantic drift.

[0093] Workflow: Data preparation: A certain number of images of the target area are labeled with precise masks; a general-purpose hint regular expression set is generated.

[0094] Model modification: Freeze the backbone network in the model, such as the Contrastive Language–Image Pre-training (CLIP) encoder, and add a region constraint module to UNetBlock of U-Net.

[0095] Training optimization: Adam optimizer with weight decay (AdamW optimizer) + gradient clipping (threshold 1.0) + validation class hints quality every 100 steps.

[0096] In an embodiment of this application, another approach is to use a conditional control fine-tuning model based on ControlNet to achieve segmented video generation, ensuring that the lip movements, facial expressions, and body movements of characters at the beginning and end of consecutive videos are coordinated. The specific implementation of model fine-tuning is as follows: Fine-tuning plan: Conditional control fine-tuning based on ControlNet. The skeletal keypoint branch (OpenPose) and facial keypoints (Dlib) of ControlNet are used as conditional inputs, and human pose features are fused with the main SD network in the SD generative model through residual connections. During training, it is necessary to ensure the alignment of the start / end pose keypoints of adjacent segments, and to achieve motion coherence by minimizing pose difference loss (such as L2 loss). An optical flow is introduced into ControlNet as a condition, and the motion trend of the current frame and the next frame is encoded into the generation process through feature concatenation. It is recommended to use a fusion strategy of residual connections + scaling factor (α=0.5) to balance the original generated features and motion guidance information.

[0097] Fine-tuning process: 1. Data preprocessing: Key point extraction: OpenPose was used to extract key points of the human skeleton (18 joint coordinates), and Dlib was used to extract 68 key points of the face to generate a standardized heatmap as input.

[0098] Motion vector field encoding: The motion vector fields of adjacent frames are calculated using optical flow algorithms (such as RAFT) and converted into feature maps.

[0099] Conditional alignment: Rigid alignment is performed on the start / end pose keypoints of adjacent segments to ensure motion continuity. 2. Training strategy: Loss function design: Generation loss: Standard diffusion loss (L1 distance between predicted noise and actual noise).

[0100] Pose alignment loss: L2 loss constrains the difference in keypoint coordinates between adjacent frames (e.g., mean joint position error ≤ 5%).

[0101] Motion consistency loss: the matching loss between the optical flow field and the motion of the generated image (e.g., cosine similarity ≥ 0.8).

[0102] 3. Training Techniques Randomly mask 50% of the text embedding to enhance the model's reliance on conditional inputs.

[0103] A two-stage training method is adopted: first, the SD main network is frozen to train ControlNet, and then joint fine-tuning is performed.

[0104] The technical solution provided in this application segments the target text into speech in the time domain, and then generates driving data for digital human images in parallel. For each speech segment, the driving task of the digital human image is further split in the spatial and temporal domains, and the facial and limb driving is decoupled and processed in parallel. Finally, the digital human video image is obtained by fusion. This reduces the waiting latency between text generation and image driving, as well as the serial computation latency between facial and limb driving, and improves the video generation efficiency to meet real-time requirements.

[0105] This application also provides a video generation apparatus. Figure 4 This is a schematic diagram of a video generation device provided in an embodiment of this application. Figure 4 As shown, in an embodiment of this application, the video generation device 1 includes: The text processing module 11 is used to split the target text into multiple text segments in the time domain, and to perform speech conversion on each text segment to obtain multiple speech segments corresponding to the multiple text segments; The speech processing module 12 is used to process the multiple speech segments in parallel and generate facial driving data and limb driving data for each speech segment; wherein, for each speech segment, the facial driving model is called to generate facial driving data, and the limb driving model is called simultaneously to generate limb driving data. The video generation module 13 is used to generate a dynamic video of the digital human corresponding to the target text by driving the standard digital human image based on the facial driving data and limb driving data of each segment of speech.

[0106] In one embodiment of this application, the text processing module 11 is used to split the target text in the time domain according to semantics and / or rhythm, based on the principles of complete semantic units, standardized punctuation and pauses, and reasonable breathing pauses, to obtain the multiple text fragments.

[0107] In one embodiment of this application, the speech processing module 12 is used to extract speech features and time features to form corresponding temporal embedding vectors for each speech segment; wherein, the speech features of each speech segment include at least one or more of the following types of features: prosodic features, phoneme features, and emotion features; for each speech segment, the face driving model is called to generate face driving data based on the corresponding temporal embedding vector, and the limb driving model is called simultaneously to generate limb driving data based on the corresponding temporal embedding vector.

[0108] In one embodiment of this application, the video generation module 13 is used to traverse facial driving data and limb driving data in parallel, call an image driving model, drive the digital human standard image based on the facial driving data of each speech segment to generate a digital human facial video, and drive the digital human standard image based on the limb driving data of each speech segment to generate a digital human limb video; align and merge the digital human facial video and digital human limb video of the same speech segment into a digital human video segment; and splice the digital human video segments of each speech segment in time sequence to generate the digital human dynamic video.

[0109] In one embodiment of this application, when the image-driven model generates digital human face video and digital human limb video for two adjacent speech segments, it controls the facial information of the end frame of the digital human face video of the previous speech segment to be consistent with the facial information of the start frame of the digital human face video of the next speech segment, and the limb information of the end frame of the digital human limb video of the previous speech segment to be consistent with the limb information of the start frame of the digital human limb video of the next speech segment.

[0110] This application also provides a model training device. Figure 5 This is a schematic diagram of a model training device provided in an embodiment of this application. Figure 6 As shown, in an embodiment of this application, the model training device 2 includes: Sample acquisition module 21 is used to acquire sample speech and annotate the sample speech with first facial driving data and first limb driving data; The model prediction module 22 is used to call the first model to predict facial driving data for the sample speech, obtain second facial driving data, and simultaneously call the second model to obtain second limb driving data based on the predicted limb driving data. Model training module 23 is used to train the first model based on the loss function between the first facial driving data and the second facial driving data to obtain a facial driving model; and to train the second model based on the loss function between the first limb driving data and the second limb driving data to obtain a limb driving model. The first model includes a first encoder, a temporal modeling layer, and a first decoder. The first encoder is used to encode the temporal embedding vector of speech, the temporal modeling layer is used to process the temporal dependencies of the encoded features, and the first decoder is used to predict face driving data. The second model includes a second encoder, a mapper, and a second decoder. The second encoder is used to encode temporal embedding vectors of speech, the mapper is used to map limb movement parameters based on encoded features, and the decoder is used to predict limb drive data.

[0111] In one embodiment of this application, the model prediction module 23 is used to extract speech features and time features to form a corresponding temporal embedding vector for the sample speech; wherein, the speech features of the sample speech include at least one or more of the following types of features: prosodic features, phoneme features, and emotion features; for the sample speech, the first model is called to predict facial driving data based on the corresponding temporal embedding vector to obtain the second facial driving data, and the second model is simultaneously called to predict limb driving data based on the corresponding temporal embedding vector to obtain the second limb driving data.

[0112] In one embodiment of this application, the sample acquisition module 21 is further used to acquire digital human sample images; The model training module 23 is also used to perform the following dual-capability training on the third model based on the digital human sample image to obtain an image-driven model: A mask is generated for the facial and limb regions of the digital human sample image. The backbone network of the third model is frozen. Local prediction of the face and limbs is performed based on the mask. The local image editing capability of the third model is trained through mask loss. The pose key points and motion vector field of the digital human sample image are introduced into the third model as input conditions to train the third model to generate a coherent sequence of actions. The third model is a diffusion model based on a neural network architecture, used to generate dynamic videos of digital humans.

[0113] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown in the embodiments of this application, the electronic device 3 includes: a processor 31, a memory 32, and a communication bus 33; The communication bus 33 is used to realize the communication connection between the processor 31 and the memory 32; The processor 31 is used to execute one or more computer programs stored in the memory 32 to implement a video generation method or a model training method.

[0114] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a video generation method or a model training method.

[0115] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a video generation method or a model training method. The computer-readable storage medium can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be a device including one or any combination of the above-mentioned memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.

[0116] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0117] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video generation method, characterized in that, The method includes: The target text is split into multiple text segments in the time domain, and each text segment is converted into speech to obtain multiple speech segments corresponding to the multiple text segments; The multiple speech segments are processed in parallel to generate facial driving data and limb driving data for each speech segment; wherein, for each speech segment, the facial driving model is called to generate facial driving data, and the limb driving model is called simultaneously to generate limb driving data. Based on facial and limb driving data from each segment of speech, a standard digital human image is generated to produce a dynamic video of the digital human corresponding to the target text.

2. The method according to claim 1, characterized in that, For each segment of speech, the process involves calling a facial driving model to generate facial driving data and simultaneously calling a limb driving model to generate limb driving data, including: For each speech segment, speech features and temporal features are extracted to form a corresponding temporal embedding vector; wherein, the speech features of each speech segment include at least one or more of the following types of features: prosodic features, phoneme features, and emotion features; For each segment of speech, the face-driven model is invoked to generate face-driven data based on the corresponding temporal embedding vector, and the limb-driven model is invoked simultaneously to generate limb-driven data based on the corresponding temporal embedding vector.

3. The method according to claim 1, characterized in that, The process of generating a dynamic video of the digital human corresponding to the target text by driving the standard digital human image based on facial and limb driving data from each segment of speech includes: The system iterates through the facial driving data and limb driving data in parallel, calls the image driving model, drives the digital human standard image based on the facial driving data of each speech segment to generate a digital human facial video, and drives the digital human standard image based on the limb driving data of each speech segment to generate a digital human limb video. Align and merge digital human facial videos and digital human body videos with the same audio recording into a single digital human video clip; The digital human video segments of each audio segment are spliced ​​together in chronological order to generate the digital human dynamic video.

4. The method according to claim 3, characterized in that, When generating digital human face videos and digital human limb videos for two adjacent speech segments, the image-driven model controls the facial information of the ending frame of the digital human face video of the preceding speech segment to be consistent with the facial information of the starting frame of the digital human face video of the following speech segment, and the limb information of the ending frame of the digital human limb video of the preceding speech segment to be consistent with the limb information of the starting frame of the digital human limb video of the following speech segment.

5. A model training method, characterized in that, The method includes: Collect sample speech and annotate the sample speech with first facial driving data and first limb driving data; For the sample speech, the first model is invoked to predict facial driving data to obtain second facial driving data, and the second model is invoked simultaneously to predict limb driving data to obtain second limb driving data. The first model is trained based on the loss function between the first facial driving data and the second facial driving data to obtain the facial driving model as described in claim 1. The second model is trained based on the loss function between the first limb driving data and the second limb driving data to obtain the limb driving model as described in claim 1. The first model includes a first encoder, a temporal modeling layer, and a first decoder. The first encoder is used to encode the temporal embedding vector of speech, the temporal modeling layer is used to process the temporal dependencies of the encoded features, and the first decoder is used to predict face driving data. The second model includes a second encoder, a mapper, and a second decoder. The second encoder is used to encode temporal embedding vectors of speech, the mapper is used to map limb movement parameters based on encoded features, and the decoder is used to predict limb drive data.

6. The method according to claim 5, characterized in that, For the sample speech, the first model is invoked to predict facial driving data to obtain second facial driving data, and simultaneously the second model is invoked based on the predicted limb driving data to obtain second limb driving data, including: For the sample speech, speech features and temporal features are extracted to form a corresponding temporal embedding vector; wherein, the speech features of the sample speech include at least one or more of the following types of features: prosodic features, phoneme features, and emotion features; For the sample speech, the first model is invoked to predict facial driving data based on the corresponding temporal embedding vector to obtain the second facial driving data, and the second model is simultaneously invoked to predict limb driving data based on the corresponding temporal embedding vector to obtain the second limb driving data.

7. The method according to claim 5, characterized in that, The method further includes: Collect digital human sample images; Based on the digital human sample images, the third model is trained using the following dual-capability method to obtain the image-driven model as described in claim 4: A mask is generated for the facial and limb regions of the digital human sample image. The backbone network of the third model is frozen. Local prediction of the face and limbs is performed based on the mask. The local image editing capability of the third model is trained through mask loss. The pose key points and motion vector field of the digital human sample image are introduced into the third model as input conditions to train the third model to generate a coherent sequence of actions. The third model is a diffusion model based on a neural network architecture, used to generate dynamic videos of digital humans.

8. An electronic device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the method according to any one of claims 1-4, or the method according to any one of claims 5-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.