Digital human video generation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of virtual human generation, and provides a digital human video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a reference image, an input text and a voice signal corresponding to the input text; performing multi-modal feature extraction on the reference image, the input text and the voice signal to obtain an identity condition feature, an action label feature and a voice feature, and obtaining a control condition feature for controlling action performance; inputting the identity condition features, the action label features, the voice features and the control condition features into an action generation model, and predicting to obtain a target action parameter sequence; and generating a target digital human video according to the target action parameter sequence. According to the method, multi-modal features such as identities, action tags and voices are fused, explicit control conditions are introduced, and fine actions from the whole body to local parts such as hands and faces can be generated, so that semantic consistency, character consistency and action naturalness are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of virtual human generation and multimodal motion synthesis technology, and in particular to a digital human video generation method, apparatus, electronic device and storage medium. Background Technology

[0002] In virtual digital human applications, a key technology is generating natural and expressive body movements that match the input speech and text. Existing speech-driven motion generation technologies typically use speech signals or text content as the main input, employing deep learning models to generate sequences of body movements that match the speech or text content. However, these methods generally suffer from some inherent drawbacks.

[0003] First, existing methods typically lack the ability to personalize character models, failing to generate personalized actions based on user-provided reference images. Second, their control dimensions are relatively limited, primarily relying on voice and text as conditions, lacking flexible control over finer-grained dimensions such as key postures, motion trajectories, and dynamic amplitudes, resulting in insufficiently rich and precise semantic expression of the generated actions. Furthermore, existing technologies often employ a single-stage prediction structure, generating only the overall movement of the body's core, lacking effective modeling of crucial details such as hand postures and facial expressions, leading to insufficient overall coordination and naturalness of the generated actions. Summary of the Invention

[0004] This invention provides a digital human video generation method, apparatus, electronic device, and storage medium to address the shortcomings of existing digital human video generation technologies, such as the lack of personalized features in generated actions, limited controllable dimensions, and insufficient expressiveness of action details.

[0005] This invention provides a method for generating digital human videos, comprising: Acquire a reference image, input text, and the corresponding speech signal of the input text; Multimodal feature extraction is performed on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, and control condition features for controlling action performance are obtained. The identity condition features, the action label features, the voice features, and the control condition features are input into a pre-trained action generation model to predict a target action parameter sequence, which includes a body skeleton parameter sequence, a hand parameter sequence, and a facial parameter sequence. Based on the target action parameter sequence, a target digital human video is generated.

[0006] According to a digital human video generation method provided by the present invention, the step of performing multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action tag features, and speech features includes: The limb key points and hand key points in the reference image are detected, and the limb key points and hand key points are encoded to obtain skeletal features; Extract the semantic features of the reference image, and fuse the skeleton features with the semantic features to obtain the identity condition features; The input text is subjected to action intent prediction to generate frame-level action tags, and the frame-level action tags are encoded to obtain the action tag features; The speech signal is segmented and its features are extracted to obtain the speech features.

[0007] According to a digital human video generation method provided by the present invention, the step of predicting action intent from the input text, generating frame-level action tags, and encoding the frame-level action tags to obtain the action tag features includes: Based on the trained text action prediction model, semantically relevant frame-level action labels are obtained by predicting the input text. The frame-level action tag is decoded into a one-hot encoded action tag, and the one-hot encoded action tag is mapped and added to the relative position code to obtain the action tag feature.

[0008] According to a digital human video generation method provided by the present invention, the step of segmenting and extracting features from the speech signal to obtain the speech features includes: The voice signal is divided into multiple audio segments according to a preset time window size; If the length of the last audio segment is less than the preset time window size, the last audio segment is supplemented by repeating the audio. The features of each audio segment are extracted using an audio encoder, and the features are then processed through a mapping layer to obtain the speech features.

[0009] According to the present invention, a digital human video generation method is provided, wherein the action generation model includes a cascaded first generation model and a second generation model, and the step of inputting the identity condition features, the action label features, the speech features, and the control condition features into a pre-trained action generation model to predict a target action parameter sequence includes: The identity condition features, the action label features, the voice features, and the control condition features are input into the first generation model to obtain the body skeleton parameter sequence output by the first generation model; Extract the hidden body features output by the first generative model, and input the hidden body features, along with the identity condition features, the action label features, the speech features, and the control condition features, into the second generative model to obtain the hand parameter sequence and facial parameter sequence output by the second generative model.

[0010] According to a digital human video generation method provided by the present invention, the first generation model adopts a network architecture based on a diffusion transformer, and the first generation model includes a cascaded first processing module and a second processing module. The step of inputting the identity condition features, the action label features, the voice features, and the control condition features into the first generation model to obtain the body skeleton parameter sequence output by the first generation model includes: Obtain an initial body noise sequence, which includes a known sequence of real motion frames and a sequence of actual image frames to be denoised; The current time step features, the speech features, and the initial body noise sequence are concatenated, and the concatenated features and the identity condition features embedded through adaptive normalization are input into the first processing module to obtain the intermediate hidden layer features output by the first processing module; wherein, the first processing module re-embeds the speech features through a cross-attention mechanism. The intermediate hidden layer features, action label features, control condition features, and speech features are concatenated along the channel dimension and then input into the second processing module for feature decoding to predict and output the body skeleton parameter sequence.

[0011] According to a digital human video generation method provided by the present invention, the second generation model adopts a network architecture based on a diffusion transformer, and the second generation model includes a hand parameter prediction branch and a face parameter prediction branch. The process involves extracting the hidden body features output by the first generative model, and inputting these hidden body features, along with the identity condition features, the action label features, the speech features, and the control condition features, into the second generative model to obtain the hand parameter sequence and facial parameter sequence output by the second generative model. The initial hand noise sequence is fused with the body hidden layer features and multimodal conditional features, and then input into the hand parameter prediction branch to obtain the hand parameter sequence output by the hand parameter prediction branch. The initial facial noise sequence is fused with the body hidden layer features and the multimodal conditional features, and then input into the facial parameter prediction branch to obtain the facial parameter sequence output by the facial parameter prediction branch. The multimodal condition features include the identity condition features, the action tag features, the voice features, the control condition features, and the current time step features.

[0012] According to a digital human video generation method provided by the present invention, the training steps of the motion generation model include: A first initial model is constructed, and the first initial model is trained based on a sample set labeled with action data to obtain the first generated model. The sample speech signals, sample reference images, and sample control conditions in the sample set are all trained using a conditional discard strategy. Construct a second initial model and freeze the parameters of the trained first generative model. Use the body hidden layer features output by the first generative model as conditional input to update and train the parameters of the second initial model to obtain the second generative model.

[0013] According to a digital human video generation method provided by the present invention, the control condition features include keyframe trajectory features and dynamic diversity condition features, and the acquisition of control condition features for controlling motion performance includes: Obtain a specified keyframe and / or keypoint trajectory sequence, and encode the keyframe and / or keypoint trajectory sequence to obtain the keyframe trajectory feature, which is used to guide the starting orientation of the action and / or the motion path. The dynamic diversity value is obtained as the dynamic diversity condition feature, which is used to adjust the amplitude and diversity of the generated actions. In the model training phase, the dynamic diversity value is calculated based on the deviation between the skeleton sequence of the training samples and the average skeleton.

[0014] According to a digital human video generation method provided by the present invention, the step of generating a target digital human video based on the target motion parameter sequence includes: A sliding window mechanism is used to divide the long sequence to be generated into multiple video segments with overlapping frames; Each video segment is subjected to progressive denoising inference to obtain the corresponding motion parameter segment; The overlapping frame regions between adjacent video segments are subjected to mean smoothing, and then spliced ​​together to obtain the target motion parameter sequence. Based on the target action parameter sequence, video rendering is performed to generate the target digital human video.

[0015] The present invention also provides a digital human video generation apparatus, comprising: The data acquisition unit is used to acquire a reference image, input text, and the speech signal corresponding to the input text. The feature extraction unit is used to perform multimodal feature extraction on the reference image, the input text and the speech signal to obtain identity condition features, action label features and speech features, and to obtain control condition features for controlling action performance. The parameter prediction unit is used to input the identity condition features, the action label features, the voice features and the control condition features into a pre-trained action generation model to predict the target action parameter sequence, which includes a body skeleton parameter sequence, a hand parameter sequence and a facial parameter sequence. The video generation unit is used to generate a target digital human video based on the target action parameter sequence.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the digital human video generation method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the digital human video generation method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the digital human video generation method as described above.

[0019] The digital human video generation method, apparatus, electronic device, and storage medium provided by this invention extract identity condition features from user-provided reference images, enabling the generated actions to maintain consistency with the features of the person in the reference image. This solves the problem of existing technologies being unable to generate personalized actions, enhancing the flexibility of applications and user customization capabilities. Secondly, this invention not only extracts action label features and speech features from speech and text, but also introduces additional control condition features, allowing users to explicitly guide and intervene in the expression of actions, thereby achieving more accurate and richer semantic expression and higher controllability. Furthermore, this invention explicitly divides the predicted target action parameter sequence into three parts: the body skeleton, hands, and face. This hierarchical and refined prediction method ensures that while generating coherent full-body main movements, it also takes into account the fine dynamics of local areas such as the hands and face, resulting in a significant improvement in the coordination, naturalness, and detail of the final digital human video. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the digital human video generation method provided by the present invention; Figure 2 This is the overall network architecture diagram of the action generation model provided by the present invention; Figure 3 This is a schematic diagram of the structure of the digital human video generation device provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] In recent years, with the increasingly widespread application of virtual digital humans (also known as virtual humans) in fields such as live streaming, customer service, education, and entertainment, how to generate full-body movements of virtual humans that are highly synchronized with voice content, natural and smooth, and expressive has become a research hotspot and key technology in this field.

[0024] Current speech-driven action generation technologies typically use speech signals and optional text content as input to predict sequences of bodily movements that match the speech rhythm, emotion, and semantic content. Currently, mainstream technical solutions can be broadly categorized as follows: One type of approach is based on retrieval-enhanced generation. For example, the GestureHYDRA model (a hybrid modal generation system for semantic co-speech gesture synthesis) achieves semantically consistent action synthesis by combining retrieval and generation stages. Specifically, this method first retrieves candidate action segments most relevant to the input speech semantics from a large-scale speech-action database as prior references; then, guided by these retrieved segments, a generative model (such as a diffusion transformer) generates the final action sequence and performs temporal smoothing. The advantage of this type of method is its ability to leverage the richness of real actions in the database, but its effectiveness is highly dependent on a high-quality and large-scale action retrieval library. Furthermore, this approach often struggles to effectively model and control the personalized style of the generated actions, character identity features, and local details such as hands and faces, resulting in significant shortcomings in achieving individual differences and fine-grained control.

[0025] Another category is diffusion generation schemes based on multimodal conditional control. A typical example of this type is the GestureDiffuCLIP (Gesture Diffusion Model with CLIP Latents) model, which introduces a multimodal joint conditional generation framework. This method maps audio features along with semantic information such as text into a unified latent space (such as the CLIP latent space), and then uses a diffusion model to generate action sequences that are consistent with both speech rhythm and semantic content. By leveraging the powerful cross-modal alignment capabilities of the CLIP model, this type of method has made significant progress in achieving semantically driven action generation. However, these methods typically use only full-body skeletal keypoints or a single parametric human model as a unified representation of actions during modeling. This single-stage, single-representation prediction structure, while capable of generating the human body trunk or overall pose, lacks effective modeling at the level of details such as hands and face.

[0026] Existing speech-driven action generation methods primarily rely on speech or text features as input, lacking comprehensive constraints from multimodal conditions. This results in insufficient individual variability in generated actions, uncoordinated limb levels, and ambiguous semantic expression. Furthermore, traditional methods often employ single-stage, single-representation prediction structures, capable of generating only the main body posture or overall pose, lacking effective modeling of fine-grained movements such as those of the hands and face, making it difficult to achieve natural, nuanced speech-semantics and controllable trajectory linkage.

[0027] In response, this invention provides a digital human video generation method, which aims to construct a multimodal controllable generation system that integrates voice, image and action intent tags, and achieves fine-grained action generation from the whole body to the parts through a phased hierarchical prediction structure, thereby significantly improving semantic consistency, character consistency and action naturalness, and overcoming the above-mentioned defects.

[0028] It should be noted that this invention belongs to the field of virtual human generation and multimodal motion synthesis technology, and particularly relates to multimodal controllable motion generation technology based on speech perception, which can be applied to various scenarios such as virtual anchors, intelligent customer service, digital human content creation, online education, and film and television entertainment.

[0029] The execution entity of the method provided by this invention can be one or more computer programs deployed on a server, personal computer, mobile terminal, or other computing device. The user provides input information through a client or interface. After receiving the information, the execution entity runs the method described in this invention, ultimately generating and outputting a digital human video that meets the user's requirements, features natural and smooth movements, and is rich in expressiveness.

[0030] It should be noted that all steps in this invention involving the acquisition of users' personal information such as images, text, and voice are subject to strict compliance with relevant data protection laws and regulations and can only be executed after obtaining explicit authorization and consent from the user, in order to ensure the compliant application of the technology and the security of user privacy.

[0031] This invention provides a digital human video generation method, aiming to address the problems of insufficient personalization capabilities, poor multimodal semantic consistency, lack of action details, and weak controllability in existing digital human motion generation technologies. By fusing input information from multiple modalities and employing hierarchical motion parameter prediction, this method can generate digital human videos that are highly consistent with specific identity features, speech rhythm, and text semantics, and that are rich in action details and highly controllable. Figure 1 This is a flowchart illustrating the digital human video generation method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step S10: Obtain the reference image, the input text, and the speech signal corresponding to the input text.

[0032] Specifically, this step is the data preparation phase of the entire process, designed to collect the basic multimodal data needed to generate digital human videos.

[0033] A reference image is an image used to define the identity of the target digital human. Specifically, it can be any static image containing a person provided by the user, such as a frontal or side-view half-body photo, a full-body photo, etc. The core function of this image is to provide identity information about the person, such as body shape and facial features, for subsequent processes, so that the generated digital human maintains consistency in identity with the person in the reference image, achieving personalized customization, rather than using a preset, uniform virtual human model.

[0034] Input text refers to the language content that the target digital human is expected to express, i.e., dialogue or script. For example, it could be a news broadcast, a story, or a simple greeting.

[0035] The audio signal corresponding to the input text refers to the audio data that matches the content of the input text. In a specific example, this step can first receive the input text provided by the user, and then use text-to-speech (TTS) technology to synthesize the text into a corresponding audio signal. Of course, the present invention is not limited to this; the audio signal can also be an audio file directly recorded or provided by the user, whose content matches the input text. This audio signal is not only used for the sound track of the final video, but more importantly, the rhythm, rhyme, and pauses it contains will serve as an important basis for driving the digital human's movements.

[0036] Step S20: Perform multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, and obtain control condition features for controlling action performance.

[0037] Specifically, this step aims to transform the raw multimodal data into structured features that the model can understand and process. Among these, identity-conditional features are feature vectors extracted from reference images to uniquely identify and characterize a specific person. These features aim to capture personalized information about the person, such as skeletal structure, posture habits, and other identity-related visual semantic information. By extracting these features, the model can generate actions that conform to the specific person's persona, solving the problem of existing technologies relying on fixed identity IDs and failing to adaptively generate personalized actions. For example, this feature can capture the difference between a person with upright posture and one who habitually has a slight hunchback, thus reflecting this difference in subsequent action generation.

[0038] Action label features are extracted from the input text and used to guide the generation of body movements that conform to the semantics of the text. It's essentially transforming the intent in the text into a series of action instructions. For example, when the input text is "Hello everyone, I'm here to introduce this new product," the extracted action label features might correspond to semantic action labels such as "greeting" or "hand gestures" in a time series. The introduction of these features ensures that the generated movements are no longer aimless, random swings, but rather meaningful expressions closely related to the linguistic content.

[0039] Speech features are extracted from speech signals and are primarily used to control the rhythm, rate, and dynamics of generated actions. Prosodic information in speech signals, such as pitch, energy, speech rate, and pauses, is highly correlated with the rhythm of nonverbal behaviors (such as gestures and nodding) during human speech. Extracting speech features aims to capture this correlation, ensuring that the generated actions are precisely synchronized temporally with the speech, making the digital human's movements appear to naturally follow the rhythm of the speech, rather than simply lip-syncing.

[0040] Control condition features are externally introduced features designed to enhance the user's control over generated actions. These can include various forms of control signals. For example, a user can specify the starting posture of the first frame of a video, or the movement trajectory of a limb (such as a hand) within a specific time period. They can even set a parameter to adjust the overall magnitude of the generated action (e.g., whether the action is more exaggerated and unrestrained, or more restrained and subtle). Acquiring these features provides users with significant creative freedom and precise control, enabling them to meet more customized and diverse application needs.

[0041] Step S30: Input the identity condition features, the action label features, the voice features, and the control condition features into the pre-trained action generation model to predict the target action parameter sequence, which includes the body skeleton parameter sequence, the hand parameter sequence, and the facial parameter sequence.

[0042] It should be noted that the pre-trained action generation model is a deep learning model trained on a large amount of multimodal data. It has learned how to generate a complete, coordinated and natural action sequence based on various input conditional features.

[0043] Specifically, in this step, the model receives the above four features as inputs simultaneously. The model will comprehensively consider: who should perform this action (identity condition feature), what type of action should be performed (action label feature), what is the rhythm of the action (speech feature), and whether certain specific control requirements need to be followed (control condition feature).

[0044] The target action parameter sequence predicted by the model is a mathematical description of the motion state of the digital human body in each frame. Preferably, this parameter sequence can be hierarchical, including body skeleton parameter sequences, hand parameter sequences, facial parameter sequences, etc. Among them, the body skeleton parameter sequence defines the global posture and position of the main body and limbs, and is the skeleton of the action; the hand parameter sequence finely describes the bending and extending of the fingers, which is the key to realizing rich gesture expression; the facial parameter sequence defines the movement of facial muscles in detail, including mouth shape, eye expression, eyebrows, and facial expressions, which is crucial for emotional expression and lip-syncing.

[0045] Through this multi-level parameter representation from the whole to the part, the model can generate movements that are coordinated as a whole and rich in detail, avoiding the problem of disconnect between body movements and hand and facial expressions that often occurs in traditional methods.

[0046] Step S40: Generate a target digital human video based on the target action parameter sequence.

[0047] Specifically, in this step, a video rendering or generation module receives the target motion parameter sequence predicted in the previous step. This module applies these abstract parameter sequences to a 3D digital human model and renders them frame by frame, ultimately synthesizing a continuous, visualized target digital human video. The digital human in this video not only appears identical to the reference image, but its movements and expressions are also highly synchronized, natural, and expressive with the input text and speech content. This embodiment of the invention acquires and fuses multimodal conditional features such as identity, text, speech, and control, and utilizes a powerful generative model to predict hierarchical body, hand, and facial motion parameters, ultimately generating a highly personalized, semantically consistent, rhythmically synchronized, and detail-rich digital human video.

[0048] The method provided in this invention extracts identity-related features from user-provided reference images, ensuring that the generated actions match the features of the person in the reference image. This solves the problem of existing technologies being unable to generate personalized actions, enhancing the flexibility of the application and user customization capabilities. Secondly, this invention not only extracts action label features and speech features from speech and text, but also introduces additional control-related features, allowing users to explicitly guide and intervene in the expression of actions, thereby achieving more accurate and richer semantic expression and higher controllability. Furthermore, this invention explicitly divides the predicted target action parameter sequence into three parts: the body skeleton, hands, and face. This hierarchical and refined prediction method ensures that while generating coherent full-body movements, it also takes into account the fine dynamics of local areas such as the hands and face, resulting in a significant improvement in the coordination, naturalness, and detail of the final digital human video.

[0049] Based on the above embodiments, the action generation model includes a cascaded first generation model and a second generation model. Accordingly, step S30 specifically includes: Step S31: Input the identity condition features, the action label features, the voice features and the control condition features into the first generation model to obtain the body skeleton parameter sequence output by the first generation model; Step S32: Extract the hidden body features output by the first generation model, and input the hidden body features, the identity condition features, the action label features, the voice features, and the control condition features into the second generation model to obtain the hand parameter sequence and face parameter sequence output by the second generation model.

[0050] It should be noted that, in order to achieve refined motion generation from coarse to fine and from the whole to the part, the motion generation model in this embodiment of the invention adopts a cascaded generation architecture, which includes a first generation model and a second generation model. This staged prediction framework decomposes the complex whole-body motion generation task into two related but distinct sub-tasks, thereby improving the coordination and detail of the final generated motion.

[0051] Specifically, after obtaining various conditional features, these features are input into the motion generation model in stages. First, the first generation model generates the body skeleton motion, and then the second generation model generates the fine motion parameters of the hands and face.

[0052] The first generative model, also known as the first-stage Body Module or the basic skeleton prediction model, focuses on generating the main framework of a digital human's movements. It receives all multimodal conditional features extracted from preceding steps, including identity-defining features, action label features guiding semantic actions, speech features controlling rhythm, and control conditional features for precise control. Based on these integrated inputs, the first generative model focuses on predicting the sequence of body skeleton parameters. This sequence defines the posture and position of the body trunk and limbs in each frame, forming the foundation and skeleton of the entire movement. The model's goal is to generate global body dynamics that are synchronized with speech rhythm, conform to textual semantics, and possess the style of a specified character.

[0053] The second generative model, also known as the second-stage Hand & Face Module, focuses on refining local details based on the already generated main action framework. The body hidden layer features are high-dimensional feature vectors extracted from the intermediate layers (e.g., the last layer) of the first generative model's network. These features not only contain the generated body skeleton action information but also a comprehensive understanding of all input conditions used to generate the action. This feature is crucial for connecting the two models, ensuring that the generation of hand and facial details is based on the overall body movement, thus avoiding inconsistencies between body, hand, and facial movements.

[0054] The hidden body features, along with all the original multimodal conditional features (i.e., identity, action, speech, control, etc.), are then fed back into the second generative model. This is crucial because the generation of gestures and facial expressions depends not only on body posture but also, more directly, on the emotion of the speech and the specific semantics of the text. For example, a nodding gesture, combined with different speech emotions and text content, will result in drastically different facial expressions and gesture details.

[0055] After receiving this rich conditional information, the second generative model performs inference and outputs hand parameter sequences and facial parameter sequences respectively. The hand parameters can be parameterized descriptions of hand models such as MANO, while the facial parameters can be parameterized descriptions of facial models such as PD-FGC, including components such as pose, emotion, eyes, and lips.

[0056] Through this cascaded and hierarchical prediction design, the embodiments of the present invention decouple the complex task of generating full-body movements. The first generation model is responsible for generating global, large-scale body movements, while the second generation model, based on this, focuses on generating high-frequency, fine local details, thereby improving the overall coordination and local expressiveness of the generated movements, making the final digital human video more vivid and natural.

[0057] Based on any of the above embodiments Figure 2 This is the overall network architecture diagram of the action generation model provided by the present invention, such as... Figure 2 As shown, the user inputs a reference image and text. The text is used by a trained text action prediction model to generate semantically relevant frame-level intent action labels. Subsequently, speech features are extracted from the speech signal in real time streaming, and human skeleton and semantic features are extracted from the reference image. These are then combined with action labels, keyframes, trajectory conditions, and L1-Div conditions as multimodal inputs. All conditions are input into a DiT (Diffusion Transformer)-based action generation model in stages. The first stage generates body skeleton actions, and the second stage generates fine-grained hand and facial action parameters. During inference, noise is progressively denoised in fragment form, and overlapping frames are smoothly connected to ultimately obtain a digital human video with consistent speech and semantics and controllable action trajectories. The specific implementation steps of feature extraction, body skeleton action generation, and hand and facial parameter generation will be described in detail below.

[0058] Based on the above embodiments, step S20, which involves performing multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, includes: Step S21: Detect limb key points and hand key points in the reference image, and encode the limb key points and hand key points to obtain skeleton features; Step S22: Extract the semantic features of the reference image and fuse the skeleton features with the semantic features to obtain the identity condition features.

[0059] Specifically, the process of generating identity condition features aims to extract composite features containing skeletal structure and semantic information from a single reference image provided by the user, which are sufficient to represent the unique identity of the person. First, limb key points and hand key points in the reference image are detected and encoded to obtain skeletal features.

[0060] Here, key points refer to specific anatomical locations on the human body used to describe posture and movement, such as joints like the shoulder, elbow, and wrist, as well as the various joints of the fingers. For example... Figure 2 As shown, in a preferred embodiment, a pose estimation algorithm such as DWPose can be used to process the input reference image (e.g., a frontal half-body or full-body photograph) to detect 18 limb keypoints and 42 hand keypoints. For each detected keypoint, its two-dimensional coordinates on the image can be obtained. and a confidence score .make: Understandably, a confidence threshold can be set to ensure the stability of the features. Key points with confidence levels above the threshold are retained, and their confidence levels are identified. Set the keypoints to 1, and set them to 0 or discard them if they are below the threshold, thus forming a reliable set of keypoints. .

[0061] Subsequently, a scaling factor was calculated. Translation vector The keypoint coordinates are aligned from the original image space to a preset reference coordinate system (i.e., a reference skeleton map) to eliminate the influence of different shooting distances and positions. Finally, all aligned keypoint coordinates are encoded, for example, flattened into a 180×1 one-dimensional vector, and mapped through a multilayer perceptron (MLP) network to obtain the skeleton features that can characterize the body structure of the person. .

[0062] Next, semantic features are extracted from the reference image, and the skeleton features are fused with the semantic features to obtain the identity condition features. Here, semantic features refer to high-level, abstract conceptual information extracted from the image. They transcend the low-level pixels and structure, describing the content and style of the image, such as a person's clothing, temperament, and background environment. In a specific embodiment, a pre-trained multimodal large model (e.g., the CLIP model) can be used to encode the reference image and extract its high-dimensional semantic features. .

[0063] Finally, the skeleton features obtained in the previous step Semantic features extracted in this step The fusion process can be performed in various ways. For example, two feature vectors can be concatenated along their channel dimensions (channel-wise concatenation), and then the concatenated long vector can be input into another MLP network for dimensionality reduction and information integration to obtain the final identity conditional features (also known as ID conditions, denoted as ) of the reference image. ).

[0064] In this embodiment of the invention, by combining structured skeleton features with high-level semantic features, the resulting identity condition features not only include the body skeleton information of the person, but also incorporate personalized semantic information such as appearance and style, forming a more comprehensive and robust representation of the person's identity, laying a solid foundation for generating actions with a high degree of consistency between the person and the character.

[0065] Step S23: Predict the action intent of the input text, generate frame-level action tags, and encode the frame-level action tags to obtain the action tag features.

[0066] Specifically, the generation of action label features aims to transform the action intent implied in the input text into a feature sequence usable by the model. Specifically, by predicting action intent from the input text, frame-level action labels can be generated, and these frame-level action labels are then encoded to obtain action label features. For example, for the text "I feel very surprised," the action intent prediction model analyzes its emotion and semantics and generates a label sequence corresponding to the video frame rate, which might include labels such as "widening eyes," "slightly leaning back," and "raising hand to cover mouth." Subsequently, these textual labels are converted into numerical action label features through encoding (e.g., embedding layers or MLPs).

[0067] Furthermore, step S23 specifically includes: Step S231: Based on the trained text action prediction model, the input text is used to predict semantically related frame-level action labels. Step S232: Decode the frame-level action tag into a one-hot encoded action tag, and map the one-hot encoded action tag and add it to the relative position code to obtain the action tag feature.

[0068] Here, the trained text action prediction model can be a Large Language Model (LLM) that has undergone supervised fine-tuning (SFT) on a high-quality text-semantic action pairing dataset. This targeted fine-tuning can significantly improve the accuracy of the LLM in predicting contextually and semantically appropriate action intentions from natural language text. For example, when the model receives input text, it predicts the most appropriate action label frame by frame or at small time steps. The prediction result can be in a structured label format, such as XML style. <action1>< / action1> .

[0069] Next, as Figure 2 As shown, frame-level action tags are decoded into one-hot encoded action tags, and then mapped using an MLP to obtain features. Adding the relative position embedding to the action label feature (also known as the action condition, denoted as ) yields the action label feature. This process involves processing the raw labels obtained in the previous step to make them more suitable numerical features for input into the action generation model.

[0070] Understandably, one-hot encoding is a widely used feature representation method. Assuming the model predefines N possible action categories (such as "waving," "nodding," "pointing," etc.), for each frame of the video, its action label is converted into an N-dimensional vector. In this vector, the value in the dimension corresponding to the action category of the current frame is 1, while the values ​​in all other dimensions are 0. This method clearly represents the specific action in each frame.

[0071] Furthermore, adding the one-hot encoded action tags to the relative positional encoding after MLP mapping helps the model better understand the action's context. Specifically, the sparse one-hot encoded vector is first transformed into a denser, lower-dimensional action embedding vector through a mapping layer (e.g., an MLP network). Simultaneously, the system generates a relative positional encoding vector, which represents the temporal position of the current frame within its action segment or the entire sequence. Finally, adding the action embedding vector to the relative positional encoding vector yields the action tag features. .

[0072] The final action label features obtained in this way not only contain explicit instructions on what action should be performed in each frame (from one-hot encoding), but also incorporate contextual information about the position of the action in the time series (from relative position encoding). This enables the action generation model to generate more coherent, smooth, and logical action sequences.

[0073] Step S24: Segment feature extraction is performed on the speech signal to obtain the speech features.

[0074] Specifically, the process of generating speech features aims to extract acoustic features from speech signals that drive the rhythm and dynamics of actions. In particular, by segmenting and extracting features from the speech signal, speech features (also known as speech conditions, denoted as...) can be obtained. Considering the need for real-time generation and processing efficiency, the entire long audio file is typically not processed at once. Instead, the audio signal (i.e., the speech signal) is segmented into shorter segments, and features are extracted from each segment independently. This segmented processing approach enables this method to support streaming input, achieving real-time digital human interaction that allows users to speak and move simultaneously.

[0075] Furthermore, step S24 specifically includes: Step S241: Divide the voice signal into multiple audio segments according to a preset time window size; Step S242: If the length of the last audio segment is less than the preset time window size, the last audio segment is supplemented by repeating the audio. Step S243: Use an audio encoder to extract features from each audio segment, and process the features through a mapping layer to obtain the speech features.

[0076] Specifically, user-input text can be synthesized into speech using TTS technology to generate a corresponding 16k 25fps audio signal. Then, as... Figure 2 As shown, to achieve streaming processing and alignment with video frames, a trained Whisper-tiny model encoder can be used to extract features from audio chunks. This involves dividing the continuous speech signal into segments of fixed duration for feature extraction, with each segment's length being a preset time window size. Here, the preset time window size (denoted as len) is an important hyperparameter, and its selection is usually consistent with the video slice length used during model training to ensure feature synchronization in the temporal dimension. For example, if the video slice length used during model training is 4 seconds, then during inference, the speech signal should also be divided into 4-second audio segments.

[0077] When segmenting a speech signal, the length of the last segment may be insufficient for a complete time window. To ensure that all data input to the subsequent encoder has a uniform dimension, this tail segment needs to be padded. This invention employs a simple and effective method: repeating the audio content of this tail segment until its length reaches the preset time window size.

[0078] Finally, features of each audio segment are extracted using an audio encoder, and these features are mapped through several MLP layers to obtain the final speech features. Here, the audio encoder can be a pre-trained speech recognition or acoustic model; for example, a lightweight and efficient model like Whisper-tiny can be used as the encoder. For each (pasted) audio segment, the encoder extracts a feature vector that represents its acoustic and prosodic information.

[0079] Understandably, the mapping layer typically consists of one or more MLP networks. Features extracted directly from the audio encoder may not be perfectly suited for direct action generation in terms of dimensionality, numerical range, or feature space. Therefore, these raw features need to be input into the mapping layer for adaptation and transformation, ultimately yielding optimized speech features that match the action generation model.

[0080] Through the refined feature extraction process described above, the embodiments of the present invention can extract more accurate, structured features from different data modalities that have clear guiding significance for action generation, significantly improving the performance of subsequent action generation models in terms of personalization, semantic relevance, and rhythm synchronization.

[0081] Based on any of the above embodiments, the control condition features include keyframe trajectory features and dynamic diversity condition features. Accordingly, in step S20, obtaining the control condition features for controlling the action performance includes: Step S25: Obtain the specified keyframe and / or keypoint trajectory sequence, and encode the keyframe and / or keypoint trajectory sequence to obtain the keyframe trajectory feature. The keyframe trajectory feature is used to guide the starting orientation of the action and / or the motion path. Step S26: Obtain the dynamic diversity value as the dynamic diversity condition feature, which is used to adjust the amplitude and diversity of the generated actions; In the model training phase, the dynamic diversity value is calculated based on the deviation between the skeleton sequence of the training samples and the average skeleton.

[0082] It should be noted that, based on the basic motion generation, this embodiment of the invention further introduces control condition features, which aim to give users a higher level of creative freedom and precise control over the generated motion. It mainly consists of two parts: keyframe trajectory features and dynamic diversity condition features.

[0083] Specifically, when acquiring the control condition features used to control the performance of actions, the first step is to acquire the keyframe trajectory features, which provide users with the ability to directly intervene in the structure and path of the generated actions.

[0084] To obtain keyframe trajectory features, you can first acquire specified keyframes and / or keypoint trajectory sequences. Here, a keyframe refers to a complete pose that the digital human needs to achieve at a specific point in time, as specified by the user. For example, a user can specify a random keyframe in a video clip to control the return of the action as the first or last frame. Keypoint trajectory sequences offer a more refined control. Instead of defining a complete pose, the user can specify the movement path of one or more keypoints (such as the wrist or head) over a period of time to guide a part of the limb to move along a predetermined trajectory (i.e., trajectory conditions). These two control methods can be used individually or in combination, greatly facilitating the precise design of specific actions by the user.

[0085] Subsequently, the keyframe and / or keypoint trajectory sequences are encoded to obtain the keyframe trajectory features (also known as keyframe conditions, denoted as...). It should be understood that the acquired keyframe poses or keypoint trajectory sequences are usually in the form of coordinates, which cannot be used directly by the model. Therefore, an encoder (such as a multilayer perceptron MLP network) is needed to convert them into a fixed-dimensional feature vector, i.e., keyframe trajectory features. This feature is then fed into the action generation model as a strong guiding condition to ensure that the model strictly follows the pose or path constraints set by the user when generating actions.

[0086] Furthermore, control condition characteristics can also include dynamic diversity condition characteristics. For example... Figure 2 As shown, to obtain diversity condition features, dynamic diversity values ​​can be directly acquired and mapped through an MLP layer to obtain the corresponding features. Then, this feature is added element-wise to the feature at the current time step to obtain the dynamic diversity conditional feature (also known as the L1-Div condition, denoted as L1-Div). Here, the L1-Div value is a simple scalar value. During the inference (i.e., generation) phase, the user can directly set this value to adjust the amplitude and diversity of the generated actions. A larger L1-Div value results in greater action amplitude and diversity, and vice versa. The key to this feature is that the model has already learned the correlation between this value and action style during the training phase.

[0087] Specifically, during the model training phase, the dynamic diversity value can be calculated based on the deviation between the skeleton sequence of the training samples and the average skeleton. For example, it can be used to calculate an evaluation metric for the sequence of len image frames: in, Indicates the first The skeleton of a frame. This represents the average skeleton of `len` frames. This calculated value objectively quantifies the dynamic range of the action. By training the model with this value as a condition along with the action data, the model learns the mapping relationship between dynamic diversity values ​​and the dynamic range of actions. Therefore, during inference, the user only needs to provide a target value, and the model can generate actions of the corresponding style.

[0088] This invention achieves dual control over digital human motion generation by introducing a control mechanism that includes keyframe trajectory features and dynamic diversity condition features. The former provides control over the motion structure and path, while the latter provides macroscopic control over the overall style and dynamism of the motion. The combination of these two mechanisms enhances the flexibility, controllability, and expressiveness of the method.

[0089] Based on any of the above embodiments, the first generation model adopts a network architecture based on a diffusion converter, and the first generation model includes a cascaded first processing module and a second processing module; correspondingly, step S31 specifically includes: Step S311: Obtain an initial body noise sequence, which includes a known sequence of real motion frames and an actual sequence of image frames to be denoised; Step S312: The current time step features, the speech features, and the initial body noise sequence are concatenated, and the concatenated features and the identity condition features embedded through adaptive normalization are input into the first processing module to obtain the intermediate hidden layer features output by the first processing module; wherein, the first processing module re-embeds the speech features through a cross-attention mechanism. Step S313: After concatenating the intermediate hidden layer features with the action label features, the control condition features and the speech features in the channel dimension, the concatenation is input to the second processing module for feature decoding, and the body skeleton parameter sequence is predicted and output.

[0090] It should be noted that, in order to achieve high-quality generation results, the first generation model in this embodiment of the invention preferably adopts a network architecture based on a diffusion transformer (DiT). This model is further divided into a cascaded first processing module and a second processing module to achieve step-by-step injection of different modal conditions.

[0091] Specifically, when inputting multimodal conditions into the first generative model (i.e., the Body Module) to generate the body skeleton parameter sequence, an initial body noise sequence is first obtained, which is the starting point of the diffusion model. This initial body noise sequence is random noise sampled from a standard normal distribution. It should be understood that, to support video continuation or editing, this sequence can be a hybrid sequence, consisting of a known sequence of real motion frames (e.g., the first few frames of the video) and a sequence of image frames that actually need to be denoised (i.e., random noise). This setup allows the model to smoothly predict future actions based on historical actions.

[0092] For example, noise that follows a normal distribution can be initialized. ,in ,forward The frame was replaced with a known real motion frame, then The frames are those that actually need to be predicted and denoised.

[0093] Next, the first step of conditional distribution embedding is performed: modeling audio dependencies. This involves using the current time step representing the diffusion process. After projection through MLP, the image is copied to the len frame, and then conditionally... ,noise After being stitched together at the channel level, it is mapped to 256 dimensions, and then referenced to graph conditions (i.e., ID conditions). AdaLN (Adaptive Layer Normalization) is used for embedding. Rotary Position Embedding (RoPE) and Cross Local Attention modules are used to model temporal dependencies within local time windows, and finally processed by a network consisting of multiple DiT Blocks 1. Furthermore, to enhance the guiding role of speech, each DiT Block uses a cross attention mechanism to integrate speech features. The audio conditions are reinforced by re-embedding, allowing the model to dynamically focus on the most relevant parts of the speech signal in each frame of motion generation, achieving more accurate audio-motion synchronization. After processing by the first processing module, the output is the intermediate hidden layer feature. .

[0094] Understandably, in the reverse denoising process (i.e., the generation process) of the diffusion model, the model's task is to start with noisy data, iteratively estimate and remove noise, and gradually recover the original data. In this process, the time step (usually represented by the variable t) specifies the current stage of the denoising iteration, i.e., which step in the denoising process is currently in. The current time step is usually a discrete scalar integer. If it is directly input into the neural network, its representational power is limited, which is not conducive to model generalization and learning the smooth relationship between different time steps. Therefore, the current time step feature is the mapping of this discrete scalar time step t to a high-dimensional, continuous vector space, forming an information-rich feature representation (i.e., feature vector or embedding). Finally, the second step of conditional embedding is performed, namely, adding action labels, keyframes, L1-Div, and other conditions. Specifically, With conditions , , , After concatenation by channel and mapping to 256 dimensions, the data passes through RoPE, Cross Local Attention, and DiT Block 2, and finally, an MLP outputs the predicted body keypoint sequence. That is, the sequence of body skeleton parameters.

[0095] Through this phased and hierarchical condition injection strategy, the embodiments of the present invention enable the first generative model to process different types of information in an orderly manner, thereby achieving fine control over the generation process and ensuring a high degree of consistency between the final output body movements and the multimodal inputs.

[0096] Based on any of the above embodiments, the second generative model adopts a network architecture based on a diffusion transformer, and the second generative model includes a hand parameter prediction branch and a face parameter prediction branch; Accordingly, step S32 specifically includes: Step S321: After fusing the initial hand noise sequence with the body hidden layer features and multimodal conditional features, the result is input into the hand parameter prediction branch to obtain the hand parameter sequence output by the hand parameter prediction branch. Step S322: After fusing the initial facial noise sequence with the body hidden layer features and the multimodal conditional features, the result is input into the facial parameter prediction branch to obtain the facial parameter sequence output by the facial parameter prediction branch. The multimodal condition features include the identity condition features, the action tag features, the voice features, the control condition features, and the current time step features.

[0097] It should be noted that, similar to the first generative model, the second generative model in this embodiment of the invention also preferably adopts a network architecture based on a diffusion transformer (DiT). Moreover, this model contains two parallel branches, namely a hand parameter prediction branch and a face parameter prediction branch, to specifically model these two highly complex local actions with different kinematic characteristics, respectively.

[0098] Specifically, we first extract the features from the final output of the Body Module (i.e., the hidden features of the body). The model cascades hand parameter prediction and face parameter prediction branches, with both branches sharing multimodal conditions but having independent parameters. The hand parameter prediction uses Mano parameters, while the face parameter prediction uses PD-FGC parameters, including pose, emotion, eye, and lip components.

[0099] In the hand parameter prediction branch, noise following a normal distribution (i.e., the initial hand noise sequence) is initialized. , and conditions After concatenation at the channel level, the sequence is mapped to 256 dimensions. Following RoPE, Cross Local Attention, and DiT Block 3, the predicted hand parameter sequence is finally output via an MLP branch. .

[0100] In the facial parameter prediction branch, noise that follows a normal distribution is initialized (i.e., the initial facial noise sequence). PD-FGC conditions of the reference image Multimodal conditions After concatenation at the channel level, the data is mapped to 256 dimensions. Following RoPE, Cross Local Attention, and DiT Block 4, the predicted facial parameter sequences are finally output through four independent MLP branches. This sequence can correspond to the parameters of the PD-FGC facial model and can be further decomposed into multiple independent sub-sequences, such as the head pose parameter sequence. Eye movement parameter sequence Emotional expression parameter sequence And the sequence of lip movements (mouth shapes) synchronized with speech. .

[0101] By employing this parallel branching architecture based on shared context, the embodiments of the present invention achieve efficient decoupling of the complex detail generation task. Although the generation processes of hand and facial movements are independent of each other, they are both strictly conditioned by global body movements (through hidden body features) and the original multimodal context, thereby ensuring that the final generated digital human can perform complex movements with highly coordinated and unified body, gestures, and expressions, rich emotional expression, and accurate lip movements.

[0102] Based on any of the above embodiments, the training steps of the action generation model include: A first initial model is constructed, and the first initial model is trained based on a sample set labeled with action data to obtain the first generated model. The sample speech signals, sample reference images, and sample control conditions in the sample set are all trained using a conditional discard strategy. Construct a second initial model and freeze the parameters of the trained first generative model. Use the body hidden layer features output by the first generative model as conditional input to update and train the parameters of the second initial model to obtain the second generative model.

[0103] It should be noted that the embodiments of the present invention provide a detailed description of the training steps for the phased action generation model. By employing a phased training strategy, a first generation model responsible for generating the main action is trained independently first, and then a second generation model responsible for generating local details is trained based on this first model. This hierarchical training method ensures that the model can stably and efficiently learn complex action patterns from global to local perspectives.

[0104] Specifically, the overall model training is divided into two stages. The first stage trains the first generative model (i.e., BodyModule). The goal of this stage is to obtain a first generative model that can generate high-quality, highly coordinated body skeletal movements based on multimodal conditions.

[0105] Specifically, first, a first initial model is constructed. This means initializing a model instance with a first generative model network architecture (e.g., a model based on a diffusion transformer), where all trainable parameters (such as weights and biases) are initialized with random values. Then, this first initial model is trained on a sample set of labeled action data to obtain a first generative model. It should be understood that the training sample set is a large-scale, high-quality multimodal dataset, where each sample contains at least: a reference image, a piece of text, the corresponding speech signal, optional control conditions (such as keyframes), and a precisely corresponding sequence of real body skeleton action parameters. During training, the model learns to gradually denoise and reconstruct the realistic body skeleton action from the input conditions and noise.

[0106] In this training process, the denoising steps were set to 1000. Audio, reference images, speech signals, and L1-Div conditions were all trained using a conditional drop strategy to facilitate subsequent CFG (Classifier-Free Guidance) inference. The reference image could be a randomly selected frame from the sampled `len` frames. It should be understood that the core purpose of the conditional drop strategy is to enable the use of CFG technology during the inference phase. By allowing the model to learn both conditional generation (with conditional input) and unconditional generation (when conditions are dropped) simultaneously during training, a guidance coefficient can be used to control the interpolation between unconditional and conditional generation during inference. This provides users with strong control; for example, the guidance coefficient can be adjusted to strengthen or weaken the association between generated actions and speech rhythm, or to enhance or weaken the similarity to identity features in the reference image, thereby achieving a better balance between generation quality and diversity.

[0107] After sufficient training, once the model's performance converges on the validation set, a well-trained first generative model capable of stably generating main actions is obtained.

[0108] The second stage of model training builds upon the first-stage model to further train a second generative model (i.e., the Hand & Face Module) capable of generating detailed hand and facial features. Specifically, first, a second initial model is constructed, initializing a model instance with a second generative model network architecture (e.g., a model containing hand and face prediction branches), whose parameters are also randomly initialized.

[0109] In the second stage, the parameters of the first generative model trained in the first stage are frozen, and only the second generative model is trained. At this time, the first generative model is used as a fixed, pre-trained feature extractor. This ensures that the quality of the generated body movements is not disturbed or degraded by the training in the second stage, thereby improving the stability and efficiency of the entire training process.

[0110] Next, the body hidden features output by the first generative model are used as conditional input to update the parameters of the second initial model during training. In this stage, the multimodal inputs from the training sample set are first fed into the frozen first generative model to extract the body hidden features. Then, these body hidden features are used as a new condition, along with other original conditions (such as speech features, identity features, etc.), and input into the second generative model. The goal of the second generative model is to learn to predict the true sequence of hand and facial parameters, given the body action context and all other conditions. The training loss function can be calculated based on the prediction errors of the hand and facial parameters, and the gradient is only fed back to the second generative model to update its parameters. Once the performance of the second generative model converges, the final trained second generative model is obtained.

[0111] By combining the aforementioned strategies of phased training, parameter freezing, and condition discarding, this embodiment of the invention not only efficiently trains complex cascaded models but also endows the models with the ability to flexibly control them during inference. This training method decouples the learning of global actions from the learning of local details, ensuring that the generated actions are coordinated as a whole and rich in detail, which is a key guarantee for achieving high-quality digital human video generation.

[0112] Based on any of the above embodiments, step S40 specifically includes: Step S41: Using a sliding window mechanism, the long sequence to be generated is divided into multiple video segments with overlapping frames; Step S42: Perform progressive denoising inference on each video segment to obtain the corresponding motion parameter segment; Step S43: Perform mean smoothing on the overlapping frame regions between adjacent video segments and splice them together to obtain the target motion parameter sequence; Step S44: Perform video rendering based on the target action parameter sequence to generate the target digital human video.

[0113] It should be noted that when generating target digital human videos, since generative models based on architectures such as Transformer usually have a fixed context window length and cannot process sequences of arbitrary length at once, this embodiment of the invention adopts an inference strategy based on sliding windows and overlapping smoothing to solve this problem.

[0114] Specifically, when a user needs to generate a video, such as one several minutes long, this long sequence far exceeds the processing capacity of a single model. The sliding window mechanism is designed to solve this problem. First, two parameters are defined: the window size (len) and the number of overlapping frames (m). The window size is consistent with the segment length used during model training. Then, sliding sampling is performed along the entire timeline to be generated, with a fixed step size, dividing the long sequence into multiple short video segments of fixed length with overlapping frames between adjacent segments. Next, each video segment is sequentially fed into the pre-trained motion generation model. For each segment, the model performs a complete stepwise denoising inference process (i.e., the inference process of the diffusion model) and finally generates the corresponding motion parameter segment for that segment. Subsequently, the overlapping frame regions between adjacent segments are smoothed using mean, achieving a smooth transition between adjacent segments. After all smoothing is completed, all processed segments (non-overlapping parts are used directly, and overlapping parts use the smoothed result) are concatenated in chronological order to obtain a complete, temporally continuous, and smoothly animated target motion parameter sequence. Finally, the obtained complete sequence of long motion parameters is input into the video rendering engine, which loads a 3D digital human model with the same identity as the reference image and applies the sequence of motion parameters to the model frame by frame to drive its movement, ultimately rendering and outputting a video of the target digital human with coherent and natural movements.

[0115] For example, to generate an infinitely long action sequence in real time, denoising is performed stepwise in 25 steps, with each video segment consisting of len frames and an overlapping window of m frames. The output includes the predicted skeleton point coordinate sequence, and the hand and face parameter sequences. .for m The overlapping windows of the frames are joined using mean smoothing. A video generation model is then cascaded on top of this to generate digital human videos with controllable speech, semantics, and actions.

[0116] The method provided in this invention extends a model limited by context length to application scenarios capable of generating videos of arbitrary length. The sliding window mechanism solves the processing challenge of long sequences, while mean smoothing of overlapping regions ensures temporal consistency and visual smoothness of the generated actions.

[0117] The digital human video generation apparatus provided by the present invention is described below. The digital human video generation apparatus described below can be referred to in correspondence with the digital human video generation method described above.

[0118] Based on any of the above embodiments Figure 3 This is a schematic diagram of the structure of the digital human video generation device provided by the present invention, as shown below. Figure 3 As shown, the device includes: The data acquisition unit 310 is used to acquire a reference image, input text, and the speech signal corresponding to the input text; The feature extraction unit 320 is used to perform multimodal feature extraction on the reference image, the input text and the speech signal to obtain identity condition features, action label features and speech features, and to obtain control condition features for controlling action performance. The parameter prediction unit 330 is used to input the identity condition features, the action label features, the voice features and the control condition features into a pre-trained action generation model to predict the target action parameter sequence, which includes a body skeleton parameter sequence, a hand parameter sequence and a facial parameter sequence. The video generation unit 340 is used to generate a target digital human video based on the target action parameter sequence.

[0119] The apparatus provided in this invention extracts identity feature characteristics from a user-provided reference image, ensuring that the generated actions match the features of the person in the reference image. This solves the problem of existing technologies being unable to generate personalized actions, enhancing the flexibility of the application and user customization capabilities. Secondly, this invention not only extracts action label features and speech features from speech and text, but also introduces additional control feature characteristics, allowing users to explicitly guide and intervene in the expression of actions, thereby achieving more accurate and richer semantic expression and higher controllability. Furthermore, this invention explicitly divides the predicted target action parameter sequence into three parts: the body skeleton, hands, and face. This layered and refined prediction method ensures that while generating coherent full-body movements, it also takes into account the fine dynamics of local areas such as the hands and face, resulting in a significant improvement in the coordination, naturalness, and detail of the final digital human video.

[0120] Based on any of the above embodiments, the feature extraction unit includes: The identity feature extraction subunit is used to detect limb key points and hand key points in the reference image, encode the limb key points and hand key points to obtain skeleton features; extract semantic features of the reference image, and fuse the skeleton features and semantic features to obtain the identity condition features; An action feature extraction subunit is used to predict the action intent of the input text, generate frame-level action tags, and encode the frame-level action tags to obtain the action tag features. The speech feature extraction subunit is used to extract segmented features from the speech signal to obtain the speech features.

[0121] Based on any of the above embodiments, the action feature extraction subunit is specifically used for: Based on the trained text action prediction model, semantically relevant frame-level action labels are obtained by predicting the input text. The frame-level action tag is decoded into a one-hot encoded action tag, and the one-hot encoded action tag is mapped and added to the relative position code to obtain the action tag feature.

[0122] Based on any of the above embodiments, the speech feature extraction subunit is specifically used for: The voice signal is divided into multiple audio segments according to a preset time window size; If the length of the last audio segment is less than the preset time window size, the last audio segment is supplemented by repeating the audio. The features of each audio segment are extracted using an audio encoder, and the features are then processed through a mapping layer to obtain the speech features.

[0123] Based on any of the above embodiments, the action generation model includes a cascaded first generation model and a second generation model, and the parameter prediction unit includes: The body skeleton prediction subunit is used to input the identity condition features, the action label features, the voice features and the control condition features into the first generation model to obtain the body skeleton parameter sequence output by the first generation model; The hand and face prediction subunit is used to extract the hidden body features output by the first generation model, and input the hidden body features, the identity condition features, the action label features, the voice features, and the control condition features into the second generation model to obtain the hand parameter sequence and face parameter sequence output by the second generation model.

[0124] Based on any of the above embodiments, the first generative model adopts a network architecture based on a diffusion transformer, and the first generative model includes a cascaded first processing module and a second processing module; correspondingly, the body skeleton prediction subunit is specifically used for: Obtain an initial body noise sequence, which includes a known sequence of real motion frames and a sequence of actual image frames to be denoised; The current time step features, the speech features, and the initial body noise sequence are concatenated, and the concatenated features and the identity condition features embedded through adaptive normalization are input into the first processing module to obtain the intermediate hidden layer features output by the first processing module; wherein, the first processing module re-embeds the speech features through a cross-attention mechanism. The intermediate hidden layer features, action label features, control condition features, and speech features are concatenated along the channel dimension and then input into the second processing module for feature decoding to predict and output the body skeleton parameter sequence.

[0125] Based on any of the above embodiments, the second generative model adopts a network architecture based on a diffusion transformer, and the second generative model includes a hand parameter prediction branch and a face parameter prediction branch; correspondingly, the hand and face prediction subunit is specifically used for: The initial hand noise sequence is fused with the body hidden layer features and multimodal conditional features, and then input into the hand parameter prediction branch to obtain the hand parameter sequence output by the hand parameter prediction branch. The initial facial noise sequence is fused with the body hidden layer features and the multimodal conditional features, and then input into the facial parameter prediction branch to obtain the facial parameter sequence output by the facial parameter prediction branch. The multimodal condition features include the identity condition features, the action tag features, the voice features, the control condition features, and the current time step features.

[0126] Based on any of the above embodiments, the device further includes a model training unit, the model training unit being used for: A first initial model is constructed, and the first initial model is trained based on a sample set labeled with action data to obtain the first generated model. The sample speech signals, sample reference images, and sample control conditions in the sample set are all trained using a conditional discard strategy. Construct a second initial model and freeze the parameters of the trained first generative model. Use the body hidden layer features output by the first generative model as conditional input to update and train the parameters of the second initial model to obtain the second generative model.

[0127] Based on any of the above embodiments, the control condition features include keyframe trajectory features and dynamic diversity condition features, and the feature extraction unit is further configured to: Obtain a specified keyframe and / or keypoint trajectory sequence, and encode the keyframe and / or keypoint trajectory sequence to obtain the keyframe trajectory feature, which is used to guide the starting orientation of the action and / or the motion path. The dynamic diversity value is obtained as the dynamic diversity condition feature, which is used to adjust the amplitude and diversity of the generated actions. In the model training phase, the dynamic diversity value is calculated based on the deviation between the skeleton sequence of the training samples and the average skeleton.

[0128] Based on any of the above embodiments, the video generation unit is specifically used for: A sliding window mechanism is used to divide the long sequence to be generated into multiple video segments with overlapping frames; Each video segment is subjected to progressive denoising inference to obtain the corresponding motion parameter segment; The overlapping frame regions between adjacent video segments are subjected to mean smoothing, and then spliced ​​together to obtain the target motion parameter sequence. Based on the target action parameter sequence, video rendering is performed to generate the target digital human video.

[0129] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a digital human video generation method, which includes: acquiring a reference image, input text, and a speech signal corresponding to the input text; performing multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, and acquiring control condition features for controlling action performance; inputting the identity condition features, the action label features, the speech features, and the control condition features into a pre-trained action generation model to predict a target action parameter sequence, the target action parameter sequence including a body skeleton parameter sequence, a hand parameter sequence, and a facial parameter sequence; and generating a target digital human video based on the target action parameter sequence.

[0130] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the digital human video generation method provided by the above methods. The method includes: acquiring a reference image, input text, and a speech signal corresponding to the input text; performing multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, and acquiring control condition features for controlling action performance; inputting the identity condition features, the action label features, the speech features, and the control condition features into a pre-trained action generation model to predict a target action parameter sequence, the target action parameter sequence including a body skeleton parameter sequence, a hand parameter sequence, and a facial parameter sequence; and generating a target digital human video based on the target action parameter sequence.

[0132] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the digital human video generation method provided by the above methods. The method includes: acquiring a reference image, input text, and a speech signal corresponding to the input text; performing multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, and acquiring control condition features for controlling action performance; inputting the identity condition features, the action label features, the speech features, and the control condition features into a pre-trained action generation model to predict a target action parameter sequence, the target action parameter sequence including a body skeleton parameter sequence, a hand parameter sequence, and a facial parameter sequence; and generating a target digital human video based on the target action parameter sequence.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating digital human videos, characterized in that, include: Acquire a reference image, input text, and the corresponding speech signal of the input text; Multimodal feature extraction is performed on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features, and control condition features for controlling action performance are obtained. The identity condition features, the action label features, the voice features, and the control condition features are input into a pre-trained action generation model to predict a target action parameter sequence, which includes a body skeleton parameter sequence, a hand parameter sequence, and a facial parameter sequence. Based on the target action parameter sequence, a target digital human video is generated.

2. The digital human video generation method according to claim 1, characterized in that, The step of performing multimodal feature extraction on the reference image, the input text, and the speech signal to obtain identity condition features, action label features, and speech features includes: The limb key points and hand key points in the reference image are detected, and the limb key points and hand key points are encoded to obtain skeletal features; Extract the semantic features of the reference image, and fuse the skeleton features with the semantic features to obtain the identity condition features; The input text is subjected to action intent prediction to generate frame-level action tags, and the frame-level action tags are encoded to obtain the action tag features; The speech signal is segmented and its features are extracted to obtain the speech features.

3. The digital human video generation method according to claim 2, characterized in that, The step of predicting action intent from the input text, generating frame-level action tags, and encoding the frame-level action tags to obtain the action tag features includes: Based on the trained text action prediction model, semantically relevant frame-level action labels are obtained by predicting the input text. The frame-level action tag is decoded into a one-hot encoded action tag, and the one-hot encoded action tag is mapped and added to the relative position code to obtain the action tag feature.

4. The digital human video generation method according to claim 2, characterized in that, The step of segmenting and extracting features from the speech signal to obtain the speech features includes: The voice signal is divided into multiple audio segments according to a preset time window size; If the length of the last audio segment is less than the preset time window size, the last audio segment is supplemented by repeating the audio. The features of each audio segment are extracted using an audio encoder, and the features are then processed through a mapping layer to obtain the speech features.

5. The digital human video generation method according to claim 1, characterized in that, The action generation model includes a cascaded first generation model and a second generation model. The step of inputting the identity condition features, the action label features, the speech features, and the control condition features into the pre-trained action generation model to predict the target action parameter sequence includes: The identity condition features, the action label features, the voice features, and the control condition features are input into the first generation model to obtain the body skeleton parameter sequence output by the first generation model; Extract the hidden body features output by the first generative model, and input the hidden body features, along with the identity condition features, the action label features, the speech features, and the control condition features, into the second generative model to obtain the hand parameter sequence and facial parameter sequence output by the second generative model.

6. The digital human video generation method according to claim 5, characterized in that, The first generative model adopts a network architecture based on a diffusion converter, and the first generative model includes a cascaded first processing module and a second processing module. The step of inputting the identity condition features, the action label features, the voice features, and the control condition features into the first generation model to obtain the body skeleton parameter sequence output by the first generation model includes: Obtain an initial body noise sequence, which includes a known sequence of real motion frames and a sequence of actual image frames to be denoised; The current time step features, the speech features, and the initial body noise sequence are concatenated, and the concatenated features and the identity condition features embedded through adaptive normalization are input into the first processing module to obtain the intermediate hidden layer features output by the first processing module; wherein, the first processing module re-embeds the speech features through a cross-attention mechanism. The intermediate hidden layer features, action label features, control condition features, and speech features are concatenated along the channel dimension and then input into the second processing module for feature decoding to predict and output the body skeleton parameter sequence.

7. The digital human video generation method according to claim 5, characterized in that, The second generative model adopts a network architecture based on a diffusion transformer, and the second generative model includes a hand parameter prediction branch and a face parameter prediction branch; The process involves extracting the hidden body features output by the first generative model, and inputting these hidden body features, along with the identity condition features, the action label features, the speech features, and the control condition features, into the second generative model to obtain the hand parameter sequence and facial parameter sequence output by the second generative model. The initial hand noise sequence is fused with the body hidden layer features and multimodal conditional features, and then input into the hand parameter prediction branch to obtain the hand parameter sequence output by the hand parameter prediction branch. The initial facial noise sequence is fused with the body hidden layer features and the multimodal conditional features, and then input into the facial parameter prediction branch to obtain the facial parameter sequence output by the facial parameter prediction branch. The multimodal condition features include the identity condition features, the action tag features, the voice features, the control condition features, and the current time step features.

8. The digital human video generation method according to claim 5, characterized in that, The training steps of the action generation model include: A first initial model is constructed, and the first initial model is trained based on a sample set labeled with action data to obtain the first generated model. The sample speech signals, sample reference images, and sample control conditions in the sample set are all trained using a conditional discard strategy. Construct a second initial model and freeze the parameters of the trained first generative model. Use the body hidden layer features output by the first generative model as conditional input to update and train the parameters of the second initial model to obtain the second generative model.

9. The digital human video generation method according to any one of claims 1 to 8, characterized in that, The control condition features include keyframe trajectory features and dynamic diversity condition features. The acquisition of control condition features for controlling motion performance includes: Obtain a specified keyframe and / or keypoint trajectory sequence, and encode the keyframe and / or keypoint trajectory sequence to obtain the keyframe trajectory feature, which is used to guide the starting orientation of the action and / or the motion path. The dynamic diversity value is obtained as the dynamic diversity condition feature, which is used to adjust the amplitude and diversity of the generated actions. In the model training phase, the dynamic diversity value is calculated based on the deviation between the skeleton sequence of the training samples and the average skeleton.

10. The digital human video generation method according to any one of claims 1 to 8, characterized in that, The step of generating a target digital human video based on the target action parameter sequence includes: A sliding window mechanism is used to divide the long sequence to be generated into multiple video segments with overlapping frames; Each video segment is subjected to progressive denoising inference to obtain the corresponding motion parameter segment; The overlapping frame regions between adjacent video segments are subjected to mean smoothing, and then spliced ​​together to obtain the target motion parameter sequence. Based on the target action parameter sequence, video rendering is performed to generate the target digital human video.

11. A digital human video generation device, characterized in that, include: The data acquisition unit is used to acquire a reference image, input text, and the speech signal corresponding to the input text. The feature extraction unit is used to perform multimodal feature extraction on the reference image, the input text and the speech signal to obtain identity condition features, action label features and speech features, and to obtain control condition features for controlling action performance. The parameter prediction unit is used to input the identity condition features, the action label features, the voice features and the control condition features into a pre-trained action generation model to predict the target action parameter sequence, which includes a body skeleton parameter sequence, a hand parameter sequence and a facial parameter sequence. The video generation unit is used to generate a target digital human video based on the target action parameter sequence.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the digital human video generation method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the digital human video generation method as described in any one of claims 1 to 10.

Images