Construction of digital human driving model, digital human driving method, device and equipment

Abstract

The invention provides construction of a digital human driving model, a digital human driving method and device, equipment, a medium and a program product, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC and digital human. The specific implementation scheme is as follows: acquiring an original sample; in the action frame sequence of the original sample, covering a set number of action frames according to a random covering rule to obtain a covered action frame sequence; adding the masking action frame sequence into the original sample to form a training sample; and training the initial action model by using the training sample to obtain a digital human-driven model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generated Content) and digital humans. It specifically relates to a method for constructing a digital human-driven model, a digital human-driving method, a device for constructing a digital human-driven model, a digital human-driving device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of AIGC technology, digital humans, as a new form of human-computer interaction, are receiving more and more attention.

[0003] Related technologies can improve the interactivity of digital humans by using voice to drive their facial and body movements. However, as the field of digital human interaction continues to expand, existing voice-driven digital human movement technologies can no longer meet the growing demand for more efficient and convenient digital human driving. Summary of the Invention

[0004] The present disclosure provides a method for constructing a digital human-driven model, a digital human-driving method, a device for constructing a digital human-driven model, a digital human-driving device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a method for constructing a digital human driving model is provided, comprising:

[0006] Obtaining original samples, wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0007] In the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence;

[0008] Adding the masking action frame sequence to the original sample to form a training sample;

[0009] The initial motion model is trained using the training samples to obtain a digital human driving model.

[0010] According to another aspect of the present disclosure, a digital human driving method is provided, comprising:

[0011] Obtaining a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions;

[0012] generating a target speech feature frame sequence matching the target speech, and constructing an unknown action frame sequence matching the target speech feature frame sequence;

[0013] Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence;

[0014] Inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence that matches the target speech;

[0015] The digital human driving model is obtained by training using the method for constructing a digital human driving model as described in any one of the embodiments of the present disclosure.

[0016] According to one aspect of an embodiment of the present disclosure, there is also provided an apparatus for constructing a digital human-driven model, comprising:

[0017] The original sample acquisition module is used to obtain the original sample; wherein the original sample includes a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0018] a masking processing module, configured to mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence;

[0019] A training sample acquisition module, configured to add the masking action frame sequence to the original sample to form a training sample;

[0020] The model training module is used to train the initial action model using the training samples to obtain a digital human driving model.

[0021] According to one aspect of an embodiment of the present disclosure, a digital human driving device is provided, comprising:

[0022] a broadcast voice acquisition module, configured to acquire a target voice for voice-driven 2D body movements of a target virtual image, wherein the target virtual image includes a head and body regions;

[0023] A first sequence construction module is used to generate a target speech feature frame sequence matching the target speech, and to construct an unknown action frame sequence matching the target speech feature frame sequence;

[0024] A second sequence acquisition module is configured to fill at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence;

[0025] A third sequence acquisition module is configured to input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence that matches the target speech;

[0026] The digital human driving model is obtained by training using the method for constructing a digital human driving model as described in any one of the embodiments of the present disclosure.

[0027] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0028] at least one processor; and

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a digital human driven model or the digital human driving method as described in any one of the embodiments of the present disclosure.

[0031] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method for constructing a digital human-driven model or the method for driving a digital human according to any one of the embodiments of the present disclosure.

[0032] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0034] Figure 1 is a schematic diagram of a method for constructing a digital human driving model provided according to an embodiment of the present disclosure;

[0035] Figure 2 is a schematic diagram of another method for constructing a digital human driving model provided according to an embodiment of the present disclosure;

[0036] Figure 3 is a schematic diagram of a digital human driving method provided according to an embodiment of the present disclosure;

[0037] Figure 4 is a schematic diagram of another digital human driving method provided according to an embodiment of the present disclosure;

[0038] Figure 5 is a schematic diagram of an action frame applicable to an embodiment of the present disclosure;

[0039] Figure 6 This is a schematic diagram of a first-frame voice-driven 2D full-body motion applicable to an embodiment of the present disclosure;

[0040] Figure 7 is a general voice-driven 2D full-body motion diagram applicable to embodiments of the present disclosure;

[0041] Figure 8 is a structural diagram of a device for constructing a digital human-driven model according to an embodiment of the present disclosure;

[0042] Figure 9 is a structural diagram of a digital human driving device provided according to an embodiment of the present disclosure;

[0043] Figure 10 It is a block diagram of an electronic device used to implement the method for constructing a digital human driving model or the digital human driving method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] Figure 1 This is a schematic diagram of a method for constructing a digital human-driven model, according to an embodiment of the present disclosure. This embodiment is applicable to constructing a mapping relationship between speech and 2D body movements based on video and training a neural network model. This method can be performed by a device for constructing a digital human-driven model, which can be implemented using hardware and / or software and is generally configured in an electronic device.

[0046] Correspondingly, such as Figure 1 As shown, the method may specifically include:

[0047] S110: Obtain original samples.

[0048] The original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frames include 2D head and limb key points.

[0049] An action frame refers to a single image or image segment that constitutes an action sequence, which records the movement of the human body over a period of time, and each frame represents a moment of the action composed of various key points. Among them, the action frame can be used to describe the position information of the 2D head and limb key points in the image when the human body performs a set action (that is, the position coordinates of the key points). The limb key points may only include key points in the upper limb area, or may also include key points in the upper limb and lower limb areas at the same time. Key points refer to the 2D key point coordinates extracted from portrait video data, which are used to represent various parts of the human body and capture the movements and motion paths of the human body. Among them, the collection parts of key points may include the face (such as eyes, ears, mouth, eyebrows and nose, etc.), hands (such as wrists, fingers and palms, etc.) and limbs (such as shoulders, elbows, upper arms and forearms, etc.).

[0050] A speech feature frame is a discrete data unit extracted from a continuous speech signal and used to represent speech characteristics. The speech feature frame specifically includes acoustic features (e.g., spectrum, pitch, and tone) and speech features of a speech segment.

[0051] Optionally, the action frames can be obtained by a pre-trained 2D key point recognition model, and the speech feature frames can be extracted from the speech signal by a pre-trained speech feature extraction model, such as Wav2Vec 2.0.

[0052] The time-aligned action frame sequence and speech feature frame sequence can be understood as synchronized speech and picture in a video. In this embodiment, the number of action frames included in the action frame sequence is consistent with the number of speech feature frames included in the speech feature frame sequence.

[0053] S120 . Mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence.

[0054] The random masking rule can be understood as randomly selecting a portion of the action frames in the sequence and masking (or covering) them, such as replacing the pixel values ​​of each pixel in the action frame with zero or other values. The set number refers to the number of action frames masked according to the random masking rule. This number can be based on a percentage of the total sequence length or a fixed number of frames.

[0055] Specifically, a time-aligned sequence of action frames containing 2D head and limb keypoints and a sequence of speech feature frames are obtained as raw samples. A set number of action frames are masked according to a random masking rule. For example, a masking mechanism randomly masks 50% of the input action frames to generate a masked action frame sequence. By masking the action frames, a diverse sequence of masked action frames can be constructed as training samples for model training, helping to improve the model's generalization capabilities.

[0056] Through the above settings, a masked action frame sequence corresponding to each action frame sequence in each original sample can be generated.

[0057] S130: Add the masking action frame sequence to the original sample to form a training sample.

[0058] It can be understood that a training sample contains three pieces of information, namely, a time-aligned action frame sequence and a speech feature frame sequence, and a masked action frame sequence obtained by randomly masking one or more action frames in the action frame sequence.

[0059] S140: Use the training samples to train the initial motion model to obtain a digital human driving model.

[0060] Specifically, the masked action frame sequence is added to the original sample to form a training sample including the masked action frame sequence, the action frame sequence and the speech feature frame sequence, and the initial action model is trained using the training sample.

[0061] In an optional implementation of this embodiment, the initial action model may be a Transformer neural network model.

[0062] The input data includes preprocessed action frame sequences, masked action frame sequences, and extracted speech feature frame sequences. The model architecture consists of an input layer, a multi-layered encoder layer, a multi-layered decoder layer, and an output layer. The encoder layer encodes the known actions, while the decoder layer learns the correlation features between the actions and speech. During training, the encoder layer output serves as the decoder layer input. A self-attention mechanism and an encoder-decoder attention mechanism are used to generate a prediction sequence. A loss, such as diffusion loss, is calculated between the predicted output and the true keypoint coordinates. Backpropagation and an optimizer are used to update the model weights to minimize the loss. Training is performed in batches and over multiple epochs, with the model gradually improving its feature representation and reducing the loss in each epoch. Furthermore, a validation set is used to monitor model performance and prevent overfitting, and hyperparameters are adjusted based on performance. Finally, when optimal performance on the validation set is achieved, the model parameters are saved for subsequent testing and application, resulting in a digital human-driven model.

[0063] It should be noted that related technologies for voice-driven full-body movements primarily employ 3D motion representation. These methods directly generate corresponding three-dimensional motion sequences from speech to drive the full-body movement of the digital human. During the implementation of this invention, the inventors discovered the following drawbacks in related technologies: Acquiring 3D motion representations presents certain difficulties. Furthermore, when 3D motions are rendered onto a two-dimensional surface, they can become distorted or unnatural, impacting the final visual effect and user experience.

[0064] Based on this, the technical solution of the embodiment of the present disclosure is as follows: obtaining an original sample; in the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence; adding the masked action frame sequence to the original sample to form a training sample; using the training sample to train the initial action model to obtain a digital human driving model. The implementation method of training the digital human driving model by using an action frame sequence including 2D action representations, randomly masking and combining with a voice feature frame sequence to form a training sample can simply and conveniently train a machine learning model for driving 2D limb movements, effectively reducing the construction complexity and resource consumption of the training samples, and enriching the diversity of the training samples to a certain extent, improving the training efficiency and reliability of the digital human driving model, and improving the realism of the generated driving movements.

[0065] On the basis of the above embodiments, in addition to being able to train a digital human driving model based on training samples including masked action frame sequences, action frame sequences and voice feature frame sequences, the embodiments of the present disclosure can also perform personalized model training on the original samples according to the application scenarios or the types of emotions contained in the samples, because different emotional states will be expressed through voice features and action features. For example, entertainment news corresponds to original samples with light-hearted and happy emotions, traditional news broadcasts correspond to original samples with neutral or serious emotions, natural disasters, accidents or tragic events correspond to original samples with sad emotions, and sports events correspond to original samples with tense and excited emotions, etc., so that the trained model is more targeted and the reliability of action prediction of the digital human driving model in different application scenarios is improved.

[0066] That is, original samples of different voice emotions can be grouped and independently trained to adapt to digital human-driven models of different voice emotions, so as to further improve the scene-specificity of the digital human-driven model and improve the final visual presentation effect and user experience.

[0067] Figure 2It is a schematic diagram of another method for constructing a digital human driving model provided in accordance with an embodiment of the present disclosure. This embodiment concretizes the operation of "obtaining original samples" in the above embodiment as follows: "obtaining a real-person video, and sampling the image and voice of the real-person video at a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence; performing 2D key point recognition in each image frame, and forming an action frame corresponding to each of the image frames according to the recognition results; performing voice feature extraction in each voice frame, and forming a voice feature frame corresponding to each of the voice frames according to the extraction results; forming the original sample corresponding to the real-person video according to the action frame sequence obtained by organizing each of the action frames and the voice feature frame sequence obtained by organizing each of the voice feature frames."

[0068] Correspondingly, such as Figure 2 As shown, the method may specifically include:

[0069] S210 , obtaining a live video, and performing image and voice sampling on the live video according to a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence.

[0070] In this embodiment, the time-aligned motion frame sequence and speech frame sequence refer to the motion frame sequence and speech frame sequence collected by using the same sampling rate for video images and audio, for example, both are 25 frames / s.

[0071] By temporally aligning the motion features and speech features, and using the associated features to train the model, the model can better understand and predict motion sequences, thereby improving the accuracy and naturalness of motion prediction.

[0072] In an optional implementation of this embodiment, in the real-person video, only the upper body area of ​​the real person is displayed.

[0073] Specifically, by focusing on the upper body area of ​​a real person, the model can more accurately capture and learn the announcer's movement characteristics, such as facial expressions and upper body movements, thereby improving training efficiency and model performance, improving the quality of movement synchronization and expression simulation, enhancing the authenticity and naturalness of digital humans in various application scenarios, and improving visual effects and user experience.

[0074] S220 , performing 2D key point recognition in each image frame, and forming an action frame corresponding to each image frame according to the recognition result.

[0075] Specifically, by using a pre-trained key point estimation model to perform 2D key point recognition in each image frame, 2D key points including the head and limbs can be recognized in each image frame to form action frames corresponding to each of the image frames.

[0076] S230 , extracting speech features in each speech frame, and forming speech feature frames corresponding to each speech frame according to the extraction results.

[0077] Specifically, a pre-trained speech feature extraction model can be used to extract speech features from each speech frame to form a speech feature frame corresponding to each speech frame. Optionally, each speech feature frame can be understood as a feature sequence composed of multiple speech features.

[0078] S240. Form the original sample corresponding to the live-action video based on the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

[0079] S250 : Mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence.

[0080] In an optional implementation of this embodiment, in the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence may include:

[0081] Randomly select the target probability value within the preset probability interval;

[0082] Obtaining the total number of action frames contained in the action frame sequence of the original sample;

[0083] Calculating the masked number of action frames according to the total number of action frames and the target probability value;

[0084] In the action frame sequence of the original sample, action frames matching the mask quantity are randomly selected for masking to obtain a masked action frame sequence.

[0085] Specifically, a set number of action frames are masked according to a random masking rule, and the set number can be randomly selected according to the total number of action frames and a set optional interval range.

[0086] Within a preset probability interval, a target probability value is randomly selected, for example, the preset probability interval is [30%, 70%]. For each original sample, a target probability value is randomly selected, for example, 50%. The total number of action frames contained in the action frame sequence of the original sample is obtained, for example, 200 frames. According to the total number of action frames and the target probability value, the masked number of action frames is calculated. The masked number of action frames = the total number of action frames × the target probability value. In the action frame sequence of the original sample, each action frame that matches the masked number is randomly selected for masking. For example, the masked number of action frames is 200 frames × 50% = 100 frames. A total of 100 action frames at any position are randomly selected for masking to obtain a masked action frame sequence.

[0087] By masking the action frames in random numbers and positions in the action frame sequence, the data scale is artificially expanded without actually increasing the dataset, providing the model with more training samples, thereby improving the model's performance. Due to randomness, the model needs to pay more attention to timing information, learn the dependencies between frames, and reduce the model's excessive dependence on specific training samples, thereby reducing the risk of overfitting and improving the reliability of the digital human-driven model.

[0088] S260: Add the masking action frame sequence to the original sample to form a training sample.

[0089] S270: Use the training samples to train the initial action model to obtain a digital human driving model.

[0090] In an optional implementation of this embodiment, using the training samples to train the initial motion model to obtain the digital human driving model may include:

[0091] Inputting the training samples into the initial motion model;

[0092] Generating a first labeled embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second labeled embedding vector sequence corresponding to the action frame sequence;

[0093] wherein the first marker embedding vector sequence includes an unknown vector corresponding to the masked action frame;

[0094] Performing multi-level encoding on the first tag embedding vector sequence through a model encoder layer to obtain an encoding result sequence;

[0095] Performing multi-stage decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each of the unknown vectors;

[0096] Calculating a model loss value based on a prediction result vector for each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence;

[0097] The weight of the initial action model is updated according to the model loss value.

[0098] Optionally, the first labeled embedding vector sequence can be specifically understood as: the input masked action frame sequence is converted into an embedding vector form that the model can process (typically, Token embedding) through the Transformer model input layer. Specifically, in the model input layer, each action frame (i.e., key point coordinates) in the masked action frame sequence is projected into the embedding space through a projection matrix to obtain the labeled embedding vector. These embedding vectors can capture the connection between action frames. Similar action frames or action frames that often appear in succession have similar embedding vectors. These embedding vectors will be adjusted during the training process to better represent the characteristics of the action frames.

[0099] Specifically, each action frame in the masked action frame sequence (either an unmasked action frame or a masked action frame) corresponds to a first marker embedding vector. The difference is that an unmasked action frame corresponds to a known vector with a specific value, while a masked action frame corresponds to an unknown vector with an unspecified value.

[0100] The second marker embedding vector sequence can be specifically understood as: the input action frame sequence is converted into a numerical vector form that can be processed by the model through the Transformer model input layer. Since the second marker embedding vector sequence is generated by the action frame sequence, and each action frame in the action frame sequence is not masked, each second marker embedding vector in the second marker embedding vector sequence is a known vector, that is, the second marker embedding vector sequence contains known vectors corresponding to the masked action frames.

[0101] The encoder layer consists of a self-attention mechanism and a feedforward network, typically comprising multiple encoder layers. Each sublayer utilizes residual connections and layer normalization to stabilize training and improve performance, enabling the model to capture dependencies within the sequence. The encoded sequence can be understood as a sequence of results reflecting the internal relationships within the first token embedding sequence, obtained by performing multi-level encoding on the first token embedding sequence through the encoder layer. The decoder layer, typically comprising multiple decoders, combines information from the input sequence to predict an output sequence that matches the characteristics of the input sequence.

[0102] Specifically, training samples are input into the initial motion model. The model input layer generates a first sequence of labeled embedding vectors corresponding to the masked motion frame sequence, and a second sequence of labeled embedding vectors corresponding to the motion frame sequence. The model encoder layer performs multi-level encoding on the first sequence of labeled embedding vectors to obtain an encoded result sequence. The model decoder layer performs multi-level decoding on the sum of the encoded result sequence and the speech feature frame sequence, for example by directly adding the feature vectors of the two modalities to fuse the two feature information. The input is processed step by step through multiple decoder layers, and the prediction result is gradually refined to ultimately obtain a prediction result vector for each unknown vector. Based on the prediction result vector for each unknown vector and the known vectors corresponding to each unknown vector in the second sequence of labeled embedding vectors, a model loss value, such as diffusion loss, is calculated. The weights of the initial motion model are updated based on the model loss value. By using training samples formed by random masking of 2D motion representations and combining them with speech feature information, the model weight parameters are optimized through continuous training, verification, and tuning, bringing the model to an optimal state, thereby improving the accuracy of motion prediction and the reliability of the digital human driving model.

[0103] In another optional implementation of this embodiment, calculating the model loss value based on the prediction result vector for each unknown vector and the known vector corresponding to each unknown vector in the second label embedding vector sequence may include:

[0104] A diffusion loss function is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors.

[0105] The diffusion loss function measures the difference between the model's predicted masked motion and the actual motion. During model training, the model needs to learn how to recover the original motion data from the masked motion data. The diffusion loss function is used to guide this process, ensuring that the model's predictions are as close as possible to the actual motion data.

[0106] Specifically, during the digital human-driven training process, at each training step, we know the preceding and following (or contextual) action frames, as well as the true value of the current predicted action frame (the coordinates of each pixel in a frame of action). Therefore, we simply use the preceding and following action frames as constraints and the true value of the current predicted action frame to train the constrained diffusion model. The error function that implicitly describes the next action frame is called the diffusion loss function.

[0107] Compared to the most basic constrained denoising diffusion probability model (DDPM), this diffusion loss function incorporates constraints into its calculation formula. This constraint information can be understood as the output of the initial motion model for the preceding and following action frames in the action frame sequence. Because the constraints come from the initial motion model, the gradient of the diffusion loss function can be propagated back to the parameters of the initial motion model through this constraint information, enabling the model training process.

[0108] Specifically, a diffusion loss function is used to calculate the model loss between the predicted result vector and the known vector corresponding to each unknown vector. By minimizing the difference between the predicted masked motion and the actual motion, the model can better learn the distribution of data, thereby generating higher-quality predicted motions, improving the stability of model training, and reducing fluctuations during training, thereby accelerating convergence and improving the training efficiency and reliability of the digital human-driven model.

[0109] The technical solution of the disclosed embodiment obtains a live video and samples the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and voice frame sequence; performs 2D key point recognition in each image frame and, based on the recognition results, generates an action frame corresponding to each image frame; extracts voice features in each voice frame and, based on the extraction results, generates a voice feature frame corresponding to each voice frame; forms an original sample corresponding to the live video based on the action frame sequence obtained from each action frame and the voice feature frame sequence obtained from each voice feature frame; in the action frame sequence of the original sample, a set number of action frames are masked according to a random masking rule to obtain a masked action frame sequence; the masked action frame sequence is added to the original sample to form a training sample; and the training sample is used to train an initial action model to obtain a digital human driven model. By acquiring live video and collecting action and voice features to train the model, the accuracy of action recognition can be improved, the generalization ability of the model in real-world language feature recognition and action prediction tasks can be enhanced, action generation and prediction can be improved, and the naturalness and realism of digital human expressions and actions can be enhanced, with rich details, and improved visual effects and user experience.

[0110] Figure 3 This is a schematic diagram of a digital human driving method provided according to an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to situations where a trained neural network model (digital human driving model) is used to generate matching actions based on speech prediction. This method can be performed by a digital human driving device, which can be implemented in hardware and / or software and can generally be configured in an electronic device.

[0111] Correspondingly, such as Figure 3 As shown, the method may specifically include:

[0112] S310: Obtain a target voice for voice-driven 2D body movements of the target virtual image.

[0113] The target virtual image includes a head and limb areas.

[0114] Specifically, the target avatar can be understood as a voice-driven virtual character or image created in digital media, games, or virtual reality. Here, the avatar refers to a 2D avatar consisting of a head and limbs. Specifically, the target voice can be understood as the voice data used to drive the 2D body movements of the target avatar.

[0115] S320: Generate a target speech feature frame sequence that matches the target speech, and construct an unknown action frame sequence that matches the target speech feature frame sequence.

[0116] The target speech feature frame sequence can be specifically understood as a sequence of feature frames that are extracted by the speech feature extraction model and represent speech characteristics that match the target speech. The unknown action frame sequence can be specifically understood as an action frame sequence that needs to be predicted or generated by the algorithm model.

[0117] Generally speaking, since the action frame sequence and the speech feature frame sequence need to be time-aligned, the length of the unknown action frame sequence is usually the same as the length of the target speech feature frame sequence. It can be understood that each frame in the unknown action frame sequence is a blank frame and does not contain any 2D key points.

[0118] S330 : Fill at least one required action frame that matches the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence.

[0119] The required action frame can be specifically understood as: a 2D body movement that is expected or preset to be performed by the target virtual image at a certain moment, such as raising a hand or standing still. Based on this moment, the frame position of the target virtual image in the unknown action frame sequence can be limited. The position of the required action frame can be the head, middle or tail of the unknown action frame sequence. The local unknown action frame sequence can be specifically understood as: after filling at least one required action frame into the target frame position in the unknown action frame sequence, the unknown action frame sequence forms a local unknown action frame sequence, part of which is a known action frame.

[0120] S340: Input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence that matches the target speech.

[0121] The digital human driving model is obtained by training using the method described in any one of the embodiments of the present disclosure.

[0122] The known action frame sequence can be specifically understood as: inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model trained by the aforementioned method, and starting from the known action frame in the local unknown action frame sequence according to the speech characteristics of the target speech, predicting the previous one or several unknown action frames of the action frame (when the required action frame is at the end of the unknown action frame sequence, only the previous unknown action frame is included), the next one or several unknown action frames (when the required action frame is at the head of the unknown action frame sequence, only the next unknown action frame is included), or, at the same time, predicting the previous and next one or several unknown action frames.

[0123] Through multiple predictions, the previous and next unknown action frames are predicted based on the newly predicted action frame, and so on, a complete prediction of the unknown action frame is performed until all unknown action frames in the local unknown action frame sequence are predicted and filled with prediction results.

[0124] Generally speaking, when the desired action frame is inserted at the beginning of the unknown action frame sequence, the order in which action predictions are generated aligns with the order in which the speech is delivered. Therefore, the action can be predicted and generated in real time as the speech is delivered. However, when the desired action frame is inserted in the middle or end of the unknown action frame sequence, the order in which action predictions are generated is reversed from the order in which the speech is delivered. Therefore, it can only be used for offline prediction and generation of actions that match the speech.

[0125] The technical solution of the disclosed embodiment is as follows: obtaining a target voice for voice-driven 2D body movements of a target virtual image; generating a target voice feature frame sequence matching the target voice, and constructing an unknown action frame sequence matching the target voice feature frame sequence; filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a local unknown action frame sequence; inputting the target voice feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target voice. By randomly filling at least one required action frame at any position and combining the target voice feature frame sequence for action prediction, the accuracy of key actions is ensured, the adaptability of predicted actions to different voice changes is improved, a coherent action transition is generated before and after the key frame, the authenticity and naturalness of the action are improved, the randomness of the filling position and number of required action frames is improved, the diversity of action prediction results is improved, and the flexibility and generalization ability of digital human driving are improved.

[0126] On the basis of the above-mentioned embodiments, the embodiments of the present disclosure can perform personalized model training on the original samples according to the application scenarios or the types of emotions contained in the samples to obtain targeted digital human driving models. Therefore, it is also possible to select a matching digital human driving model according to the application scenarios or the types of emotional colors of the voice, thereby improving the reliability of action prediction of digital human driving in different application scenarios.

[0127] Figure 4 Schematic diagram of another digital human driving method provided according to an embodiment of the present disclosure. This embodiment is a refinement of the above embodiments. In this embodiment, the operation of "inputting the target speech feature frame sequence and the locally unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target speech" is specifically implemented as follows: "inputting the target speech feature frame sequence and the locally unknown action frame sequence into the digital human driving model; generating an original tag embedding vector sequence matching the locally unknown action frame sequence through a model input layer; generating a target encoding result sequence matching the original tag embedding vector through a model encoder layer; obtaining a prediction result vector for at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence through a model decoder layer; replacing each target unknown vector in the original tag embedding vector sequence with each prediction result vector, and then returning to the operation of generating an original tag embedding vector sequence matching the locally unknown action frame sequence through the model input layer until all unknown vectors in the original tag embedding vector are predicted; and outputting the known action frame sequence matching the predicted original tag embedding vector through a model output layer."

[0128] Correspondingly, such as Figure 4 As shown, the method may specifically include:

[0129] S410: Obtain a target voice for voice-driven 2D body movements of a target virtual image.

[0130] The target virtual image includes a head and limb areas.

[0131] S420: Generate a target speech feature frame sequence that matches the target speech, and construct an unknown action frame sequence that matches the target speech feature frame sequence.

[0132] S430 : Fill at least one required action frame that matches the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence.

[0133] S440: Input the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model.

[0134] S450: Generate an original tag embedding vector sequence matching the local unknown action frame sequence through a model input layer.

[0135] The original labeled embedding vector sequence can be specifically understood as: through the input layer of the digital human driving model, each action frame in the input local unknown action frame sequence is converted into an embedding vector form that the model can process. The sequence composed of these embedding vectors is the original labeled embedding vector sequence.

[0136] Among them, in the original tag embedding vector sequence, the original tag embedding vector that matches the required action frame is a known vector, and the original tag embedding vector that matches the non-required action frame (unknown action frame) is an unknown vector. That is, the original tag embedding vector sequence contains at least one unknown vector.

[0137] S460. Generate a target encoding result sequence that matches the original tag embedding vector through a model encoder layer.

[0138] The target encoding result sequence can be specifically understood as: a sequence of encoding results that reflects the internal relationship of the original tag embedding vector obtained by multi-level encoding of the original tag embedding vector through the encoder layer of the digital human-driven model.

[0139] The target encoding result sequence that matches the original tag embedding vector is generated through the multi-layer encoder in the model encoder layer. This sequence contains the association information between the unknown action frames to be predicted and the known required action frames in the subsequent action prediction, ensuring the relevance, continuity and naturalness of the subsequent action prediction results.

[0140] S470. Obtain a prediction result vector for at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence through the model decoder layer.

[0141] The target unknown vector can be specifically understood as: the unknown vector in the original tag embedding vector predicted by the digital human-driven model in the current prediction round.

[0142] Specifically, based on the sum of the target encoding result sequence and the target speech feature frame sequence, for example, the feature vectors of the two modalities are directly added together, the two feature information are fused, and through multi-layer decoding of the model decoder layer, the input is processed step by step, and the prediction results are gradually refined to obtain a prediction result vector of at least one target unknown vector in the original tag embedding vector.

[0143] In an optional implementation of this embodiment, obtaining, by the model decoder layer, a prediction result vector for at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence may include:

[0144] Obtaining, through the model decoder layer, a quantity value of unknown vectors contained in the original tag embedding vector, and randomly determining a target quantity based on the quantity value;

[0145] Randomly selecting target unknown vectors matching the target number from all unknown vectors in the original tag embedding vector sequence through the model decoder layer;

[0146] The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

[0147] Specifically, the model decoder layer obtains the number of unknown vectors to be predicted in the original tag embedding vector, and randomly determines the number of unknown vectors to be predicted within the range of the unknown vector number, i.e., the target number. For example, when the number of unknown vectors is 4, the target number can be 1, 2, 3, or 4. The model decoder layer randomly selects target unknown vectors that match the target number from all unknown vectors in the original tag embedding vector sequence, i.e., selects target unknown vectors at a target number of random positions in the original tag embedding vector sequence as the unknown vectors for this prediction. The model decoder obtains the prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

[0148] By predicting a random number of unknown vectors at random positions in each prediction process, the corresponding action frames generated have greater randomness, increasing the diversity of the prediction results. For the same input speech, each prediction may produce a different action sequence, bringing diverse visual effects and improving the user experience.

[0149] S480. After replacing each of the target unknown vectors in the original tag embedding vector sequence with each of the prediction result vectors, return to executing the operation of generating an original tag embedding vector sequence that matches the local unknown action frame sequence through the model input layer until the prediction of all unknown vectors in the original tag embedding vector is completed.

[0150] S490. Outputting, through the model output layer, the known action frame sequence that matches the original tag embedding vector that completes the prediction.

[0151] Specifically, after using each predicted result vector to replace each target unknown vector in the original labeled embedding vector sequence at the corresponding position, the operation of generating the original labeled embedding vector sequence that matches the local unknown action frame sequence through the model input layer is returned and re-executed. The unknown vectors corresponding to one or several unknown action frames before and after the known action frame are predicted and filled until all unknown vectors in the original labeled embedding vector are predicted, and the prediction results are filled into the corresponding target unknown vector in the original labeled embedding vector sequence. The known action frame sequence that matches the predicted original labeled embedding vector is output through the model output layer.

[0152] The technical solution of the embodiment of the present disclosure obtains the target speech, generates a target speech feature frame sequence, and constructs an unknown action frame sequence; fills at least one required action frame into the unknown action frame sequence, inputs the obtained local unknown action frame sequence and the target speech feature frame sequence into the digital human driving model, generates an original tag embedding vector sequence through the input layer, generates a target encoding result sequence through the encoder layer, obtains a prediction result vector of at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence through the decoder layer, replaces each target unknown vector in the original tag embedding vector sequence with each prediction result vector, and then returns to execute the operation of generating the original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer until the prediction of all unknown vectors in the original tag embedding vector is completed; and outputs the known action frame sequence matching the predicted original tag embedding vector through the model output layer. Combined with speech feature information, the pre-trained digital human driving model is used to predict the unknown vectors in the original labeled embedding vector constructed from the unknown action frame sequence, and finally a known action frame sequence that matches the original labeled embedding vector is obtained. Through the pre-trained initial action model, the complex mapping from speech to body movements can be learned, thereby improving the accuracy and reliability of action prediction, and adaptively combining the rhythm and content of the audio to generate head or upper body movements synchronized with the audio, thereby improving the visual effects of the digital human or virtual character and enhancing the user experience.

[0153] In an optional implementation of this embodiment, filling at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence may include:

[0154] Acquire at least one item of requirement description information matching the target speech, wherein the requirement description information includes a requirement time point and a requirement action;

[0155] generating at least one required action frame for the target virtual image according to the required action;

[0156] Locating at least one target unknown action frame in the unknown action frame sequence according to the required time point;

[0157] In the unknown action frame sequence, each target unknown action frame is replaced with the matching required action frame to obtain the local unknown action frame sequence.

[0158] Specifically, the at least one required action frame that matches the target avatar can be specifically set based on the requirements of the target speech. For example, at a specific moment in the broadcast, the digital human is required to perform a hand-raising gesture. Therefore, at least one required action description information matching the target speech can be obtained. This required action description information records the required time point and the corresponding specific required action in the target speech. Based on the motion process included in the required action, at least one required action frame can be generated for the target avatar. Each required action frame records the 2D head and limb key points corresponding to the current motion process of the required action. Based on the required time point, the target unknown action frame corresponding to the required time point is located in the unknown action frame sequence and replaced with the matching required action frame. After replacing the target unknown action frames corresponding to each required time point in the required action description information with the matching required action frame, a partial unknown action frame sequence is obtained. By adding preset key actions at specified time positions based on the actual action requirements of the speech, the action prediction results are diverse, controllable, and flexible, making the generated actions more natural and realistic, enhancing the visual effects of the digital human or virtual character, and improving the user experience.

[0159] In another optional implementation of this embodiment, filling at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence may include:

[0160] When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result;

[0161] The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

[0162] Specifically, at least one required action frame that the target virtual image matches can be specifically set according to the requirements of the target voice. For example, at a specific moment in the broadcast, the digital human is required to be in a stationary state. 2D key point recognition is performed on the target virtual image in a stationary state, and a required action frame is formed based on the recognition result. The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence. In human motion prediction, when the first frame is in a non-stationary state, its head or limbs are often in a special position or at a certain angle. Due to the influence of uncertain factors such as movement speed and amplitude, the subsequently generated action frames are often discontinuous with the first frame in terms of action performance. In order to generate the continuity and naturalness of the action, using the stationary state as the starting point can reduce this discontinuity, making the predicted action more stable and accurate, enhancing the realism and naturalness of the digital person or virtual character's action, and improving the user experience.

[0163] In another optional implementation of this embodiment, filling at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence may include:

[0164] If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target avatar;

[0165] generating the required action frame according to the end action;

[0166] The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

[0167] Specifically, the target voice has a previous related voice, that is, the ending voice and action of the previous related voice are consistent with the beginning voice and action of the target voice. In order to ensure the continuity of the generated action, the ending action of the target virtual image when broadcasting the previous related voice can be obtained, and the corresponding required action frame can be generated. The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain a local unknown action frame sequence, realizing the model's global action prediction of the voice file, which can make the predicted action smooth at the boundary of the target voice file, realize the naturalness and smoothness of the action switching, enhance the realism of the digital person or virtual character's action, and improve the user experience.

[0168] In another optional implementation of this embodiment, the digital human driving method may further include:

[0169] When the condition for using the target virtual image to broadcast the target voice is met, forming an image frame sequence that matches the target virtual image based on the known action frame sequence;

[0170] The image frame sequence and the target voice are combined to obtain a target video, and the target video is played.

[0171] Specifically, when the conditions for using a target virtual image to broadcast the target voice are met, that is, after a known action frame sequence is generated by a pre-trained digital human-driven model based on the target voice and at least one required action frame, an image frame sequence matching the target virtual image is formed based on the known action frame sequence. That is, the key points of the head and limbs recorded in the action frame are associated with the image matching the target virtual image, so that the target virtual image performs the corresponding action in the corresponding action frame, and the image corresponding to the current target virtual image is used as the image frame matching the action frame. Thus, an image frame sequence matching the known action frame sequence is obtained, and the image frame sequence and the target voice are combined to obtain a target video, which is then played. By combining the known action frame sequence with the target virtual image and the target voice to obtain the target video, the randomness and diversity of the known action frame sequence can bring different forms of expression to the video content, increasing the richness and innovation of the video content. Users can see the virtual image react in real time according to the voice content. This interactivity can enhance the audience's sense of participation and immersion.

[0172] For ease of understanding, the specific application scenarios applicable to each embodiment of the present disclosure are now described. Existing solutions for driving the whole-body movement of digital humans require good 3D motion data. However, the data acquisition process has strict requirements on the shooting environment and conditions, which not only limits the flexibility of data acquisition, but also significantly increases the cost, making it difficult to carry out large-scale data acquisition. Even if 3D motion data is collected under ideal conditions, when it is rendered onto a two-dimensional plane, problems such as motion distortion or unnaturalness will be encountered, which directly affects the final visual effect and user experience. To solve the above problems, the embodiments of the present disclosure propose a method for constructing a digital human driving model and a digital human driving method.

[0173] The construction method of the digital human-driven model includes two important operations, one is the data preprocessing stage, and the other is the model training stage. The digital human-driven method corresponds to one important operation, namely the inference and prediction stage.

[0174] 1. Data preprocessing stage

[0175] Obtain multiple videos and use a keypoint estimation model to extract the 2D keypoint coordinates in each broadcast image. The videos can be real people, with the upper body in the frame.

[0176] Specifically, in Figure 5 FIG. 1 shows a schematic diagram of an action frame applicable to an embodiment of the present disclosure. Figure 5As shown in the figure, by processing each broadcast image, a total of 122 key points, including those on the face, hands, and upper body, are extracted to form a sequence of action frames. Simultaneously, a pre-trained Wav2Vec 2.0 model is used to extract speech features from the speech and time-align them with the action frame sequence. This lays the foundation for subsequent models to learn the mapping between speech and action. Time alignment can be achieved by maintaining a consistent sampling rate, for example, 25 frames / s. Audio content can also be understood as frames, meaning that a pre-trained Wav2Vec 2.0 model is used to extract speech feature frame sequences.

[0177] 2. Model training phase

[0178] Specifically, a Transformer-based neural network model is used during model training. This Transformer-based neural network model consists of an encoder layer consisting of L stacked encoders and a decoder layer consisting of L stacked decoders. Before model training, a masking mechanism is used to randomly mask portions of the action frames in each action frame sequence. For example, a probability interval is first set, such as [30%, 70%]. Then, for each action frame sequence, a probability value, such as 50%, is randomly determined. 50% of the action frames in this sequence are then randomly masked and input into the neural network model. The model then predicts the masked action frames based on the synchronously input, time-aligned speech feature frame sequence. The model then learns to complete the masked action frames from the speech. A diffusion loss function is used for supervision during training, ultimately resulting in a trained Transformer-based neural network model, i.e., the digital human-driven model.

[0179] 3. Reasoning and prediction stage.

[0180] Specifically, in Figure 6 This is a schematic diagram illustrating a scenario where voice-driven 2D full-body motion is performed using the first frame, applicable to embodiments of the present disclosure. This application scenario specifically describes the steps involved in training a digital human driving model based on a video-based mapping relationship between voice and 2D body motion, generating matching motions based on this digital human driving model and voice prediction, and ultimately producing a matching avatar video.

[0181] Specifically, such as Figure 6 As shown in Figure 2, Token embedding refers to the embedded vector sequence obtained through a projection matrix transformation, that is, the input masking action frame sequence is converted into an embedded vector form that can be processed by the model through the Transformer model input layer.

[0182] Among them, the first-frame speech-driven 2D full-body motion method specifically includes: by inputting a piece of speech and a frame of known motion as the first frame (i.e., the reference frame), allowing the model to sequentially predict the 2D key point coordinates of the remaining frames in an autoregressive manner.

[0183] Specifically, the input reference frame can obtain a corresponding action frame through the whole-body key point estimation model. After the action frame sequence with the action frame as the first frame and the speech feature frame sequence matching the speech are input into the digital human driving model, the input layer of the model first converts the action frame sequence into a marker embedding vector sequence, and the first marker embedding vector in the marker embedding vector sequence is a known token, and the rest are unknown tokens.

[0184] The L Transformer encoders of this model use multi-stage encoding to first generate a coding result sequence for the tag embedding vector sequence. In this coding result sequence, only the first coding result is a known token, and the remaining coding results are unknown tokens. The above coding result sequence is summed with the speech feature sequence of this speech, and then input into the L Transformer decoders of the model. The L Transformer decoders use multi-stage decoding to obtain the prediction result of the second tag embedding vector in the tag embedding vector sequence, that is, the predicted token.

[0185] After taking the first and second token embedding vectors in the token embedding vector sequence as known tokens, we can proceed to the next round of prediction to obtain the prediction result of the third token embedding vector. After performing the above operation T times, we will finally obtain the prediction result of the action frame sequence that matches the speech.

[0186] also, Figure 7 This is a schematic diagram of another application scenario of general voice-driven 2D full-body motion applicable to the embodiments of the present disclosure, such as Figure 7 As shown, during model inference, the input known action frames can be the desired action frames at a set time point, and the number of known action frames can be one or more. For example, at what time point do we want the digital figure to raise its hand? During each inference, predictions can be made at any one or more positions before or after the known portion, randomly determined based on the known portion, to increase the diversity of action generation.

[0187] and Figure 6 The implementation method of the general voice-driven 2D full-body motion method is similar to that of the general voice-driven 2D full-body motion method. Specifically, the method includes: inputting a piece of voice and a frame of known motion as the action frame at the set position (i.e., the reference frame), and allowing the model to randomly predict the 2D key point coordinates of the remaining frames in an autoregressive manner.

[0188] Specifically, the input reference frame can obtain a corresponding known action frame through the whole-body key point estimation model. After filling the known action frame into any position in the action frame sequence, the action frame sequence and the speech feature frame sequence matching the speech are input into the digital human driving model. The input layer of the model first converts the action frame sequence into a marker embedding vector sequence, and the marker embedding vector in the marker embedding vector sequence that matches the known action frame is a known token, and the rest are unknown tokens.

[0189] The L Transformer encoders of the model first generate a coding result sequence for the tag embedding vector sequence through multi-stage encoding. In this coding result sequence, only the coding results that match the known action frame are known tokens, and the remaining coding results are unknown tokens. The above coding result sequence is summed with the speech feature sequence of this speech and then input into the L Transformer decoders of the model. The L Transformer decoders obtain the prediction results of the tag embedding vectors at any one or more positions in the tag embedding vector sequence through multi-stage decoding, that is, the predicted token.

[0190] After taking the predicted token embedding vector in the token embedding vector sequence and the token embedding vector matching the known action frame as the known token, we can proceed to the next round of prediction to obtain the prediction results of the token embedding vector at any one or more positions in the token embedding vector sequence. After performing the above operation T times, we will finally obtain the prediction results of the action frame sequence matching this speech.

[0191] This scheme uses a random prediction order, including the random number of action frames predicted each time and the random position of the action frames, to enhance the diversity of generated actions and avoid generating a single result.

[0192] Through the inference method of interpolation action completion, since the model can accept known action frames at any time and in any number as input, during inference, the key frame of a certain action can be input into the model, allowing the model to automatically complete the actions before and after the key frame. In this way, flexible action editing needs can be met.

[0193] In addition, different types of models can be selected for training according to different types of broadcast content (happy or sad) to improve the adaptability of the model in predicting actions.

[0194] The technical solutions proposed in the disclosed embodiments reduce computational complexity and resource consumption, enrich training samples, and improve the training efficiency and reliability of the digital human-driven model. By randomly filling at least one required action frame at any location and combining it with a target speech feature frame sequence for action prediction, the accuracy of key actions is ensured, the predicted action's adaptability to varying speech variations is improved, and coherent action transitions are generated before and after key frames, enhancing the authenticity and naturalness of the actions. The randomness of the placement and number of required action frames increases the diversity of action prediction results, improving the flexibility and generalization of digital human-driven models.

[0195] It's worth reiterating that this disclosed implementation proposes a high-quality, voice-driven 2D full-body motion solution, supporting the subsequent generation of motion-based digital human videos. By employing 2D motion representation and innovative model training and prediction strategies, this implementation overcomes the shortcomings of related technologies in terms of computational complexity, resource consumption, data acquisition difficulties, and generation diversity. The resulting motion sequences are natural, realistic, and rich in detail, offering advantages such as simplicity and efficiency. This approach is widely applicable to fields requiring voice-driven full-body motion, such as digital human live streaming, intelligent customer service, and virtual characters.

[0196] As an implementation of the aforementioned methods for constructing digital human drive models, the present disclosure further provides an optional embodiment of an execution device for implementing the aforementioned methods for constructing digital human drive models.

[0197] Figure 8 FIG is a structural diagram of a device for constructing a digital human driving model according to an embodiment of the present disclosure. Figure 8 The device includes: an original sample acquisition module 810, a mask processing module 820, a training sample acquisition module 830 and a model training module 840, wherein:

[0198] The original sample acquisition module 810 is used to acquire original samples; wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0199] a masking processing module 820 configured to mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence;

[0200] A training sample acquisition module 830 is configured to add the masking action frame sequence to the original sample to form a training sample;

[0201] The model training module 840 is used to train the initial motion model using the training samples to obtain a digital human driving model.

[0202] The technical solution of the disclosed embodiment comprises obtaining an original sample; masking a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence; adding the masked action frame sequence to the original sample to form a training sample; and using the training sample to train the initial action model to obtain a digital human driving model. The training sample is formed by randomly masking an action frame sequence including 2D action representations and combining it with a speech feature frame sequence to obtain a digital human driving model. This reduces computational complexity and resource consumption, enriches the training samples, improves the training efficiency and reliability of the digital human driving model, and enhances the realism of the generated driving actions.

[0203] Based on the above embodiments, the original sample acquisition module 810 is specifically configured to:

[0204] Acquire a live video, and sample the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence;

[0205] Performing 2D key point recognition in each image frame, and forming action frames corresponding to each of the image frames based on the recognition results;

[0206] Extracting speech features in each speech frame, and forming speech feature frames corresponding to each speech frame according to the extraction results;

[0207] The original sample corresponding to the live video is formed based on the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

[0208] Based on the above embodiments, in the real-person video, only the upper body area of ​​the real person is displayed.

[0209] Based on the above embodiments, the mask processing module 820 is specifically configured to:

[0210] Randomly select the target probability value within the preset probability interval;

[0211] Obtaining the total number of action frames contained in the action frame sequence of the original sample;

[0212] Calculating the masked number of action frames according to the total number of action frames and the target probability value;

[0213] In the action frame sequence of the original sample, action frames matching the mask quantity are randomly selected for masking to obtain a masked action frame sequence.

[0214] Based on the above embodiments, the model training module 840 is specifically configured to:

[0215] Inputting the training samples into the initial motion model;

[0216] Generating a first labeled embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second labeled embedding vector sequence corresponding to the action frame sequence;

[0217] wherein the first marker embedding vector sequence includes an unknown vector corresponding to the masked action frame;

[0218] Performing multi-level encoding on the first tag embedding vector sequence through a model encoder layer to obtain an encoding result sequence;

[0219] Performing multi-stage decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each of the unknown vectors;

[0220] Calculating a model loss value based on a prediction result vector for each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence;

[0221] The weight of the initial action model is updated according to the model loss value.

[0222] Based on the above embodiments, the model training module 840 is further configured to:

[0223] It is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors using a diffusion loss function.

[0224] The above-mentioned product can execute the method for constructing a digital human-driven model provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the method for constructing a digital human-driven model.

[0225] As an implementation of the above-mentioned digital human driving methods, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned digital human driving methods.

[0226] Figure 9 FIG. 1 is a structural diagram of a digital human driving device provided according to an embodiment of the present disclosure. Figure 9 The device includes: a speech acquisition module 910, a first sequence construction module 920, a second sequence acquisition module 930 and a third sequence acquisition module 940, wherein:

[0227] A voice acquisition module 910 is configured to acquire a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions;

[0228] A first sequence construction module 920 is configured to generate a target speech feature frame sequence matching the target speech, and to construct an unknown action frame sequence matching the target speech feature frame sequence;

[0229] A second sequence acquisition module 930 is configured to fill the unknown action frame sequence with at least one required action frame that matches the target avatar, to obtain a partial unknown action frame sequence;

[0230] The third sequence acquisition module 940 is used to input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence matching the target speech.

[0231] The digital human driving model is trained by the method described in any one of the embodiments of the present disclosure.

[0232] The technical solution of the disclosed embodiment is as follows: obtaining a target voice for voice-driven 2D body movements of a target virtual image; generating a target voice feature frame sequence matching the target voice, and constructing an unknown action frame sequence matching the target voice feature frame sequence; filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a local unknown action frame sequence; inputting the target voice feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target voice. By randomly filling at least one required action frame at any position and combining the target voice feature frame sequence for action prediction, the accuracy of key actions is ensured, the adaptability of predicted actions to different voice changes is improved, a coherent action transition is generated before and after the key frame, the authenticity and naturalness of the action are improved, the randomness of the filling position and number of required action frames is improved, the diversity of action prediction results is improved, and the flexibility and generalization ability of digital human driving are improved.

[0233] Based on the above embodiments, the second sequence acquisition module 930 is specifically configured to:

[0234] Acquire at least one item of requirement description information matching the target speech, wherein the requirement description information includes a requirement time point and a requirement action;

[0235] generating at least one required action frame for the target virtual image according to the required action;

[0236] Locating at least one target unknown action frame in the unknown action frame sequence according to the required time point;

[0237] In the unknown action frame sequence, each target unknown action frame is replaced with the matching required action frame to obtain the local unknown action frame sequence.

[0238] Based on the above embodiments, the second sequence acquisition module 930 is specifically configured to:

[0239] When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result;

[0240] The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

[0241] Based on the above embodiments, the second sequence acquisition module 930 is specifically configured to:

[0242] If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target avatar;

[0243] generating the required action frame according to the end action;

[0244] The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

[0245] Based on the above embodiments, the third sequence acquisition module 940 is specifically configured to:

[0246] Inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model;

[0247] Generating an original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer;

[0248] Generating a target encoding result sequence that matches the original token embedding vector through a model encoder layer;

[0249] Obtaining, by a model decoder layer, a prediction result vector for at least one target unknown vector in the original tag embedding vector based on a summation result of the target encoding result sequence and the target speech feature frame sequence;

[0250] After replacing each of the target unknown vectors in the original labeled embedding vector sequence with each of the predicted result vectors, returning to the operation of generating an original labeled embedding vector sequence matching the local unknown action frame sequence through the model input layer until all unknown vectors in the original labeled embedding vector are predicted;

[0251] The model output layer outputs the known action frame sequence that matches the original label embedding vector that completes the prediction.

[0252] Based on the above embodiments, the third sequence acquisition module 940 is further configured to:

[0253] Obtaining, through the model decoder layer, a quantity value of unknown vectors contained in the original tag embedding vector, and randomly determining a target quantity based on the quantity value;

[0254] Randomly selecting target unknown vectors matching the target number from all unknown vectors in the original tag embedding vector sequence through the model decoder layer;

[0255] The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

[0256] Based on the above embodiments, the device further includes: an image frame sequence acquisition module and a target video acquisition module, wherein:

[0257] an image frame sequence acquisition module, configured to generate an image frame sequence matching the target virtual image based on the known action frame sequence when a condition for adopting the target virtual image to broadcast the target voice is met;

[0258] The target video acquisition module is used to combine the image frame sequence and the target voice to obtain a target video, and play the target video.

[0259] The above-mentioned product can execute the digital human driving method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the digital human driving method.

[0260] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0261] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0262] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0263] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0264] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0265] The computing unit 1001 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 executes the various methods and processes described above, such as executing a method for constructing a digital human-driven model, namely:

[0266] Obtaining original samples, wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0267] In the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence;

[0268] Adding the masking action frame sequence to the original sample to form a training sample;

[0269] The initial motion model is trained using the training samples to obtain a digital human driving model.

[0270] Alternatively, implement a digital human-driven approach, which means:

[0271] Obtaining a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions;

[0272] generating a target speech feature frame sequence matching the target speech, and constructing an unknown action frame sequence matching the target speech feature frame sequence;

[0273] Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence;

[0274] Inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence that matches the target speech;

[0275] The digital human driving model is obtained by training using the method described in any one of the embodiments of the present disclosure.

[0276] For example, in some embodiments, the method for constructing a digital human-driven model or the method for driving a digital human may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method for constructing a digital human-driven model or the method for driving a digital human described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the method for constructing a digital human-driven model or the method for driving a digital human by any other appropriate means (e.g., by means of firmware).

[0277] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0278] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0279] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0280] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0281] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0282] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0283] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0284] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0285] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0286] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for constructing a digital human driving model, comprising: Acquire original samples, wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points; In the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence; Adding the masking action frame sequence to the original sample to form a training sample; The initial action model is trained using the training samples to obtain a digital human driving model.

2. The method according to claim 1, wherein: The obtaining of the original sample comprises: Acquire a live video, and perform image and voice sampling on the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence; Performing 2D key point recognition in each image frame, and forming action frames corresponding to each of the image frames according to the recognition results; Extracting speech features in each speech frame, and forming speech feature frames corresponding to each of the speech frames according to the extraction results; The original sample corresponding to the live video is formed according to the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

3. The method according to claim 2, wherein: In the real-person video, only the upper body area of ​​the real person is shown.

4. The method according to claim 1, wherein: The step of masking a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence includes: Randomly select the target probability value within the preset probability interval; Obtaining the total number of action frames contained in the action frame sequence of the original sample; Calculating the masked number of action frames according to the total number of action frames and the target probability value; In the action frame sequence of the original sample, each action frame matching the masking quantity is randomly selected for masking processing to obtain a masked action frame sequence.

5. The method according to claim 1, wherein: The step of training the initial motion model using the training samples to obtain the digital human motion model comprises: Inputting the training samples into the initial network model; Generating a first labeled embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second labeled embedding vector sequence corresponding to the action frame sequence; wherein the first marker embedding vector sequence includes an unknown vector corresponding to the masked action frame; Performing multi-level encoding on the first tag embedding vector sequence through a model encoder layer to obtain an encoding result sequence; Performing multi-stage decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each of the unknown vectors; Calculating a model loss value according to a prediction result vector of each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence; The weights of the initial action model are updated according to the model loss value.

6. The method according to claim 5, wherein: The step of calculating the model loss value according to the prediction result vector of each unknown vector and the known vectors corresponding to each unknown vector in the second tag embedding vector sequence includes: The diffusion loss function is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors.

7. A digital human driving method, comprising: Acquire a target voice for voice-driven 2D body movements of a target virtual image, wherein the target virtual image includes a head and body regions; Generate a target speech feature frame sequence matching the target speech, and construct an unknown action frame sequence matching the target speech feature frame sequence; Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a local unknown action frame sequence; Inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target speech; Wherein, the digital human driving model is obtained by training through the method described in any one of claims 1-6.

8. The method according to claim 7, wherein: Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence, including: Acquire at least one item of requirement description information matching the target voice, wherein the requirement description information includes a requirement time point and a requirement action; generating at least one required action frame for the target virtual image according to the required action; According to the required time point, locating at least one target unknown action frame in the unknown action frame sequence; In the unknown action frame sequence, each of the target unknown action frames is replaced with the matching required action frame to obtain the local unknown action frame sequence.

9. The method according to claim 7, wherein: Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence, including: When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

10. The method according to claim 7, wherein: Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence, including: If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target virtual image; According to the end action, generating the required action frame; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

11. The method according to claim 7, wherein: The step of inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target speech comprises: Inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model; Generate an original tag embedding vector sequence matching the local unknown action frame sequence through a model input layer; Generating a target encoding result sequence matching the original token embedding vector through a model encoder layer; Obtaining a prediction result vector for at least one target unknown vector in the original tag embedding vector according to the sum of the target encoding result sequence and the target speech feature frame sequence through a model decoder layer; After replacing each of the target unknown vectors in the original tag embedding vector sequence with each of the prediction result vectors, returning to perform the operation of generating an original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer until the prediction of all unknown vectors in the original tag embedding vector is completed; The known action frame sequence matched with the original labeled embedding vector that completes the prediction is output through the model output layer.

12. The method according to claim 11, wherein: The method of obtaining a prediction result vector of at least one target unknown vector in the original tag embedding vector by the model decoder layer according to the sum of the target encoding result sequence and the target speech feature frame sequence includes: Obtaining the quantity value of the unknown vectors contained in the original tag embedding vector through the model decoder layer, and randomly determining the target quantity according to the quantity value; Randomly selecting a target unknown vector matching the target quantity from all unknown vectors of the original tag embedding vector sequence through the model decoder layer; The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

13. The method according to any one of claims 7 to 12, further comprising: When the condition for using the target virtual image to broadcast the target voice is met, forming an image frame sequence matching the target virtual image according to the known action frame sequence; The image frame sequence and the target voice are combined to obtain a target video, and the target video is played.

14. A device for constructing a digital human driving model, comprising: The original sample acquisition module is used to acquire the original sample; wherein the original sample includes a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points; A mask processing module, used for masking a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence; A training sample acquisition module, used for adding the masking action frame sequence to the original sample to form a training sample; The model training module is used to train the initial action model using the training samples to obtain a digital human driving model.

15. The device according to claim 14, wherein: The original sample acquisition module is specifically used for: Acquire a live video, and sample the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence; Performing 2D key point recognition in each image frame, and forming action frames corresponding to each of the image frames according to the recognition results; Extracting speech features in each speech frame, and forming speech feature frames corresponding to each of the speech frames according to the extraction results; The original sample corresponding to the live video is formed according to the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

16. The device according to claim 15, wherein: In the real-person video, only the upper body area of ​​the real person is shown.

17. The device according to claim 14, wherein: The mask processing module is specifically used for: Randomly select the target probability value within the preset probability interval; Obtaining the total number of action frames contained in the action frame sequence of the original sample; Calculating the masked number of action frames according to the total number of action frames and the target probability value; In the action frame sequence of the original sample, each action frame matching the masking quantity is randomly selected for masking processing to obtain a masked action frame sequence.

18. The device according to claim 14, wherein: The model training module is specifically used for: Inputting the training samples into an initial motion model; Generating a first labeled embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second labeled embedding vector sequence corresponding to the action frame sequence; wherein the first marker embedding vector sequence includes an unknown vector corresponding to the masked action frame; Performing multi-level encoding on the first tag embedding vector sequence through a model encoder layer to obtain an encoding result sequence; Performing multi-stage decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each of the unknown vectors; Calculating a model loss value according to a prediction result vector of each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence; The weights of the initial action model are updated according to the model loss value.

19. The device according to claim 18, wherein: The model training module is further used to: The diffusion loss function is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors.

20. A digital human driving device, comprising: A voice acquisition module, used to acquire a target voice for voice-driven 2D body movements of a target virtual image, wherein the target virtual image includes a head and body regions; A first sequence construction module is used to generate a target speech feature frame sequence matching the target speech, and to construct an unknown action frame sequence matching the target speech feature frame sequence; A second sequence acquisition module, configured to fill at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a local unknown action frame sequence; A third sequence acquisition module is used to input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence matching the target speech; Wherein, the digital human driving model is obtained by training through the method described in any one of claims 1-6.

21. The device according to claim 20, wherein: The second sequence acquisition module is specifically used to: Acquire at least one item of requirement description information matching the target voice, wherein the requirement description information includes a requirement time point and a requirement action; generating at least one required action frame for the target virtual image according to the required action; According to the required time point, locating at least one target unknown action frame in the unknown action frame sequence; In the unknown action frame sequence, each of the target unknown action frames is replaced with the matching required action frame to obtain the local unknown action frame sequence.

22. The device according to claim 20, wherein: The second sequence acquisition module is specifically used to: When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

23. The device according to claim 20, wherein: The second sequence acquisition module is specifically used to: If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target virtual image; According to the end action, generating the required action frame; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

24. The device according to claim 20, wherein: The third sequence acquisition module is specifically used to: Inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human-driven model; Generate an original tag embedding vector sequence matching the local unknown action frame sequence through a model input layer; Generating a target encoding result sequence matching the original token embedding vector through a model encoder layer; Obtaining a prediction result vector for at least one target unknown vector in the original tag embedding vector according to the sum of the target encoding result sequence and the target speech feature frame sequence through a model decoder layer; After replacing each of the target unknown vectors in the original tag embedding vector sequence with each of the prediction result vectors, returning to perform the operation of generating an original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer until the prediction of all unknown vectors in the original tag embedding vector is completed; The known action frame sequence matched with the original labeled embedding vector that completes the prediction is output through the model output layer.

25. The device according to claim 24, wherein: The third sequence acquisition module is further used to: Obtaining the quantity value of the unknown vectors contained in the original tag embedding vector through the model decoder layer, and randomly determining the target quantity according to the quantity value; Randomly selecting a target unknown vector matching the target quantity from all unknown vectors of the original tag embedding vector sequence through the model decoder layer; The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

26. The apparatus according to any one of claims 20 to 25, further comprising: An image frame sequence acquisition module, configured to form an image frame sequence matching the target virtual image according to the known action frame sequence when the condition for using the target virtual image to broadcast the target voice is met; The target video acquisition module is used to combine the image frame sequence and the target voice to obtain a target video, and play the target video.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-13.

29. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images