Construction of digital human driving model, digital human driving method, device and equipment

Abstract

The present disclosure provides a digital human driving model construction, a digital human driving method, device, equipment, medium and program product, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and large model, and can be applied to the scene of AIGC and digital human. The specific implementation scheme is: obtaining an original sample; in the action frame sequence of the original sample, a set number of action frames are masked according to a random masking rule to obtain a masked action frame sequence; the masked action frame sequence is added to the original sample to form a training sample; and the initial action model is trained using the training sample to obtain a digital human driving model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generated Content) and digital humans. It specifically relates to a method for constructing a digital human-driven model, a digital human-driving method, a device for constructing a digital human-driven model, a digital human-driving device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of AIGC technology, digital humans, as a new form of human-computer interaction, are receiving more and more attention.

[0003] Related technologies can improve the interactivity of digital humans by using voice to drive their facial and body movements. However, as the field of digital human interaction continues to expand, existing voice-driven digital human movement technologies can no longer meet the growing demand for more efficient and convenient digital human driving. Summary of the Invention

[0004] The present disclosure provides a method for constructing a digital human-driven model, a digital human-driving method, a device for constructing a digital human-driven model, a digital human-driving device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a method for constructing a digital human driving model is provided, comprising:

[0006] Obtaining original samples, wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0007] In the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence;

[0008] Adding the masking action frame sequence to the original sample to form a training sample;

[0009] The initial motion model is trained using the training samples to obtain a digital human driving model.

[0010] According to another aspect of the present disclosure, a digital human driving method is provided, comprising:

[0011] Obtaining a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions;

[0012] generating a target speech feature frame sequence matching the target speech, and constructing an unknown action frame sequence matching the target speech feature frame sequence;

[0013] Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence;

[0014] Inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence that matches the target speech;

[0015] The digital human driving model is obtained by training using the method for constructing a digital human driving model as described in any one of the embodiments of the present disclosure.

[0016] According to one aspect of an embodiment of the present disclosure, there is also provided an apparatus for constructing a digital human-driven model, comprising:

[0017] The original sample acquisition module is used to obtain the original sample; wherein the original sample includes a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0018] a masking processing module, configured to mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence;

[0019] A training sample acquisition module, configured to add the masking action frame sequence to the original sample to form a training sample;

[0020] The model training module is used to train the initial action model using the training samples to obtain a digital human driving model.

[0021] According to one aspect of an embodiment of the present disclosure, a digital human driving device is provided, comprising:

[0022] a broadcast voice acquisition module, configured to acquire a target voice for voice-driven 2D body movements of a target virtual image, wherein the target virtual image includes a head and body regions;

[0023] A first sequence construction module is used to generate a target speech feature frame sequence matching the target speech, and to construct an unknown action frame sequence matching the target speech feature frame sequence;

[0024] A second sequence acquisition module is configured to fill at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence;

[0025] A third sequence acquisition module is configured to input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence that matches the target speech;

[0026] The digital human driving model is obtained by training using the method for constructing a digital human driving model as described in any one of the embodiments of the present disclosure.

[0027] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0028] at least one processor; and

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a digital human driven model or the digital human driving method as described in any one of the embodiments of the present disclosure.

[0031] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method for constructing a digital human-driven model or the method for driving a digital human according to any one of the embodiments of the present disclosure.

[0032] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0034] Figure 1 is a schematic diagram of a method for constructing a digital human driving model provided according to an embodiment of the present disclosure;

[0035] Figure 2 is a schematic diagram of another method for constructing a digital human driving model provided according to an embodiment of the present disclosure;

[0036] Figure 3 is a schematic diagram of a digital human driving method provided according to an embodiment of the present disclosure;

[0037] Figure 4 is a schematic diagram of another digital human driving method provided according to an embodiment of the present disclosure;

[0038] Figure 5 is a schematic diagram of an action frame applicable to an embodiment of the present disclosure;

[0039] Figure 6 This is a schematic diagram of a first-frame voice-driven 2D full-body motion applicable to an embodiment of the present disclosure;

[0040] Figure 7 is a general voice-driven 2D full-body motion diagram applicable to embodiments of the present disclosure;

[0041] Figure 8 is a structural diagram of a device for constructing a digital human-driven model according to an embodiment of the present disclosure;

[0042] Figure 9 is a structural diagram of a digital human driving device provided according to an embodiment of the present disclosure;

[0043] Figure 10 It is a block diagram of an electronic device used to implement the method for constructing a digital human driving model or the digital human driving method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] Figure 1 This is a schematic diagram of a method for constructing a digital human-driven model, according to an embodiment of the present disclosure. This embodiment is applicable to constructing a mapping relationship between speech and 2D body movements based on video and training a neural network model. This method can be performed by a device for constructing a digital human-driven model, which can be implemented using hardware and / or software and is generally configured in an electronic device.

[0046] Correspondingly, such as Figure 1 As shown, the method may specifically include:

[0047] S110: Obtain original samples.

[0048] The original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frames include 2D head and limb key points.

[0049] An action frame refers to a single image or image segment that constitutes an action sequence, which records the movement of the human body over a period of time. Each frame represents a moment of the action composed of various key points. Among them, the action frame can be used to describe the position information of the 2D head and limb key points in the image when the human body performs a set action (that is, the position coordinates of the key points). The limb key points may only include key points in the upper limb area, or may also include key points in the upper limb and lower limb areas at the same time. Key points refer to the 2D key point coordinates extracted from portrait video data, which are used to represent various parts of the human body and capture the movements and motion paths of the human body. Among them, the collection parts of key points may include the face (such as eyes, ears, mouth, eyebrows and nose, etc.), hands (such as wrists, fingers and palms, etc.) and limbs (such as shoulders, elbows, upper arms and forearms, etc.).

[0050] A speech feature frame is a discrete data unit extracted from a continuous speech signal and used to represent speech characteristics. The speech feature frame specifically includes acoustic features (e.g., spectrum, pitch, and tone) and speech features of a speech segment.

[0051] Optionally, the action frames can be obtained by a pre-trained 2D key point recognition model, and the speech feature frames can be extracted from the speech signal by a pre-trained speech feature extraction model, such as Wav2Vec 2.0.

[0052] The time-aligned action frame sequence and speech feature frame sequence can be understood as synchronized speech and picture in a video. In this embodiment, the number of action frames included in the action frame sequence is consistent with the number of speech feature frames included in the speech feature frame sequence.

[0053] S120 . Mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence.

[0054] The random masking rule can be understood as randomly selecting a portion of the action frames in the sequence and masking (or covering) them, such as replacing the pixel values ​​of each pixel in the action frame with zero or other values. The set number refers to the number of action frames masked according to the random masking rule. This number can be based on a percentage of the total sequence length or a fixed number of frames.

[0055] Specifically, the time-aligned action frame sequence containing 2D head and limb key points and the speech feature frame sequence are obtained as original samples, a certain number of action frames are masked according to a random masking rule, for example, 50% of the input action frames are randomly covered using a mask mechanism, and a masked action frame sequence is obtained. By masking the action frames, a diverse masked action frame sequence can be constructed as a training sample for model training, which helps to improve the generalization ability of the model.

[0056] Through the above setting, a masked action frame sequence corresponding to each action frame sequence in each original sample can be generated.

[0057] S130, the masked action frame sequence is added to the original sample to form a training sample.

[0058] It can be understood that in one training sample, there are three pieces of information, respectively, the time-aligned action frame sequence and the speech feature frame sequence, and the masked action frame sequence obtained by randomly masking one or more action frames in the action frame sequence.

[0059] S140, using the training sample to train the initial action model to obtain a digital human driving model.

[0060] Specifically, the masked action frame sequence is added to the original sample to form a training sample including the masked action frame sequence, the action frame sequence and the speech feature frame sequence, and the initial action model is trained using the training sample.

[0061] In an optional embodiment of the embodiment, the initial action model can be a Transformer neural network model.

[0062] The input data includes the pre-processed action frame sequence, the masked action frame sequence and the extracted speech feature frame sequence. The model architecture includes an input layer, a multi-layer structure of an encoder layer, a multi-layer structure of a decoder layer and an output layer. The encoder layer is responsible for encoding the known action, while the decoder layer is responsible for learning the associated features between the action and the speech. During training, the output of the encoder layer is used as the input of the decoder layer, a prediction sequence is generated through a self-attention mechanism and an encoder-decoder attention mechanism, and a loss, such as a diffusion loss, between the predicted output and the real key point coordinates is calculated. The model weights are updated using backpropagation and an optimizer to minimize the loss. Training is performed through batch processing and multiple cycles of iteration, and in each cycle, the model gradually improves the feature representation and reduces the loss. In addition, a validation set is used to monitor the model performance and prevent overfitting, and the hyperparameters are adjusted according to the performance. Finally, when the performance on the validation set reaches the best, the model parameters are saved for subsequent testing and application, and a digital human driving model is obtained.

[0063] It should be noted that related technologies for voice-driven full-body movements primarily employ 3D motion representation. These methods directly generate corresponding three-dimensional motion sequences from speech to drive the full-body movement of the digital human. During the implementation of this invention, the inventors discovered the following drawbacks in related technologies: Acquiring 3D motion representations presents certain difficulties. Furthermore, when 3D motions are rendered onto a two-dimensional surface, they can become distorted or unnatural, impacting the final visual effect and user experience.

[0064] Based on this, the technical solution of the embodiment of the present disclosure is as follows: obtaining an original sample; in the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence; adding the masked action frame sequence to the original sample to form a training sample; using the training sample to train the initial action model to obtain a digital human driving model. The implementation method of training the digital human driving model by using an action frame sequence including 2D action representations, randomly masking and combining with a voice feature frame sequence to form a training sample can simply and conveniently train a machine learning model for driving 2D limb movements, effectively reducing the construction complexity and resource consumption of the training samples, and enriching the diversity of the training samples to a certain extent, improving the training efficiency and reliability of the digital human driving model, and improving the realism of the generated driving movements.

[0065] On the basis of the above embodiments, in addition to being able to train a digital human driving model based on training samples including masked action frame sequences, action frame sequences and voice feature frame sequences, the embodiments of the present disclosure can also perform personalized model training on the original samples according to the application scenarios or the types of emotions contained in the samples, because different emotional states will be expressed through voice features and action features. For example, entertainment news corresponds to original samples with light-hearted and happy emotions, traditional news broadcasts correspond to original samples with neutral or serious emotions, natural disasters, accidents or tragic events correspond to original samples with sad emotions, and sports events correspond to original samples with tense and excited emotions, etc., so that the trained model is more targeted and the reliability of action prediction of the digital human driving model in different application scenarios is improved.

[0066] That is, original samples of different voice emotions can be grouped and independently trained to adapt to digital human-driven models of different voice emotions, so as to further improve the scene-specificity of the digital human-driven model and improve the final visual presentation effect and user experience.

[0067] Figure 2is a schematic diagram of another method for constructing a digital human driving model according to an embodiment of the present disclosure. In this embodiment, the operation of "obtaining original samples" in the above-mentioned embodiment is specified as: "obtaining a real person video, and performing image and voice sampling on the real person video respectively according to a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence; performing 2D key point identification in each image frame, and forming an action frame corresponding to each image frame according to the identification result; performing voice feature extraction in each voice frame, and forming a voice feature frame corresponding to each voice frame according to the extraction result; forming the original sample corresponding to the real person video according to an action frame sequence organized by the action frames and a voice feature frame sequence organized by the voice feature frames".

[0068] Correspondingly, as shown in Figure 2 , the method can specifically include:

[0069] S210, obtaining a real person video, and performing image and voice sampling on the real person video respectively according to a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence.

[0070] In this embodiment, the time-aligned action frame sequence and the voice frame sequence refer to that the same sampling rate is adopted for video image and audio, for example, both are 25 frames / s, and the action frame sequence and the voice frame sequence are collected.

[0071] Through time alignment, the action features and the voice features are associated, and through the associated features, model training is performed, so that the model can better understand and predict the action sequence, and the accuracy and naturalness of action prediction are improved.

[0072] In an optional implementation of this embodiment, in the real person video, only the upper body limb region of the real person is displayed.

[0073] Specifically, by focusing on the upper body limb region of the real person, the model can more accurately capture and learn the action features of the announcer, such as facial expressions and upper body actions, thereby improving the training efficiency and model performance, improving the quality of action synchronization and expression simulation, enhancing the authenticity and naturalness of the digital person in various application scenarios, and improving the visual effect and user experience.

[0074] S220, performing 2D key point identification in each image frame, and forming an action frame corresponding to each image frame according to the identification result.

[0075] Specifically, by adopting a pre-trained key point estimation model to perform 2D key point identification in each image frame, 2D key points including the head and the limbs can be identified in each image frame, and an action frame corresponding to each image frame is formed.

[0076] S230 , extracting speech features in each speech frame, and forming speech feature frames corresponding to each speech frame according to the extraction results.

[0077] Specifically, a pre-trained speech feature extraction model can be used to extract speech features from each speech frame to form a speech feature frame corresponding to each speech frame. Optionally, each speech feature frame can be understood as a feature sequence composed of multiple speech features.

[0078] S240. Form the original sample corresponding to the live-action video based on the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

[0079] S250 : Mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence.

[0080] In an optional implementation of this embodiment, in the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence may include:

[0081] Randomly select the target probability value within the preset probability interval;

[0082] Obtaining the total number of action frames contained in the action frame sequence of the original sample;

[0083] Calculating the masked number of action frames according to the total number of action frames and the target probability value;

[0084] In the action frame sequence of the original sample, action frames matching the mask quantity are randomly selected for masking to obtain a masked action frame sequence.

[0085] Specifically, a set number of action frames are masked according to a random masking rule, and the set number can be randomly selected according to the total number of action frames and a set optional interval range.

[0086] In a preset probability interval range, a target probability value is randomly selected, for example, the preset probability interval range is [30%, 70%], for each original sample, a target probability value is randomly selected, for example, 50%, the total number of action frames contained in the action frame sequence of the original sample is obtained, for example, 200 frames, the number of masked action frames is calculated according to the total number of action frames and the target probability value, the number of masked action frames = total number of action frames * target probability value, in the action frame sequence of the original sample, each action frame matching the number of masked action frames is randomly selected for masking processing, for example, the number of masked action frames is 200 frames * 50% = 100 frames, 100 frames of action frames at random positions are randomly selected for masking, and a masked action frame sequence is obtained.

[0087] By randomly masking a certain number of action frames at random positions in the action frame sequence, the data size is artificially enlarged without actually increasing the data set, more training samples are provided for the model, and the performance of the model is improved. Due to randomness, the model needs to pay more attention to time sequence information and learn the dependency between frames, thereby reducing the over-reliance of the model on specific training samples and reducing the risk of overfitting, thereby improving the reliability of the digital human driving model.

[0088] S260, the masked action frame sequence is added to the original sample to form a training sample.

[0089] S270, training an initial action model using the training sample to obtain a digital human driving model.

[0090] In an optional implementation of the embodiment, training an initial action model using the training sample to obtain a digital human driving model can include:

[0091] inputting the training sample into an initial action model;

[0092] generating a first token embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second token embedding vector sequence corresponding to the action frame sequence;

[0093] The first token embedding vector sequence includes an unknown vector corresponding to the masked action frame.

[0094] performing multi-level encoding on the first token embedding vector sequence through a model encoder layer to obtain an encoding result sequence;

[0095] performing multi-level decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each unknown vector;

[0096] Calculating a model loss value based on a prediction result vector for each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence;

[0097] The weight of the initial action model is updated according to the model loss value.

[0098] Optionally, the first labeled embedding vector sequence can be specifically understood as: the input masked action frame sequence is converted into an embedding vector form that the model can process (typically, Token embedding) through the Transformer model input layer. Specifically, in the model input layer, each action frame (i.e., key point coordinates) in the masked action frame sequence is projected into the embedding space through a projection matrix to obtain the labeled embedding vector. These embedding vectors can capture the connection between action frames. Similar action frames or action frames that often appear in succession have similar embedding vectors. These embedding vectors will be adjusted during the training process to better represent the characteristics of the action frames.

[0099] Specifically, each action frame in the masked action frame sequence (either an unmasked action frame or a masked action frame) corresponds to a first marker embedding vector. The difference is that an unmasked action frame corresponds to a known vector with a specific value, while a masked action frame corresponds to an unknown vector with an unspecified value.

[0100] The second marker embedding vector sequence can be specifically understood as: the input action frame sequence is converted into a numerical vector form that can be processed by the model through the Transformer model input layer. Since the second marker embedding vector sequence is generated by the action frame sequence, and each action frame in the action frame sequence is not masked, each second marker embedding vector in the second marker embedding vector sequence is a known vector, that is, the second marker embedding vector sequence contains known vectors corresponding to the masked action frames.

[0101] The encoder layer consists of a self-attention mechanism and a feedforward network, typically comprising multiple encoder layers. Each sublayer utilizes residual connections and layer normalization to stabilize training and improve performance, enabling the model to capture dependencies within the sequence. The encoded sequence can be understood as a sequence of results reflecting the internal relationships within the first token embedding sequence, obtained by performing multi-level encoding on the first token embedding sequence through the encoder layer. The decoder layer, typically comprising multiple decoders, combines information from the input sequence to predict an output sequence that matches the characteristics of the input sequence.

[0102] Specifically, training samples are input into the initial motion model. The model input layer generates a first sequence of labeled embedding vectors corresponding to the masked motion frame sequence, and a second sequence of labeled embedding vectors corresponding to the motion frame sequence. The model encoder layer performs multi-level encoding on the first sequence of labeled embedding vectors to obtain an encoded result sequence. The model decoder layer performs multi-level decoding on the sum of the encoded result sequence and the speech feature frame sequence, for example by directly adding the feature vectors of the two modalities to fuse the two feature information. The input is processed step by step through multiple decoder layers, and the prediction result is gradually refined to ultimately obtain a prediction result vector for each unknown vector. Based on the prediction result vector for each unknown vector and the known vectors corresponding to each unknown vector in the second sequence of labeled embedding vectors, a model loss value, such as diffusion loss, is calculated. The weights of the initial motion model are updated based on the model loss value. By using training samples formed by random masking of 2D motion representations and combining them with speech feature information, the model weight parameters are optimized through continuous training, verification, and tuning, bringing the model to an optimal state, thereby improving the accuracy of motion prediction and the reliability of the digital human driving model.

[0103] In another optional implementation of this embodiment, calculating the model loss value based on the prediction result vector for each unknown vector and the known vector corresponding to each unknown vector in the second label embedding vector sequence may include:

[0104] A diffusion loss function is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors.

[0105] The diffusion loss function measures the difference between the model's predicted masked motion and the actual motion. During model training, the model needs to learn how to recover the original motion data from the masked motion data. The diffusion loss function is used to guide this process, ensuring that the model's predictions are as close as possible to the actual motion data.

[0106] Specifically, during the digital human-driven training process, at each training step, we know the preceding and following (or contextual) action frames, as well as the true value of the current predicted action frame (the coordinates of each pixel in a frame of action). Therefore, we simply use the preceding and following action frames as constraints and the true value of the current predicted action frame to train the constrained diffusion model. The error function that implicitly describes the next action frame is called the diffusion loss function.

[0107] Compared to the most basic constrained denoising diffusion probability model (DDPM), this diffusion loss function incorporates constraints into its calculation formula. This constraint information can be understood as the output of the initial motion model for the preceding and following action frames in the action frame sequence. Because the constraints come from the initial motion model, the gradient of the diffusion loss function can be propagated back to the parameters of the initial motion model through this constraint information, enabling the model training process.

[0108] Specifically, a diffusion loss function is used to calculate the model loss between the predicted result vector and the known vector corresponding to each unknown vector. By minimizing the difference between the predicted masked motion and the actual motion, the model can better learn the distribution of data, thereby generating higher-quality predicted motions, improving the stability of model training, and reducing fluctuations during training, thereby accelerating convergence and improving the training efficiency and reliability of the digital human-driven model.

[0109] The technical solution of the disclosed embodiment obtains a live video and samples the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and voice frame sequence; performs 2D key point recognition in each image frame and, based on the recognition results, generates an action frame corresponding to each image frame; extracts voice features in each voice frame and, based on the extraction results, generates a voice feature frame corresponding to each voice frame; forms an original sample corresponding to the live video based on the action frame sequence obtained from each action frame and the voice feature frame sequence obtained from each voice feature frame; in the action frame sequence of the original sample, a set number of action frames are masked according to a random masking rule to obtain a masked action frame sequence; the masked action frame sequence is added to the original sample to form a training sample; and the training sample is used to train an initial action model to obtain a digital human driven model. By acquiring live video and collecting action and voice features to train the model, the accuracy of action recognition can be improved, the generalization ability of the model in real-world language feature recognition and action prediction tasks can be enhanced, action generation and prediction can be improved, and the naturalness and realism of digital human expressions and actions can be enhanced, with rich details, and improved visual effects and user experience.

[0110] Figure 3 This is a schematic diagram of a digital human driving method provided according to an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to situations where a trained neural network model (digital human driving model) is used to generate matching actions based on speech prediction. This method can be performed by a digital human driving device, which can be implemented in hardware and / or software and can generally be configured in an electronic device.

[0111] Correspondingly, such as Figure 3 As shown, the method may specifically include:

[0112] S310: Obtain a target voice for voice-driven 2D body movements of the target virtual image.

[0113] The target virtual image includes a head and limb areas.

[0114] Specifically, the target avatar can be understood as a voice-driven virtual character or image created in digital media, games, or virtual reality. Here, the avatar refers to a 2D avatar consisting of a head and limbs. Specifically, the target voice can be understood as the voice data used to drive the 2D body movements of the target avatar.

[0115] S320: Generate a target speech feature frame sequence that matches the target speech, and construct an unknown action frame sequence that matches the target speech feature frame sequence.

[0116] The target speech feature frame sequence can be specifically understood as a sequence of feature frames that are extracted by the speech feature extraction model and represent speech characteristics that match the target speech. The unknown action frame sequence can be specifically understood as an action frame sequence that needs to be predicted or generated by the algorithm model.

[0117] Generally speaking, since the action frame sequence and the speech feature frame sequence need to be time-aligned, the length of the unknown action frame sequence is usually the same as the length of the target speech feature frame sequence. It can be understood that each frame in the unknown action frame sequence is a blank frame and does not contain any 2D key points.

[0118] S330 : Fill at least one required action frame that matches the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence.

[0119] The required action frame can be specifically understood as: a 2D body movement that is expected or preset to be performed by the target virtual image at a certain moment, such as raising a hand or standing still. Based on this moment, the frame position of the target virtual image in the unknown action frame sequence can be limited. The position of the required action frame can be the head, middle or tail of the unknown action frame sequence. The local unknown action frame sequence can be specifically understood as: after filling at least one required action frame into the target frame position in the unknown action frame sequence, the unknown action frame sequence forms a local unknown action frame sequence, part of which is a known action frame.

[0120] S340: Input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence that matches the target speech.

[0121] The digital human driving model is obtained by training using the method described in any one of the embodiments of the present disclosure.

[0122] The known action frame sequence can be specifically understood as: inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model trained by the aforementioned method, and starting from the known action frame in the local unknown action frame sequence according to the speech characteristics of the target speech, predicting the previous one or several unknown action frames of the action frame (when the required action frame is at the end of the unknown action frame sequence, only the previous unknown action frame is included), the next one or several unknown action frames (when the required action frame is at the head of the unknown action frame sequence, only the next unknown action frame is included), or, at the same time, predicting the previous and next one or several unknown action frames.

[0123] Through multiple predictions, the previous and next unknown action frames are predicted based on the newly predicted action frame, and the complete prediction of the unknown action frames is performed in this way until all unknown action frames in the local unknown action frame sequence are predicted and filled with prediction results.

[0124] Generally speaking, when the desired action frame is inserted at the beginning of the unknown action frame sequence, the order in which action predictions are generated aligns with the order in which the speech is delivered. Therefore, the action can be predicted and generated in real time as the speech is delivered. However, when the desired action frame is inserted in the middle or end of the unknown action frame sequence, the order in which action predictions are generated is reversed from the order in which the speech is delivered. Therefore, it can only be used for offline prediction and generation of actions that match the speech.

[0125] The technical solution of the disclosed embodiment is as follows: obtaining a target voice for voice-driven 2D body movements of a target virtual image; generating a target voice feature frame sequence matching the target voice, and constructing an unknown action frame sequence matching the target voice feature frame sequence; filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a local unknown action frame sequence; inputting the target voice feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target voice. By randomly filling at least one required action frame at any position and combining the target voice feature frame sequence for action prediction, the accuracy of key actions is ensured, the adaptability of predicted actions to different voice changes is improved, a coherent action transition is generated before and after the key frame, the authenticity and naturalness of the action are improved, the randomness of the filling position and number of required action frames is improved, the diversity of action prediction results is improved, and the flexibility and generalization ability of digital human driving are improved.

[0126] On the basis of the above-mentioned embodiments, the embodiments of the present disclosure can perform personalized model training on the original samples according to the application scenarios or the types of emotions contained in the samples to obtain targeted digital human driving models. Therefore, it is also possible to select a matching digital human driving model according to the application scenarios or the types of emotional colors of the voice, thereby improving the reliability of action prediction of digital human driving in different application scenarios.

[0127] Figure 4 Schematic diagram of another digital human driving method provided according to an embodiment of the present disclosure. This embodiment is a refinement of the above embodiments. In this embodiment, the operation of "inputting the target speech feature frame sequence and the locally unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target speech" is specifically implemented as follows: "inputting the target speech feature frame sequence and the locally unknown action frame sequence into the digital human driving model; generating an original tag embedding vector sequence matching the locally unknown action frame sequence through a model input layer; generating a target encoding result sequence matching the original tag embedding vector through a model encoder layer; obtaining a prediction result vector for at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence through a model decoder layer; replacing each target unknown vector in the original tag embedding vector sequence with each prediction result vector, and then returning to the operation of generating an original tag embedding vector sequence matching the locally unknown action frame sequence through the model input layer until all unknown vectors in the original tag embedding vector are predicted; and outputting the known action frame sequence matching the predicted original tag embedding vector through a model output layer."

[0128] Correspondingly, such as Figure 4 As shown, the method may specifically include:

[0129] S410: Obtain a target voice for voice-driven 2D body movements of a target virtual image.

[0130] The target virtual image includes a head and limb areas.

[0131] S420: Generate a target speech feature frame sequence that matches the target speech, and construct an unknown action frame sequence that matches the target speech feature frame sequence.

[0132] S430 : Fill at least one required action frame that matches the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence.

[0133] S440: Input the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model.

[0134] S450: Generate an original tag embedding vector sequence matching the local unknown action frame sequence through a model input layer.

[0135] The original labeled embedding vector sequence can be specifically understood as: through the input layer of the digital human driving model, each action frame in the input local unknown action frame sequence is converted into an embedding vector form that the model can process. The sequence composed of these embedding vectors is the original labeled embedding vector sequence.

[0136] Among them, in the original tag embedding vector sequence, the original tag embedding vector that matches the required action frame is a known vector, and the original tag embedding vector that matches the non-required action frame (unknown action frame) is an unknown vector. That is, the original tag embedding vector sequence contains at least one unknown vector.

[0137] S460. Generate a target encoding result sequence that matches the original tag embedding vector through a model encoder layer.

[0138] The target encoding result sequence can be specifically understood as: a sequence of encoding results that reflects the internal relationship of the original tag embedding vector obtained by multi-level encoding of the original tag embedding vector through the encoder layer of the digital human-driven model.

[0139] The target encoding result sequence that matches the original tag embedding vector is generated through the multi-layer encoder in the model encoder layer. This sequence contains the association information between the unknown action frames to be predicted and the known required action frames in the subsequent action prediction, ensuring the relevance, continuity and naturalness of the subsequent action prediction results.

[0140] S470. Obtain a prediction result vector for at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence through the model decoder layer.

[0141] The target unknown vector can be specifically understood as: the unknown vector in the original tag embedding vector predicted by the digital human-driven model in the current prediction round.

[0142] Specifically, based on the sum of the target encoding result sequence and the target speech feature frame sequence, for example, the feature vectors of the two modalities are directly added together, the two feature information are fused, and through multi-layer decoding of the model decoder layer, the input is processed step by step, and the prediction results are gradually refined to obtain a prediction result vector of at least one target unknown vector in the original tag embedding vector.

[0143] In an optional implementation of this embodiment, obtaining, by the model decoder layer, a prediction result vector for at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence may include:

[0144] Obtaining, through the model decoder layer, a quantity value of unknown vectors contained in the original tag embedding vector, and randomly determining a target quantity based on the quantity value;

[0145] Randomly selecting target unknown vectors matching the target number from all unknown vectors in the original tag embedding vector sequence through the model decoder layer;

[0146] The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

[0147] Specifically, the model decoder layer obtains the number of unknown vectors to be predicted in the original tag embedding vector, and randomly determines the number of unknown vectors to be predicted within the range of the unknown vector number, i.e., the target number. For example, when the number of unknown vectors is 4, the target number can be 1, 2, 3, or 4. The model decoder layer randomly selects target unknown vectors that match the target number from all unknown vectors in the original tag embedding vector sequence, i.e., selects target unknown vectors at a target number of random positions in the original tag embedding vector sequence as the unknown vectors for this prediction. The model decoder obtains the prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

[0148] By predicting a random number of unknown vectors at random positions in each prediction process, the corresponding action frames generated have greater randomness, increasing the diversity of the prediction results. For the same input speech, each prediction may produce a different action sequence, bringing diverse visual effects and improving the user experience.

[0149] S480. After replacing each of the target unknown vectors in the original tag embedding vector sequence with each of the prediction result vectors, return to executing the operation of generating an original tag embedding vector sequence that matches the local unknown action frame sequence through the model input layer until the prediction of all unknown vectors in the original tag embedding vector is completed.

[0150] S490. Outputting, through the model output layer, the known action frame sequence that matches the original tag embedding vector that completes the prediction.

[0151] Specifically, after using each predicted result vector to replace each target unknown vector in the original labeled embedding vector sequence at the corresponding position, the operation of generating the original labeled embedding vector sequence that matches the local unknown action frame sequence through the model input layer is returned and re-executed. The unknown vectors corresponding to one or several unknown action frames before and after the known action frame are predicted and filled until all unknown vectors in the original labeled embedding vector are predicted, and the prediction results are filled into the corresponding target unknown vector in the original labeled embedding vector sequence. The known action frame sequence that matches the predicted original labeled embedding vector is output through the model output layer.

[0152] The technical solution of the embodiment of the present disclosure obtains the target speech, generates a target speech feature frame sequence, and constructs an unknown action frame sequence; fills at least one required action frame into the unknown action frame sequence, inputs the obtained local unknown action frame sequence and the target speech feature frame sequence into the digital human driving model, generates an original tag embedding vector sequence through the input layer, generates a target encoding result sequence through the encoder layer, obtains a prediction result vector of at least one target unknown vector in the original tag embedding vector based on the sum of the target encoding result sequence and the target speech feature frame sequence through the decoder layer, replaces each target unknown vector in the original tag embedding vector sequence with each prediction result vector, and then returns to execute the operation of generating the original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer until the prediction of all unknown vectors in the original tag embedding vector is completed; and outputs the known action frame sequence matching the predicted original tag embedding vector through the model output layer. Combined with speech feature information, the pre-trained digital human driving model is used to predict the unknown vectors in the original labeled embedding vector constructed from the unknown action frame sequence, and finally a known action frame sequence that matches the original labeled embedding vector is obtained. Through the pre-trained initial action model, the complex mapping from speech to body movements can be learned, thereby improving the accuracy and reliability of action prediction, and adaptively combining the rhythm and content of the audio to generate head or upper body movements synchronized with the audio, thereby improving the visual effects of the digital human or virtual character and enhancing the user experience.

[0153] In an optional implementation of this embodiment, filling at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence may include:

[0154] Acquire at least one item of requirement description information matching the target speech, wherein the requirement description information includes a requirement time point and a requirement action;

[0155] generating at least one required action frame for the target virtual image according to the required action;

[0156] Locating at least one target unknown action frame in the unknown action frame sequence according to the required time point;

[0157] In the unknown action frame sequence, each target unknown action frame is replaced with the matching required action frame to obtain the local unknown action frame sequence.

[0158] Specifically, the at least one required action frame that matches the target avatar can be specifically set based on the requirements of the target speech. For example, at a specific moment in the broadcast, the digital human is required to perform a hand-raising gesture. Therefore, at least one required action description information matching the target speech can be obtained. This required action description information records the required time point and the corresponding specific required action in the target speech. Based on the motion process included in the required action, at least one required action frame can be generated for the target avatar. Each required action frame records the 2D head and limb key points corresponding to the current motion process of the required action. Based on the required time point, the target unknown action frame corresponding to the required time point is located in the unknown action frame sequence and replaced with the matching required action frame. After replacing the target unknown action frames corresponding to each required time point in the required action description information with the matching required action frame, a partial unknown action frame sequence is obtained. By adding preset key actions at specified time positions based on the actual action requirements of the speech, the action prediction results are diverse, controllable, and flexible, making the generated actions more natural and realistic, enhancing the visual effects of the digital human or virtual character, and improving the user experience.

[0159] In another optional implementation of this embodiment, filling at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence may include:

[0160] When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result;

[0161] The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

[0162] Specifically, at least one required action frame that the target virtual image matches can be specifically set according to the requirements of the target voice. For example, at a specific moment in the broadcast, the digital human is required to be in a stationary state. 2D key point recognition is performed on the target virtual image in a stationary state, and a required action frame is formed based on the recognition result. The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence. In human motion prediction, when the first frame is in a non-stationary state, its head or limbs are often in a special position or at a certain angle. Due to the influence of uncertain factors such as movement speed and amplitude, the subsequently generated action frames are often discontinuous with the first frame in terms of action performance. In order to generate the continuity and naturalness of the action, using the stationary state as the starting point can reduce this discontinuity, making the predicted action more stable and accurate, enhancing the realism and naturalness of the digital person or virtual character's action, and improving the user experience.

[0163] In another optional implementation of this embodiment, filling at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence may include:

[0164] If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target avatar;

[0165] generating the required action frame according to the end action;

[0166] The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

[0167] Specifically, the target voice has a previous related voice, that is, the ending voice and action of the previous related voice are consistent with the beginning voice and action of the target voice. In order to ensure the continuity of the generated action, the ending action of the target virtual image when broadcasting the previous related voice can be obtained, and the corresponding required action frame can be generated. The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain a local unknown action frame sequence, realizing the model's global action prediction of the voice file, which can make the predicted action smooth at the boundary of the target voice file, realize the naturalness and smoothness of the action switching, enhance the realism of the digital person or virtual character's action, and improve the user experience.

[0168] In another optional implementation of this embodiment, the digital human driving method may further include:

[0169] When the condition for using the target virtual image to broadcast the target voice is met, forming an image frame sequence that matches the target virtual image based on the known action frame sequence;

[0170] The image frame sequence and the target voice are combined to obtain a target video, and the target video is played.

[0171] Specifically, when the conditions for using a target virtual image to broadcast the target voice are met, that is, after a known action frame sequence is generated by a pre-trained digital human-driven model based on the target voice and at least one required action frame, an image frame sequence matching the target virtual image is formed based on the known action frame sequence. That is, the key points of the head and limbs recorded in the action frame are associated with the image matching the target virtual image, so that the target virtual image performs the corresponding action in the corresponding action frame, and the image corresponding to the current target virtual image is used as the image frame matching the action frame. Thus, an image frame sequence matching the known action frame sequence is obtained, and the image frame sequence and the target voice are combined to obtain a target video, which is then played. By combining the known action frame sequence with the target virtual image and the target voice to obtain the target video, the randomness and diversity of the known action frame sequence can bring different forms of expression to the video content, increasing the richness and innovation of the video content. Users can see the virtual image react in real time according to the voice content. This interactivity can enhance the audience's sense of participation and immersion.

[0172] For ease of understanding, the specific application scenarios applicable to each embodiment of the present disclosure are now described. Existing solutions for driving the whole-body movement of digital humans require good 3D motion data. However, the data acquisition process has strict requirements on the shooting environment and conditions, which not only limits the flexibility of data acquisition, but also significantly increases the cost, making it difficult to carry out large-scale data acquisition. Even if 3D motion data is collected under ideal conditions, when it is rendered onto a two-dimensional plane, problems such as motion distortion or unnaturalness will be encountered, which directly affects the final visual effect and user experience. To solve the above problems, the embodiments of the present disclosure propose a method for constructing a digital human driving model and a digital human driving method.

[0173] The construction method of the digital human-driven model includes two important operations, one is the data preprocessing stage, and the other is the model training stage. The digital human-driven method corresponds to one important operation, namely the inference and prediction stage.

[0174] 1. Data preprocessing stage

[0175] Obtain multiple videos and use a keypoint estimation model to extract the 2D keypoint coordinates in each broadcast image. The videos can be real people, with the upper body in the frame.

[0176] Specifically, in Figure 5 FIG. 1 shows a schematic diagram of an action frame applicable to an embodiment of the present disclosure. Figure 5As shown in the figure, by processing each broadcast image, a total of 122 key points, including those on the face, hands, and upper body, are extracted to form a sequence of action frames. Simultaneously, a pre-trained Wav2Vec 2.0 model is used to extract speech features from the speech and time-align them with the action frame sequence. This lays the foundation for subsequent models to learn the mapping between speech and action. Time alignment can be achieved by maintaining a consistent sampling rate, for example, 25 frames / s. Audio content can also be understood as frames, meaning that a pre-trained Wav2Vec 2.0 model is used to extract speech feature frame sequences.

[0177] 2. Model training phase

[0178] Specifically, a Transformer-based neural network model is used during model training. This Transformer-based neural network model consists of an encoder layer consisting of L stacked encoders and a decoder layer consisting of L stacked decoders. Before model training, a masking mechanism is used to randomly mask portions of the action frames in each action frame sequence. For example, a probability interval is first set, such as [30%, 70%]. Then, for each action frame sequence, a probability value, such as 50%, is randomly determined. 50% of the action frames in this sequence are then randomly masked and input into the neural network model. The model then predicts the masked action frames based on the synchronously input, time-aligned speech feature frame sequence. The model then learns to complete the masked action frames from the speech. A diffusion loss function is used for supervision during training, ultimately resulting in a trained Transformer-based neural network model, i.e., the digital human-driven model.

[0179] 3. Reasoning and prediction stage.

[0180] Specifically, in Figure 6 This is a schematic diagram illustrating a scenario where voice-driven 2D full-body motion is performed using the first frame, applicable to embodiments of the present disclosure. This application scenario specifically describes the steps involved in training a digital human driving model based on a video-based mapping relationship between voice and 2D body motion, generating matching motions based on this digital human driving model and voice prediction, and ultimately producing a matching avatar video.

[0181] Specifically, such as Figure 6 As shown in Figure 2, Token embedding refers to the embedded vector sequence obtained through a projection matrix transformation, that is, the input masking action frame sequence is converted into an embedded vector form that can be processed by the model through the Transformer model input layer.

[0182] Among them, the first-frame speech-driven 2D full-body motion method specifically includes: by inputting a piece of speech and a frame of known motion as the first frame (i.e., the reference frame), allowing the model to sequentially predict the 2D key point coordinates of the remaining frames in an autoregressive manner.

[0183] Specifically, the input reference frame can obtain a corresponding action frame through the whole-body key point estimation model. After the action frame sequence with the action frame as the first frame and the speech feature frame sequence matching the speech are input into the digital human driving model, the input layer of the model first converts the action frame sequence into a marker embedding vector sequence, and the first marker embedding vector in the marker embedding vector sequence is a known token, and the rest are unknown tokens.

[0184] The L Transformer encoders of this model use multi-stage encoding to first generate a coding result sequence for the tag embedding vector sequence. In this coding result sequence, only the first coding result is a known token, and the remaining coding results are unknown tokens. The above coding result sequence is summed with the speech feature sequence of this speech, and then input into the L Transformer decoders of the model. The L Transformer decoders use multi-stage decoding to obtain the prediction result of the second tag embedding vector in the tag embedding vector sequence, that is, the predicted token.

[0185] After taking the first and second token embedding vectors in the token embedding vector sequence as known tokens, we can proceed to the next round of prediction to obtain the prediction result of the third token embedding vector. After performing the above operation T times, we will finally obtain the prediction result of the action frame sequence that matches the speech.

[0186] also, Figure 7 This is another schematic diagram of an application scenario of general voice-driven 2D full-body motion applicable to the embodiments of the present disclosure, such as Figure 7 As shown, during model inference, the input known action frames can be the desired action frames at a set time point, and the number of known action frames can be one or more. For example, at what time point do we want the digital figure to raise its hand? During each inference, predictions can be made at any one or more positions before or after the known portion, randomly determined based on the known portion, to increase the diversity of action generation.

[0187] and Figure 6 The implementation method of the general voice-driven 2D full-body motion method is similar to that of the general voice-driven 2D full-body motion method. Specifically, the method includes: inputting a piece of voice and a frame of known motion as the action frame at the set position (i.e., the reference frame), and allowing the model to randomly predict the 2D key point coordinates of the remaining frames in an autoregressive manner.

[0188] Specifically, the input reference frame can obtain a corresponding known action frame through the whole-body key point estimation model. After filling the known action frame into any position in the action frame sequence, the action frame sequence and the speech feature frame sequence matching the speech are input into the digital human driving model. The input layer of the model first converts the action frame sequence into a marker embedding vector sequence, and the marker embedding vector in the marker embedding vector sequence that matches the known action frame is a known token, and the rest are unknown tokens.

[0189] The L Transformer encoders of the model first generate a coding result sequence for the tag embedding vector sequence through multi-stage encoding. In this coding result sequence, only the coding results that match the known action frame are known tokens, and the remaining coding results are unknown tokens. The above coding result sequence is summed with the speech feature sequence of this speech and then input into the L Transformer decoders of the model. The L Transformer decoders obtain the prediction results of the tag embedding vectors at any one or more positions in the tag embedding vector sequence through multi-stage decoding, that is, the predicted token.

[0190] After taking the predicted token embedding vector in the token embedding vector sequence and the token embedding vector matching the known action frame as the known token, we can proceed to the next round of prediction to obtain the prediction results of the token embedding vector at any one or more positions in the token embedding vector sequence. After performing the above operation T times, we will finally obtain the prediction results of the action frame sequence matching this speech.

[0191] This scheme uses a random prediction order, including the random number of action frames predicted each time and the random position of the action frames, to enhance the diversity of generated actions and avoid generating a single result.

[0192] Through the inference method of interpolation action completion, since the model can accept known action frames at any time and in any number as input, during inference, the key frame of a certain action can be input into the model, allowing the model to automatically complete the actions before and after the key frame. In this way, flexible action editing needs can be met.

[0193] In addition, different types of models can be selected for training according to different types of broadcast content (happy or sad) to improve the adaptability of the model in predicting actions.

[0194] The technical solutions proposed in the disclosed embodiments reduce computational complexity and resource consumption, enrich training samples, and improve the training efficiency and reliability of the digital human-driven model. By randomly filling at least one required action frame at any location and combining it with a target speech feature frame sequence for action prediction, the accuracy of key actions is ensured, the predicted action's adaptability to varying speech variations is improved, and coherent action transitions are generated before and after key frames, enhancing the authenticity and naturalness of the actions. The randomness of the placement and number of required action frames increases the diversity of action prediction results, improving the flexibility and generalization of digital human-driven models.

[0195] It's worth reiterating that this disclosed implementation proposes a high-quality, voice-driven 2D full-body motion solution, supporting the subsequent generation of motion-based digital human videos. By employing 2D motion representation and innovative model training and prediction strategies, this implementation overcomes the shortcomings of related technologies in terms of computational complexity, resource consumption, data acquisition difficulties, and generation diversity. The resulting motion sequences are natural, realistic, and rich in detail, offering advantages such as simplicity and efficiency. This approach is widely applicable to fields requiring voice-driven full-body motion, such as digital human live streaming, intelligent customer service, and virtual characters.

[0196] As an implementation of the above-mentioned methods for constructing digital human driving models, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned methods for constructing digital human driving models.

[0197] Figure 8 FIG is a structural diagram of a device for constructing a digital human driving model according to an embodiment of the present disclosure. Figure 8 The device includes: an original sample acquisition module 810, a mask processing module 820, a training sample acquisition module 830 and a model training module 840, wherein:

[0198] The original sample acquisition module 810 is used to acquire original samples; wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0199] a masking processing module 820 configured to mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence;

[0200] A training sample acquisition module 830 is configured to add the masking action frame sequence to the original sample to form a training sample;

[0201] The model training module 840 is used to train the initial motion model using the training samples to obtain a digital human driving model.

[0202] The technical solution of the disclosed embodiment comprises obtaining an original sample; masking a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence; adding the masked action frame sequence to the original sample to form a training sample; and using the training sample to train the initial action model to obtain a digital human driving model. The training sample is formed by randomly masking an action frame sequence including 2D action representations and combining it with a speech feature frame sequence to obtain a digital human driving model. This reduces computational complexity and resource consumption, enriches the training samples, improves the training efficiency and reliability of the digital human driving model, and enhances the realism of the generated driving actions.

[0203] Based on the above embodiments, the original sample acquisition module 810 is specifically configured to:

[0204] Acquire a live video, and sample the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence;

[0205] Performing 2D key point recognition in each image frame, and forming action frames corresponding to each of the image frames based on the recognition results;

[0206] Extracting speech features in each speech frame, and forming speech feature frames corresponding to each speech frame according to the extraction results;

[0207] The original sample corresponding to the live video is formed based on the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

[0208] Based on the above embodiments, in the real-person video, only the upper body area of ​​the real person is displayed.

[0209] Based on the above embodiments, the mask processing module 820 is specifically configured to:

[0210] Randomly select the target probability value within the preset probability interval;

[0211] Obtaining the total number of action frames contained in the action frame sequence of the original sample;

[0212] Calculating the masked number of action frames according to the total number of action frames and the target probability value;

[0213] In the action frame sequence of the original sample, action frames matching the mask quantity are randomly selected for masking to obtain a masked action frame sequence.

[0214] On the basis of each of the above embodiments, the model training module 840 is specifically configured to:

[0215] input the training sample into an initial action model;

[0216] generate, through a model input layer, a first token embedding vector sequence corresponding to the masked action frame sequence, and generate a second token embedding vector sequence corresponding to the action frame sequence;

[0217] The first token embedding vector sequence includes an unknown vector corresponding to a masked action frame.

[0218] perform multi-level encoding on the first token embedding vector sequence through a model encoder layer to obtain an encoding result sequence;

[0219] perform multi-level decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each unknown vector;

[0220] According to the prediction result vector for each unknown vector and the known vector corresponding to each unknown vector in the second token embedding vector sequence, a model loss value is calculated.

[0221] According to the model loss value, the initial action model is updated in weight.

[0222] On the basis of each of the above embodiments, the model training module 840 is further configured to:

[0223] The diffusion loss function is used to calculate the model loss value between the prediction result vector corresponding to each unknown vector and the known vector.

[0224] The above product can execute the construction method of the digital human driving model provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of executing the construction method of the digital human driving model.

[0225] As an implementation of the above digital human driving method, the present disclosure further provides an optional embodiment of an execution device for implementing the above digital human driving method.

[0226] Figure 9 is a structural diagram of a digital human driving device according to an embodiment of the present disclosure. As Figure 9 described, the device includes a speech acquisition module 910, a first sequence construction module 920, a second sequence acquisition module 930, and a third sequence acquisition module 940, wherein:

[0227] A voice acquisition module 910 is configured to acquire a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions;

[0228] A first sequence construction module 920 is configured to generate a target speech feature frame sequence matching the target speech, and to construct an unknown action frame sequence matching the target speech feature frame sequence;

[0229] A second sequence acquisition module 930 is configured to fill the unknown action frame sequence with at least one required action frame that matches the target avatar, to obtain a partial unknown action frame sequence;

[0230] The third sequence acquisition module 940 is used to input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence matching the target speech.

[0231] The digital human driving model is trained by the method described in any one of the embodiments of the present disclosure.

[0232] The technical solution of the disclosed embodiment is as follows: obtaining a target voice for voice-driven 2D body movements of a target virtual image; generating a target voice feature frame sequence matching the target voice, and constructing an unknown action frame sequence matching the target voice feature frame sequence; filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a local unknown action frame sequence; inputting the target voice feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target voice. By randomly filling at least one required action frame at any position and combining the target voice feature frame sequence for action prediction, the accuracy of key actions is ensured, the adaptability of predicted actions to different voice changes is improved, a coherent action transition is generated before and after the key frame, the authenticity and naturalness of the action are improved, the randomness of the filling position and number of required action frames is improved, the diversity of action prediction results is improved, and the flexibility and generalization ability of digital human driving are improved.

[0233] Based on the above embodiments, the second sequence acquisition module 930 is specifically configured to:

[0234] Acquire at least one item of requirement description information matching the target speech, wherein the requirement description information includes a requirement time point and a requirement action;

[0235] generating at least one required action frame for the target virtual image according to the required action;

[0236] locating at least one target unknown action frame in the unknown action frame sequence according to the demand time point;

[0237] replacing each target unknown action frame in the unknown action frame sequence with a matched demand action frame to obtain the local unknown action frame sequence.

[0238] In the above embodiments, the second sequence obtaining module 930 is specifically configured to:

[0239] performing 2D key point recognition on the target virtual image when the target virtual image is in a static state, and forming a demand action frame according to the recognition result;

[0240] replacing a first action frame in the unknown action frame sequence with the demand action frame to obtain the local unknown action frame sequence.

[0241] In the above embodiments, the second sequence obtaining module 930 is specifically configured to:

[0242] if it is determined that the target speech has a previous associated speech, obtaining an ending action of the previous associated speech of the target virtual image;

[0243] generating the demand action frame according to the ending action;

[0244] replacing a first action frame in the unknown action frame sequence with the demand action frame to obtain the local unknown action frame sequence.

[0245] In the above embodiments, the third sequence obtaining module 940 is specifically configured to:

[0246] inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model;

[0247] generating an original label embedding vector sequence matched with the local unknown action frame sequence through a model input layer;

[0248] generating a target coding result sequence matched with the original label embedding vector through a model encoder layer;

[0249] obtaining a prediction result vector of at least one target unknown vector in the original label embedding vector according to a sum result of the target coding result sequence and the target speech feature frame sequence through a model decoder layer;

[0250] After replacing each of the target unknown vectors in the original labeled embedding vector sequence with each of the predicted result vectors, returning to the operation of generating an original labeled embedding vector sequence matching the local unknown action frame sequence through the model input layer until all unknown vectors in the original labeled embedding vector are predicted;

[0251] The model output layer outputs the known action frame sequence that matches the original label embedding vector that completes the prediction.

[0252] Based on the above embodiments, the third sequence acquisition module 940 is further configured to:

[0253] Obtaining, through the model decoder layer, a quantity value of unknown vectors contained in the original tag embedding vector, and randomly determining a target quantity based on the quantity value;

[0254] Randomly selecting target unknown vectors matching the target number from all unknown vectors in the original tag embedding vector sequence through the model decoder layer;

[0255] The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

[0256] Based on the above embodiments, the device further includes: an image frame sequence acquisition module and a target video acquisition module, wherein:

[0257] an image frame sequence acquisition module, configured to generate an image frame sequence matching the target virtual image based on the known action frame sequence when a condition for adopting the target virtual image to broadcast the target voice is met;

[0258] The target video acquisition module is used to combine the image frame sequence and the target voice to obtain a target video, and play the target video.

[0259] The above-mentioned product can execute the digital human driving method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the digital human driving method.

[0260] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0261] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0262] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0263] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0264] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0265] The computing unit 1001 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 executes the various methods and processes described above, such as executing a method for constructing a digital human-driven model, namely:

[0266] Obtaining original samples, wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points;

[0267] In the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence;

[0268] Adding the masking action frame sequence to the original sample to form a training sample;

[0269] The initial motion model is trained using the training samples to obtain a digital human driving model.

[0270] Alternatively, implement a digital human-driven approach, which means:

[0271] Obtaining a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions;

[0272] generating a target speech feature frame sequence matching the target speech, and constructing an unknown action frame sequence matching the target speech feature frame sequence;

[0273] Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence;

[0274] Inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence that matches the target speech;

[0275] The digital human driving model is obtained by training using the method described in any one of the embodiments of the present disclosure.

[0276] For example, in some embodiments, the method for constructing a digital human-driven model or the method for driving a digital human may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method for constructing a digital human-driven model or the method for driving a digital human described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the method for constructing a digital human-driven model or the method for driving a digital human by any other appropriate means (e.g., by means of firmware).

[0277] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0278] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0279] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0280] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0281] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0282] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0283] Artificial intelligence is a discipline that studies enabling computers to simulate some human thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning technology, big data processing technology, knowledge graph technology, etc. several major directions.

[0284] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0285] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0286] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for constructing a digital human-driven model, comprising: Obtaining original samples, wherein the original samples include a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points; In the action frame sequence of the original sample, masking a set number of action frames according to a random masking rule to obtain a masked action frame sequence; Adding the masking action frame sequence to the original sample to form a training sample; Using the training samples to train the initial motion model, to obtain a digital human driving model for driving 2D limb movements; The step of using the training samples to train the initial motion model to obtain a digital human model for driving 2D limb motions includes: Inputting the training samples into an initial neural network model; Generating a first labeled embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second labeled embedding vector sequence corresponding to the action frame sequence; wherein the first marker embedding vector sequence includes an unknown vector corresponding to the masked action frame; Performing multi-level encoding on the first tag embedding vector sequence through a model encoder layer to obtain an encoding result sequence; Performing multi-stage decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each of the unknown vectors; Calculating a model loss value based on a prediction result vector for each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence; The weight of the initial action model is updated according to the model loss value.

2. The method according to claim 1, wherein The obtaining of the original sample comprises: Acquire a live video, and sample the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and voice frame sequence; Performing 2D key point recognition in each image frame, and forming action frames corresponding to each of the image frames based on the recognition results; Extracting speech features in each speech frame, and forming speech feature frames corresponding to each speech frame according to the extraction results; The original sample corresponding to the live video is formed based on the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

3. The method according to claim 2, wherein: In the real-person video, only the upper body area of ​​the real person is displayed.

4. The method according to claim 1, wherein The step of masking a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence includes: Randomly select the target probability value within the preset probability interval; Obtaining the total number of action frames contained in the action frame sequence of the original sample; Calculating the masked number of action frames according to the total number of action frames and the target probability value; In the action frame sequence of the original sample, action frames matching the mask quantity are randomly selected for masking to obtain a masked action frame sequence.

5. The method according to claim 1, wherein The calculating the model loss value based on the prediction result vector of each unknown vector and the known vector corresponding to each unknown vector in the second tag embedding vector sequence includes: A diffusion loss function is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors.

6. A digital human driving method, comprising: Obtaining a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and body regions; generating a target speech feature frame sequence matching the target speech, and constructing an unknown action frame sequence matching the target speech feature frame sequence; Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence; Inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence that matches the target speech; The digital human driving model is obtained by training using the method described in any one of claims 1 to 5.

7. The method according to claim 6, wherein: Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence, including: Acquire at least one item of requirement description information matching the target speech, wherein the requirement description information includes a requirement time point and a requirement action; generating at least one required action frame for the target virtual image according to the required action; Locating at least one target unknown action frame in the unknown action frame sequence according to the required time point; In the unknown action frame sequence, each target unknown action frame is replaced with the matching required action frame to obtain the local unknown action frame sequence.

8. The method according to claim 6, wherein: Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence, including: When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

9. The method according to claim 6, wherein: Filling at least one required action frame matching the target virtual image into the unknown action frame sequence to obtain a partial unknown action frame sequence, including: If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target avatar; generating the required action frame according to the end action; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

10. The method according to claim 6, wherein: The step of inputting the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, performing at least one action prediction, and obtaining a known action frame sequence matching the target speech comprises: Inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human driving model; Generating an original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer; Generating a target encoding result sequence that matches the original token embedding vector through a model encoder layer; Obtaining, by a model decoder layer, a prediction result vector for at least one target unknown vector in the original tag embedding vector based on a summation result of the target encoding result sequence and the target speech feature frame sequence; After replacing each of the target unknown vectors in the original labeled embedding vector sequence with each of the predicted result vectors, returning to the operation of generating an original labeled embedding vector sequence matching the local unknown action frame sequence through the model input layer until all unknown vectors in the original labeled embedding vector are predicted; The model output layer outputs the known action frame sequence that matches the original label embedding vector that completes the prediction.

11. The method according to claim 10, wherein: The method of obtaining a prediction result vector for at least one target unknown vector in the original tag embedding vector by the model decoder layer according to the sum of the target encoding result sequence and the target speech feature frame sequence includes: Obtaining, through the model decoder layer, a quantity value of unknown vectors contained in the original tag embedding vector, and randomly determining a target quantity based on the quantity value; Randomly selecting target unknown vectors matching the target number from all unknown vectors in the original tag embedding vector sequence through the model decoder layer; The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

12. The method according to any one of claims 6 to 11, further comprising: When the condition for using the target virtual image to broadcast the target voice is met, forming an image frame sequence that matches the target virtual image based on the known action frame sequence; The image frame sequence and the target voice are combined to obtain a target video, and the target video is played.

13. A device for constructing a digital human-driven model, comprising: The original sample acquisition module is used to obtain the original sample; wherein the original sample includes a time-aligned action frame sequence and a speech feature frame sequence; the action frame includes 2D head and limb key points; a masking processing module, configured to mask a set number of action frames in the action frame sequence of the original sample according to a random masking rule to obtain a masked action frame sequence; A training sample acquisition module, configured to add the masking action frame sequence to the original sample to form a training sample; A model training module is used to train the initial motion model using the training samples to obtain a digital human driving model for driving 2D limb movements; The model training module is specifically used to: Inputting the training samples into an initial neural network model; Generating a first labeled embedding vector sequence corresponding to the masked action frame sequence through a model input layer, and generating a second labeled embedding vector sequence corresponding to the action frame sequence; wherein the first marker embedding vector sequence includes an unknown vector corresponding to the masked action frame; Performing multi-level encoding on the first tag embedding vector sequence through a model encoder layer to obtain an encoding result sequence; Performing multi-stage decoding on the sum of the encoding result sequence and the speech feature frame sequence through a model decoder layer to obtain a prediction result vector for each of the unknown vectors; Calculating a model loss value based on a prediction result vector for each unknown vector and a known vector corresponding to each unknown vector in the second label embedding vector sequence; The weight of the initial action model is updated according to the model loss value.

14. The device according to claim 13, wherein The original sample acquisition module is specifically used to: Acquire a live video, and sample the image and voice of the live video at a preset sampling frequency to obtain a time-aligned image frame sequence and a voice frame sequence; Performing 2D key point recognition in each image frame, and forming action frames corresponding to each of the image frames based on the recognition results; Extracting speech features in each speech frame, and forming speech feature frames corresponding to each speech frame according to the extraction results; The original sample corresponding to the live video is formed based on the action frame sequence obtained by organizing the action frames and the voice feature frame sequence obtained by organizing the voice feature frames.

15. The device according to claim 14, wherein In the real-person video, only the upper body area of ​​the real person is displayed.

16. The device according to claim 13, wherein The mask processing module is specifically used to: Randomly select the target probability value within the preset probability interval; Obtaining the total number of action frames contained in the action frame sequence of the original sample; Calculating the masked number of action frames according to the total number of action frames and the target probability value; In the action frame sequence of the original sample, action frames matching the mask quantity are randomly selected for masking to obtain a masked action frame sequence.

17. The device according to claim 13, wherein The model training module is further used to: A diffusion loss function is used to calculate the model loss value between the prediction result vector and the known vector corresponding to each of the unknown vectors.

18. A digital human driving device, comprising: a voice acquisition module, configured to acquire a target voice for voice-driven 2D body movements of a target avatar, wherein the target avatar includes a head and limb regions; A first sequence construction module is used to generate a target speech feature frame sequence matching the target speech, and to construct an unknown action frame sequence matching the target speech feature frame sequence; A second sequence acquisition module is configured to fill at least one required action frame that matches the target avatar into the unknown action frame sequence to obtain a partial unknown action frame sequence; A third sequence acquisition module is configured to input the target speech feature frame sequence and the local unknown action frame sequence into a pre-trained digital human driving model, perform at least one action prediction, and obtain a known action frame sequence that matches the target speech; The digital human driving model is obtained by training using the method described in any one of claims 1 to 5.

19. The device according to claim 18, wherein The second sequence acquisition module is specifically configured to: Acquire at least one item of requirement description information matching the target speech, wherein the requirement description information includes a requirement time point and a requirement action; generating at least one required action frame for the target virtual image according to the required action; Locating at least one target unknown action frame in the unknown action frame sequence according to the required time point; In the unknown action frame sequence, each target unknown action frame is replaced with the matching required action frame to obtain the local unknown action frame sequence.

20. The apparatus according to claim 18, wherein The second sequence acquisition module is specifically configured to: When the target virtual image is in a stationary state, performing 2D key point recognition on the target virtual image, and forming a required action frame according to the recognition result; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

21. The apparatus according to claim 18, wherein The second sequence acquisition module is specifically configured to: If it is determined that the target voice has a previous associated voice, obtaining an ending action of the previous associated voice of the target avatar; generating the required action frame according to the end action; The first action frame in the unknown action frame sequence is replaced with the required action frame to obtain the local unknown action frame sequence.

22. The apparatus according to claim 18, wherein The third sequence acquisition module is specifically configured to: Inputting the target speech feature frame sequence and the local unknown action frame sequence into the digital human-driven model; Generating an original tag embedding vector sequence matching the local unknown action frame sequence through the model input layer; Generating a target encoding result sequence that matches the original token embedding vector through a model encoder layer; Obtaining, by a model decoder layer, a prediction result vector for at least one target unknown vector in the original tag embedding vector based on a summation result of the target encoding result sequence and the target speech feature frame sequence; After replacing each of the target unknown vectors in the original labeled embedding vector sequence with each of the predicted result vectors, returning to the operation of generating an original labeled embedding vector sequence matching the local unknown action frame sequence through the model input layer until all unknown vectors in the original labeled embedding vector are predicted; The model output layer outputs the known action frame sequence that matches the original label embedding vector that completes the prediction.

23. The device according to claim 22, wherein The third sequence acquisition module is further configured to: Obtaining, through the model decoder layer, a quantity value of unknown vectors contained in the original tag embedding vector, and randomly determining a target quantity based on the quantity value; Randomly selecting target unknown vectors matching the target number from all unknown vectors in the original tag embedding vector sequence through the model decoder layer; The model decoder obtains a prediction result vector for each target unknown vector based on the sum of the target encoding result sequence and the target speech feature frame sequence.

24. The apparatus according to any one of claims 18 to 23, further comprising: an image frame sequence acquisition module, configured to generate an image frame sequence matching the target virtual image based on the known action frame sequence when a condition for adopting the target virtual image to broadcast the target voice is met; The target video acquisition module is used to combine the image frame sequence and the target voice to obtain a target video, and play the target video.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause a computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 12.

Images