Method of predicting motion characteristics of a target object, method of training a diffusion model, and device

Abstract

A method for predicting motion features of a target object, a method for training a diffusion model, and an apparatus, the method for predicting motion features of a target object comprising: obtaining a first image, the first image comprising a head portrait of a target object; generating a first position feature of the target object based on the first image, the first position feature comprising first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion features comprising first offsets of the plurality of preset points relative to the first positions; obtaining a first speech audio, obtaining an audio feature sequence based on the first speech audio; determining, by a diffusion model, a first sequence of second motion features based on the first position feature and the audio feature sequence, the second motion features comprising second offsets of the plurality of preset points relative to the first positions.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of machine learning technology, and in particular relate to a method for predicting the motion characteristics of a target object, a method for training a diffusion model, and an apparatus. Background Technology

[0002] One-shot audio-driven talking-head synthesis has been a hot research topic in recent years because it minimizes the need for a target image to a single photograph, generating spoken content without relying on video templates. In early research, methods based on Generative Adversarial Networks (GANs) were the mainstream approach for this type of task. While related research has achieved relatively accurate lip-syncing, it still lacks realism and diversity in terms of speaking style, facial expressions, and head movements.

[0003] Recently, a diffusion model-based approach has significantly improved the vividness and realism of facial expressions and head movements. The introduction of the diffusion model greatly enhances the effect of audio-driven speaking avatars. However, related methods all directly use variational autoencoders (VAEs) pre-trained with general text-based or video-based image generation models to extract image representations. Limited by such a redundant and implicit representation, these methods suffer from slow inference speeds and the generation results cannot be finely controlled based on facial semantics.

[0004] Therefore, current diffusion-based solutions suffer from the following problems. First, they lack control over the generated results, including control over visual detail quality and facial movement. Most current methods can only influence the generated results through audio and other weak conditions, but cannot directly and accurately control dimensions such as facial regions, expression intensity, and head movements semantically. This makes it impossible to effectively intervene when flaws appear in the generated results, and it is difficult to integrate them with other undriven body areas, limiting their application in more diverse scenarios. Furthermore, flaws commonly found in portrait generation, such as imperfections in facial features, teeth, and hair, also limit the performance of these methods. Second, they suffer from slow inference performance. Most current diffusion-based solutions are still far from achieving real-time inference, which is a fundamental requirement for most interactive scenarios. Therefore, even though these methods offer far greater vividness and expressiveness than previous solutions, making them very suitable for scenarios like AI assistants, their inference speed limitations prevent their application in these scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a scheme for predicting the motion characteristics of a target object.

[0006] The first aspect of this specification provides a method for predicting the motion characteristics of a target object, including:

[0007] Acquire a first image, which includes the headshot of the target object;

[0008] Based on the first image, a first position feature of the target object is generated. The first position feature includes a first position of a plurality of preset points of the target object in a preset state and a first motion feature of the plurality of preset points. The first motion feature includes a first offset of the plurality of preset points relative to the first position.

[0009] Obtain the first audio recording, and obtain an audio feature sequence based on the first audio recording;

[0010] Using a diffusion model, a first sequence of second motion features is determined based on the first positional feature and the audio feature sequence. The second motion features include the second offset of the plurality of preset points relative to the first position.

[0011] In one embodiment, the method further includes: generating a speaking video based on a first sequence of the appearance features of the target object, the first position features, and the second motion features.

[0012] In one embodiment, the first motion feature includes offsets corresponding to at least one of the following: facial movements, head rotation, and head translation.

[0013] In one embodiment, the method further includes obtaining a pre-determined correspondence between the dimensions of motion features and facial semantics, and adjusting a first sequence of the second motion features based on the correspondence and the target action to obtain a second sequence.

[0014] The first sequence of generating a speaking video based on the appearance features of the target object, the first position features, and the second motion features includes:

[0015] A speaking video is generated based on the appearance features of the target object, the first location feature, and the second sequence.

[0016] In one implementation, obtaining a first sequence of second motion features based on the first positional features and the audio feature sequence using a diffusion model includes:

[0017] Using a diffusion model, based on the first location feature, the audio feature sequence, preset facial expression labels, and preset eye state sequence, a first sequence of the second motion features is obtained.

[0018] In one embodiment, the first audio segment is obtained by cutting from a second audio segment, and the duration of the first audio segment corresponds to a preset number of video frames.

[0019] The step of obtaining audio feature sequence based on first speech audio includes: inputting the first speech audio and a pre-stored third speech audio into an audio feature extraction model to obtain the audio features of the first speech audio, wherein the third speech audio includes preset timbre and content information.

[0020] In one embodiment, obtaining a first sequence of second motion features based on the first positional feature and the audio feature sequence using a diffusion model includes: outputting a third sequence of third motion features based on the first positional feature and the audio feature sequence using a diffusion model; performing weighted calculations on each of the third motion features to obtain the first sequence, wherein the weight of each third motion feature is determined based on the distance of the third motion feature from the center position of the third sequence.

[0021] In one implementation, generating a first positional feature of the target object based on the first image includes inputting the first image into a motion extractor to obtain the first positional feature.

[0022] In one embodiment, the method further includes: inputting the first image into an appearance feature extractor to obtain the appearance features of the target object.

[0023] In one embodiment, obtaining a first sequence of second motion features based on the first positional feature and the audio feature sequence using a diffusion model includes: obtaining a first sequence of second motion features based on the first positional feature, the audio feature sequence, and a sequence of random noise using a diffusion model.

[0024] The second aspect of this specification provides a method for training a diffusion model, including:

[0025] Acquire a first speaking video, which includes an initial first frame and multiple second frames;

[0026] Based on each frame of the first speaking video, generate a first position of multiple preset points of the target object in a preset state and a first motion feature of each frame, wherein the first motion feature includes a first offset of the multiple preset points relative to the first position;

[0027] Add noise to the sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence;

[0028] An audio feature sequence is obtained based on the audio sequences corresponding to the multiple second frames;

[0029] Using a diffusion model, based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, a sequence of second motion features corresponding to the plurality of second frames is obtained. The second motion feature includes the second offset of the plurality of preset points relative to the first position.

[0030] The parameters of the diffusion model are adjusted based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

[0031] In one embodiment, the method further includes: acquiring a second speaking video for training a diffusion model, wherein each frame of the second speaking video is generated by horizontally flipping each frame of the first speaking video.

[0032] In one implementation, adjusting the parameters of the diffusion model based on the sequences of first motion features and second motion features corresponding to the plurality of second frames includes: grouping the dimensions in each second motion feature, weighting the dimensions of each group to obtain a sequence of third motion features, and adjusting the parameters of the diffusion model based on the sequences of first motion features and the sequences of third motion features.

[0033] In one embodiment, the method further includes: after training the diffusion model, generating a predicted speaking video based on the appearance features of the target object, a first position, a first motion feature of the first frame, and a sequence of second motion features; determining a lip-sync score based on the predicted speaking video and the audio sequence; and evaluating the trained diffusion model based on the lip-sync score.

[0034] In one implementation, the method further includes training the diffusion model based on at least one of the following losses:

[0035] The loss between the first motion feature of the first frame and the first second motion feature of the sequence of the second motion features;

[0036] The loss between the velocity and acceleration corresponding to the sequence of the first motion features of the plurality of second frames and the velocity and acceleration corresponding to the second motion features.

[0037] A third aspect of this specification provides an apparatus for predicting the motion characteristics of a target object, comprising:

[0038] An acquisition unit is used to acquire a first image, wherein the first image includes the headshot of the target object;

[0039] A generation unit is configured to generate a first position feature of the target object based on the first image. The first position feature includes a first position of a plurality of preset points of the target object in a preset state and a first motion feature of the plurality of preset points. The first motion feature includes a first offset of the plurality of preset points relative to the first position.

[0040] The acquisition unit is further configured to acquire a first speech audio and acquire an audio feature sequence based on the first speech audio;

[0041] The determining unit is configured to determine a first sequence of second motion features based on the first positional features and the audio feature sequence using a diffusion model, wherein the second motion features include a second offset of the plurality of preset points relative to the first position.

[0042] A fourth aspect of this specification provides an apparatus for training a diffusion model, comprising:

[0043] An acquisition unit is used to acquire a first speaking video, the first speaking video including an initial first frame and multiple second frames;

[0044] The generation unit is configured to generate, based on each frame of the first speaking video, a first position of multiple preset points of the target object in a preset state and a first motion feature of each frame, wherein the first motion feature includes a first offset of the multiple preset points relative to the first position; and to add noise to the sequence of the first motion features corresponding to the multiple second frames to obtain a noise sequence.

[0045] The acquisition unit is further configured to acquire an audio feature sequence based on the audio sequences corresponding to the plurality of second frames;

[0046] The generation unit is further configured to, through a diffusion model, obtain a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion features of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion features include a second offset of the plurality of preset points relative to the first position;

[0047] The training unit is used to adjust the parameters of the diffusion model based on the sequence of the first motion features corresponding to the plurality of second frames and the sequence of the second motion features.

[0048] The fifth aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in the first or second aspect.

[0049] A sixth aspect of this specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in the first or second aspect.

[0050] The seventh aspect of this specification provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0051] This specification provides a method for predicting the motion features of a target object. By using an explicit, ID-independent motion space instead of a latent vector space in the diffusion model, motion generation and facial rendering are decoupled, reducing the generation space dimension of the diffusion model and improving the prediction speed. Furthermore, by using this diffusion model to predict the motion features of the target object, precise control of facial motion can be achieved based on the predicted motion features. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram illustrating the process of training the diffusion model in the embodiments of this specification;

[0054] Figure 2 This is a flowchart illustrating the method for training the diffusion model in the embodiments of this specification;

[0055] Figure 3 This is a schematic diagram illustrating the process of outputting the second motion feature sequence through a diffusion model in the embodiments of this specification.

[0056] Figure 4 This is a schematic diagram illustrating the process of obtaining a target object's speaking video through prediction using a diffusion model in an embodiment of this specification.

[0057] Figure 5 This is a flowchart of the prediction method of the diffusion model in one embodiment of this specification;

[0058] Figure 6 This is a schematic diagram illustrating the effect of adjusting facial expressions in the embodiments of this specification;

[0059] Figure 7 This is an architectural diagram of a prediction device for predicting the motion characteristics of a target object, as described in an embodiment of this specification.

[0060] Figure 8 This is an architectural diagram of a training device for training a diffusion model, as described in an embodiment of this specification. Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0062] The aforementioned problems with current diffusion model-based solutions mainly stem from the fact that these methods train the diffusion model in a redundant and uninterpretable high-dimensional space. Specifically, most current methods follow the EMO dual U-Net architecture, performing denoising within the latent vector space defined by the pre-trained VAE. This latent vector space mixes motion representations and texture representations, making both the learning of the diffusion model and the rendering by the decoder more difficult.

[0063] Therefore, this specification provides a method for generating speaking videos based on a diffusion model, achieving controllable and real-time generation of speaking avatars. By using an explicit, ID-independent motion space instead of the general latent vector space extracted by VAE, motion generation and facial rendering are decoupled, reducing the dimensionality of the diffusion model's generation space while achieving precise control over facial movements.

[0064] Figure 1 This is a schematic diagram illustrating the process of training the diffusion model in the embodiments of this specification.

[0065] like Figure 1 As shown in the embodiments of this specification, the system architecture includes a pre-trained motion extractor. Audio Feature Extractor and the diffusion model to be trained

[0066] Among them, motion extractor Using ConvNeXt-V2-Tiny as the backbone network, a facial keypoint extractor, a head pose estimation network, and a facial expression change estimation network are combined. The motion extractor... When a single head image I is input, the motion extractor It can output the positional features *c* of K preset keypoints (hereinafter referred to as canonical keypoints) of the main object in the image under preset conditions and the motion features *m* of the main object. The preset conditions are, for example, a state where the head is vertically facing forward with a neutral expression. The preset keypoints include, for example, K = 21 keypoints, such as the corners of the eyes, the tip of the nose, and the corners of the mouth. For example, the positional features *c* may include the three-dimensional coordinates of the K keypoints, thus including K×3-dimensional data. The motion feature m may include {δ, R, t}, where δ corresponds to facial motion, which can cause changes in facial expression. This represents the offset data of the positional features c of K preset keypoints relative to the standard keypoints caused by facial movements. R represents the head pose. This represents the rotation angle of the head. t is the translation value. This represents the translation value of the head, which may include, for example, translation values ​​in the x, y, and z directions.

[0067] Audio Feature Extractor For example, the HUBERT model. HUBERT (Hidden Unit BERT) is a self-supervised learning model primarily used for speech processing tasks. The HUBERT model uses a multi-layer transformer architecture as its encoder and is pre-trained through self-supervised learning, aiming to learn hidden representations of audio in an unsupervised manner, thereby improving the performance of speech recognition tasks.

[0068] In addition, the diffusion model in the embodiments of this specification A diffusion model based on transformer networks can be used, which will be referenced below. Figure 3 Detailed description of the diffusion model The structure.

[0069] The following is combined with Figure 1 and Figure 2 This specification describes the method and process for training a diffusion model according to embodiments of the present specification.

[0070] Figure 2 This is a flowchart illustrating the method for training a diffusion model in an embodiment of this specification. This method can be executed by a server that trains the model.

[0071] like Figure 2 As shown, in step S201, a speaking video is acquired, which includes an initial first frame and multiple second frames.

[0072] refer to Figure 1 As shown, video clips can be obtained. Where [i-1, i+L] indicates that the video segment consists of L+2 frames (images), and seq indicates that the video segment is a sequence of L+2 frames. The video segment includes the initial first frame. And the rest of the multiple second frames The first frame Used as a reference frame, these multiple second frames With audio sequences correspond.

[0073] In one implementation, to enable the diffusion model to generate speaking videos in real-time streaming, during the training phase, multiple video segments with a preset number of frames can be extracted from multiple long videos as training samples. The preset number of frames is, for example, 25 frames.

[0074] In one implementation, because multiple training samples often exhibit an uneven distribution of head orientation on both sides, this imbalance may cause the head orientation and facial expressions corresponding to motion features generated by the diffusion model to be biased to one side. To address this issue, the facial images in the existing training samples can be horizontally flipped (i.e., swapped left and right) to obtain new training samples, and these new training samples can be used together to train the diffusion model, resulting in a more even distribution of head orientation and facial expressions on both sides in the training samples.

[0075] In step S203, based on the spoken video, the first positions of multiple preset points of the target object in a preset state (i.e., standardized key points) and the first motion features of each frame are generated. The first motion features include the offset of the multiple preset points relative to the first position.

[0076] In one implementation, reference Figure 1 By passing the first frame and multiple second frames A randomly determined frame in Input motion extractor Output can be made to the first frame The corresponding first motion feature and random frames Location features of corresponding standard key points This location feature This includes the first position of each key point in the specification. Position features are obtained from random frames. This can increase the robustness of the model's predictions. Additionally, by using multiple second frames... Input into motion extractor respectively It can output a sequence of first motion features corresponding to multiple second frames.

[0077] In step S205, noise is added to the sequence of first motion features of multiple second frames to obtain a noise sequence.

[0078] refer to Figure 1 As shown, the sequence of the first motion features can be... Each motion feature m in j Multiple noise additions are performed; specifically, m can be used. j The corresponding {δ j ,R j ,t j Each data point in} (e.g., δ) j ,R j or t j Noise is added multiple times to each dimension of the sequence to obtain a noise sequence.

[0079] In step S207, an audio feature sequence is obtained based on the audio sequences corresponding to multiple second frames.

[0080] In one implementation, reference Figure 1 As shown, by using audio sequences Input audio feature extractor Output audio feature sequence

[0081] In step S209, using a diffusion model, a sequence of second motion features corresponding to multiple second frames is determined based on the first position, the first motion features of the first frame, the noise sequence, and the audio feature sequence.

[0082] refer to Figure 1 The first position of the multiple key points of the specifications obtained above The first motion feature of the first frame noise sequence and audio feature sequences Input a diffusion model to output a sequence of predicted second motion features corresponding to multiple second frames.

[0083] In one implementation, such as Figure 1 As shown, other conditions can also be input into the diffusion model.

[0084] Figure 3 This is a schematic diagram illustrating the process of outputting the second motion feature sequence through a diffusion model in the embodiments of this specification.

[0085] like Figure 3As shown, the diffusion model may include an upper neural network module for inputting the initial conditional signal (ICS) and a lower neural network module for inputting the enhanced conditional signal (ECS).

[0086] The ICS signal includes, for example, a reference motion feature sequence, which is obtained by stitching together the motion features m of the target head image at a preset time step. ref (as mentioned above) This time step is obtained by matching the time step of the audio feature sequence. This reference motion feature sequence serves as a guide for motion generation in each video clip, thereby enhancing the continuity between video clips and periodically guiding motion to a specified target state, thus reducing error accumulation in long video sequences (or long sequences of motion features).

[0087] ECS signals include, for example, a reference position sequence and an audio feature sequence a (as mentioned above). ), and as other conditions, expression tag sequences and eye state sequences. The reference position sequence includes the positions of canonical key points of the target avatar, copied by splicing at a preset time step. ref (for example, the above) The expression label sequence is obtained by inputting the target head image into a pre-trained expression classifier to obtain expression labels s, and then concatenating the copied expression labels s at a preset time step to obtain the expression label sequence. It is understood that, in the embodiments of this specification, for real-time streaming output, the number of motion features (corresponding to a video segment) in the motion feature sequence output by the diffusion model is preset to be within a preset range (e.g., 25). Therefore, consistent expression labels s can be set for the motion feature sequence. In one embodiment, multiple expression label sequences can also be manually set for the motion feature sequence. The eye state sequence includes pre-set eye states corresponding to each motion feature in the motion feature sequence, and each eye state includes identifiers indicating whether the eyes are open or closed and pupil position information, etc. The diffusion model outputs a predicted sequence of motion features corresponding to the target image based on the input ECS and ICS. (for example, the above) ).

[0088] The aforementioned upper neural network module, starting from the input on the left, may include a multilayer perceptron (MLP) layer and multiple ( Figure 3The diagram shows two encoder layers, each comprising a self-attention layer (SelfAtt) and an MLP layer. The lower neural network module, starting from the input on the left, may include an MLP layer as the input layer, multiple (eight shown in the diagram) encoder layers, and an MLP layer as the output layer. The encoder layers may sequentially include a SelfAtt layer, an element-wise linear modulation layer (FiLM), a cross-attention layer (CrossAtt), a FiLM layer, an MLP layer, and another FiLM layer. The output of the upper neural network module is connected to the CrossAtt layer in the lower neural network module, allowing the CrossAtt to fuse features corresponding to ECS and ICS.

[0089] Back Figure 2 In step S211, the parameters of the diffusion model are adjusted based on the sequence of first motion features and the sequence of second motion features of multiple second frames.

[0090] refer to Figure 1 The loss function can be expressed as shown in formula (1) below. Training the diffusion model:

[0091]

[0092] in, For mean squared error loss, C represents the conditional signal, such as ICS, ECS, etc. mentioned above, m t This is a noise sequence obtained by adding noise t times to the actual motion feature sequence m0. In the embodiments of this specification, m0 is, for example, the sequence of the first motion feature described above. m t Noise sequence The sequence of the second motion feature mentioned above

[0093]

[0094] In one implementation, as described above, the motion feature m may include {δ, R, t}, where δ represents facial movement, R represents head rotation angle, and t represents head translation. Since different people will exhibit different facial and head movements in response to the same audio, dynamically adjusted weights can be added to different dimensions of the motion feature during training to make the trained model more realistic. Specifically, δ, R, t in the motion feature m can be divided into two groups: δ and R, t, with weights dynamically adjusted for each group. For example, in multiple training iterations of the diffusion model, in one training phase, weight w1 can be set for δ and weight w2 for R and t in the motion feature m, participating in the loss function calculation. In the next training phase, weight w3 can be set for δ and weight w4 for R and t, participating in the loss function calculation. Experiments have shown that this strategy of dynamically adjusting the weights of each group in the motion feature accelerates model convergence and improves the quality of the diffusion model's prediction results.

[0095] By adjusting the parameters of the diffusion model, the loss function is made more efficient. Minimizing the value of makes the sequence of the second motion features output by the diffusion model more efficient. A sequence that more closely resembles the first motion features of an actual video This makes the output of the diffusion model closer to the real motion representation.

[0096] In one implementation, such as Figure 1 As shown, it can also be based on the initial motion loss. Training the diffusion model, where the initial motion loss For reference motion characteristics (e.g., the above) The sequence of the second motion features output by the diffusion model.

[0097] The first one in The losses between them.

[0098] In one implementation, to enhance time stability, the loss of velocity and acceleration corresponding to the motion characteristics can also be calculated as shown in formula (2).

[0099]

[0100] in, This represents the predicted motion feature sequence output by the diffusion model (e.g.) The first derivative of m) ′ Represents the actual motion feature sequence (e.g.) The first derivative of the motion feature sequence can be calculated by dividing the distance difference between each motion feature by time. denoted by , m″ represents the second derivative of the predicted motion feature sequence output by the diffusion model, and m″ represents the second derivative of the actual motion feature sequence. This second derivative can be calculated by dividing the velocity difference corresponding to each motion feature in the motion feature sequence by time.

[0101] Combining the above loss terms, the total loss can be obtained as shown in formula (3).

[0102]

[0103] After training the diffusion model through the above process, the model performance can be evaluated based on the existing lip-sync model. Specifically, validation samples are obtained, which include the head image of the target object and the target audio sequence. Using the system of this embodiment, a predicted speaking video corresponding to the head image is output based on the head image and the target audio sequence. This predicted speaking video and the target audio are input into the lip-sync model, which outputs a lip-sync score. The performance of the diffusion model can then be evaluated based on this lip-sync score; a higher lip-sync score indicates more accurate predictions. The validation samples can also serve as training samples for training the diffusion model. Specifically, after training the diffusion model, a predicted speaking video is generated based on the appearance features of the target object, the first position corresponding to the training sample, the first motion feature, and the sequence of the second motion feature. The lip-sync score is determined based on the predicted speaking video and the audio sequence corresponding to the training sample, and the trained diffusion model is evaluated based on the lip-sync score.

[0104] In one implementation, it can be achieved through... Figure 1 and Figure 2 The training method shown trains multiple diffusion models, and the evaluation method is used to evaluate these models. For example, the diffusion model with the highest lip-sync score can be selected as the model for practical use. This method solves a common problem in the training of diffusion models, namely, that the loss function cannot accurately reflect the model's performance.

[0105] After training the diffusion model, the trained model can be used to predict a sequence of motion features of the target object based on a single portrait and audio of the target object.

[0106] Figure 4 This is a schematic diagram illustrating the process of obtaining the target object's speaking video through prediction using a diffusion model in the embodiments of this specification.

[0107] like Figure 4 As shown, the system architecture in this scenario includes, in addition to Figure 1 Motion extractor in Audio Feature Extractor and the trained diffusion model In addition, it includes a pre-trained appearance feature extractor. and renderer

[0108] Among them, appearance feature extractor Used to map an input avatar image s to 3D appearance features f s Renderer Appearance features f used based on reference frames (i.e., single portraits) ref The position of the key point in the reference frame (x) ref Location of key points in the prediction frame generated by the diffusion model Composite Frame

[0109] The following is for reference. Figure 5 Describe in detail the prediction process of the diffusion model. Figure 5 The method shown can be executed by a server used to generate speaking videos based on a single headshot image and audio.

[0110] like Figure 5 As shown, in step S501, the head image of the target object is obtained.

[0111] The server can receive a video generation request from the user device, which may include a head image I for generating a speaking video. ref And voice audio.

[0112] In step S503, the first location feature of the target object is generated based on the headshot image.

[0113] refer to Figure 4 By using the avatar image I ref Input motion extractor The first positional feature is obtained, which includes the positions c of multiple canonical keypoints. ref (e.g., 3D coordinates) and head image I ref The first motion characteristic m ref .

[0114] In step S505, a first speech audio is obtained, and an audio feature sequence is obtained based on the first speech audio.

[0115] In one implementation, the first voice audio audio1 can be the voice audio audio received from the user equipment as described above.

[0116] In another implementation, when the received audio duration is relatively long, in order to... Figure 4The system shown performs streaming output and can divide the audio into multiple sequentially arranged audio chips according to a preset time step. Each audio chip includes a preset number of audio frames, and the duration of each audio frame is the same as the duration of a video frame. Currently, the first audio chip can be acquired as the first audio.

[0117] Input the first audio message into the audio feature extractor. The audio feature sequence a of the first speech audio is obtained. seq In scenarios where the first audio segment is short, embodiments of this specification can concatenate pre-stored, designed speech samples with the first audio segment to provide pseudo-contextual information. This concatenation method involves appending the speech samples before the first audio segment. These speech samples may include preset timbre and content information, and the pseudo-contextual information is extracted from a fixed corpus. This method allows models pre-trained on longer audio segments, such as HUBERT, to still output high-quality audio feature sequences when given a shorter first audio segment as input. This optimization allows the diffusion model to process shorter audio stream units (e.g., 0.4 seconds) in real time. It is understood that this optimization method can also be used in the training process of the diffusion model. Furthermore, in this optimization method, the audio feature extractor can be... The structure is changed so that for the input concatenated audio, for the speech sampling part, by using a mask in the attention layer, attention is only calculated for the speech sampling part and its subsequent audio features. That is, only forward attention is calculated for the speech sampling part to reduce the amount of computation brought by the speech sampling part.

[0118] In step S507, based on the first positional feature and audio feature sequence a, using the diffusion model... seq The sequence of second motion features is determined by identifying multiple preset key points.

[0119] By using the first position feature (i.e., c) ref and m ref ), audio feature sequence a seq The random noise is input into the diffusion model, causing the diffusion model to output a sequence of predicted second motion features. The model structure of the diffusion model can be found in [reference]. Figure 3 As shown, this will not be repeated here. In one implementation, the diffusion model can automatically generate random noise.

[0120] In one implementation, reference Figure 3 and Figure 4 As shown, besides the first position feature (i.e., c) ref and m ref ), audio feature sequence a seqIn addition to inputting random noise into the diffusion model, other conditions can also be input into the diffusion model, as shown in the reference. Figure 3 The facial expression labels s and eye states e described in the text are used to make the diffusion model output more accurate prediction results.

[0121] Optionally, after training the second motion features predicted by the diffusion model output, in step S509, a speaking video is generated based on the sequence of the target object's appearance features, first position features, and second motion features.

[0122] refer to Figure 4 By obtaining the appearance features f of the target object as described above ref First positional feature (i.e., c) ref and m ref ), and the sequence of second motion features Input Renderer Make the renderer Can output spoken video

[0123] In one implementation, assume m ref ={δ ref ,R ref ,t ref Motion features of each predicted frame As shown in formulas (4) and (5), the position x of the reference key point is generated based on the first position feature. ref and the location of key points in each predicted frame

[0124] x ref =c ref R ref +δ ref +t ref (4)

[0125]

[0126] Then f can be ref x ref and various Input Renderer Obtain each frame of the predicted video

[0127] In one implementation, as described in this specification, a direct and explicit mapping relationship between the dimensions of motion feature m and facial semantics is pre-established using empirical values, thereby allowing adjustment... The specific dimensions within the renderer enable precise semantic control over the driving region, facial movements, head actions, etc., thereby allowing the renderer to... Output frames that conform to this semantic control

[0128] For example, by adjusting In Taking adjusting facial motion in a rendering frame as an example, as described above, It is a 63-dimensional vector, which includes 21 preset key three-dimensional coordinates. Each dimension affects different facial areas; for example, the 34th dimension controls the closing movement of the right eye, and the 58th dimension controls the opening of the mouth. For Each dimension can be adjusted by applying a positive or negative offset. Therefore, it can be based on Two types of control are implemented: (1) Facial area control, by selecting (1) Specific dimensions, which can limit motion generation to specific local areas of the face; (2) Size control, through adjustment The size of a specific dimension can be adjusted to regulate the amplitude of facial movements, for example, by adjusting... The size of a specific dimension is used to avoid predicting unnatural facial movements in videos.

[0129] Figure 6 The embodiments in this specification are achieved by adjusting The illustration shows the effect of adjusting facial expressions in terms of dimensions. Figure 6 The central portrait among the nine portraits represents the portrait without any offset applied to any dimension. The top-left corner marker in each portrait indicates the keypoint and its coordinates. For example, in the top-left portrait, "15-0" indicates the 15th keypoint and "0" represents the x-axis. This means the portrait is obtained by adjusting the x-axis value of the 15th keypoint based on the motion feature m of the central portrait. According to the mapping relationship between dimensions and facial semantics in the preset motion feature m, the x-axis value of the 15th keypoint corresponds to eye rotation. Therefore, by adjusting the x-axis value of the 15th keypoint, the rendered portrait's eyes turn to the left. It can be understood that this can be achieved by simultaneously adjusting... The values ​​of multiple dimensions in the expression can be used to create complex facial expressions.

[0130] In one implementation, in order to stream the predicted video, the first audio segment in the long audio is processed as described above to output the predicted video. After that, you can The last frame is used as reference image I ref For the second audio segment in a long audio file, perform the following... Figure 4 and Figure 5 The processing shown yields the predicted video corresponding to the second audio segment. This process can be repeated to quickly output a sequence of predicted videos corresponding to a long video. This streaming processing method achieves low first-frame latency, improving the user experience.

[0131] To further improve the streaming output effect described above, in the embodiments of this specification, it is possible to improve the performance of each motion feature sequence. adjacent motion feature sequences and An overlap of a predetermined length is introduced between each predicted video, thus introducing an overlap of a predetermined duration between the predicted videos. Additionally, for motion feature sequences... Based on each motion feature in the motion feature sequence The distance (relative position) from the motion feature located at the center determines the individual motion features. The weights, where the motion features closer to the center position are... The larger the weight, the better. This is how motion feature sequences are set. Various motion characteristics The weights of the generated predicted video sequence are such that... It is smoother and more natural when displayed in streaming mode.

[0132] In addition, in order to provide real-time display performance, the number of denoising steps when acquiring random noise can be reduced in the embodiments of this specification. Thanks to the space of motion features provided by the diffusion model in the embodiments of this specification, video quality comparable to that in related technologies can still be obtained even with a reduction in the number of denoising steps.

[0133] Figure 7 This is an architectural diagram of a prediction device for predicting the motion characteristics of a target object, as described in an embodiment of this specification, comprising:

[0134] Acquisition unit 71 is used to acquire a first image, the first image including the headshot of the target object;

[0135] The generation unit 72 is configured to generate a first position feature of the target object based on the first image. The first position feature includes a first position of a plurality of preset points of the target object in a preset state and a first motion feature of the plurality of preset points. The first motion feature includes a first offset of the plurality of preset points relative to the first position.

[0136] The acquisition unit 71 is further configured to acquire a first speech audio and acquire an audio feature sequence based on the first speech audio.

[0137] The determining unit 73 is configured to determine a first sequence of second motion features based on the first position feature and the audio feature sequence using a diffusion model, wherein the second motion feature includes a second offset of the plurality of preset points relative to the first position.

[0138] Figure 8 This is an architectural diagram of a training device for training a diffusion model, as described in an embodiment of this specification, comprising:

[0139] Acquisition unit 81 is used to acquire a first speaking video, the first speaking video including an initial first frame and multiple second frames;

[0140] The generation unit 82 is configured to generate, based on each frame of the first speaking video, a first position of multiple preset points of the target object in a preset state and a first motion feature of each frame, wherein the first motion feature includes a first offset of the multiple preset points relative to the first position; and to add noise to the sequence of the first motion features corresponding to the multiple second frames to obtain a noise sequence.

[0141] The acquisition unit 81 is further configured to acquire an audio feature sequence based on the audio sequences corresponding to the plurality of second frames;

[0142] The determining unit 83 is configured to determine, through a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion features of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion features include a second offset of the plurality of preset points relative to the first position;

[0143] Training unit 84 is used to adjust the parameters of the diffusion model based on the sequence of the first motion features corresponding to the plurality of second frames and the sequence of the second motion features.

[0144] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0145] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0146] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0147] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0148] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0149] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0154] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0155] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0157] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0158] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for predicting the motion characteristics of a target object, comprising: Acquire a first image, which includes the headshot of the target object; Based on the first image, a first position feature of the target object is generated. The first position feature includes a first position of a plurality of preset points of the target object in a preset state and a first motion feature of the plurality of preset points. The first motion feature includes a first offset of the plurality of preset points relative to the first position. Obtain the first audio recording, and obtain an audio feature sequence based on the first audio recording; Using a diffusion model, a first sequence of second motion features is determined based on the first positional feature and the audio feature sequence. The second motion features include the second offset of the plurality of preset points relative to the first position.

2. The method according to claim 1, further comprising: A speaking video is generated based on a first sequence of the appearance features of the target object, the first position features, and the second motion features.

3. The method according to claim 1, wherein the first motion feature includes an offset corresponding to at least one of the following: facial movement, head rotation, and head translation.

4. The method according to claim 2 further includes: obtaining a predetermined correspondence between the dimensions of motion features and facial semantics; adjusting a first sequence of the second motion features based on the correspondence and the target action to obtain a second sequence. The first sequence of generating a speaking video based on the appearance features of the target object, the first position features, and the second motion features includes: A speaking video is generated based on the appearance features of the target object, the first location feature, and the second sequence.

5. The method according to claim 1, wherein determining the first sequence of the second motion features based on the first positional features and the audio feature sequence using a diffusion model comprises: Using a diffusion model, based on the first location feature, the audio feature sequence, preset facial expression labels, and preset eye state sequence, a first sequence of the second motion features is determined.

6. The method according to claim 1, wherein the first audio is obtained by segmenting from the second audio, and the duration of the first audio corresponds to a preset number of video frames. The process of obtaining the audio feature sequence based on the first speech audio includes: The third audio file is concatenated with the first audio file and then input into the audio feature extraction model to obtain the audio features of the first audio file. The third audio file includes preset timbre and content information.

7. The method according to claim 6, wherein the third audio segment is concatenated before the first audio segment, and the step of inputting the pre-stored third audio segment and the first audio segment into an audio feature extraction model to obtain the audio features of the first audio segment includes: The audio feature extraction model is configured to perform forward attention calculation on the third speech audio.

8. The method according to claim 6, wherein determining the first sequence of the second motion features based on the first positional features and the audio feature sequence using a diffusion model comprises: Using a diffusion model, based on the first positional feature and the audio feature sequence, a third sequence of third motion features is output. The weighted calculation of each third motion feature is performed to obtain the first sequence, wherein the weight of each third motion feature is determined based on the distance of the third motion feature from the center position of the third sequence.

9. The method according to claim 1, wherein generating the first position feature of the target object based on the first image comprises inputting the first image into a motion extractor to obtain the first position feature.

10. The method according to claim 2, further comprising: The first image is input into the appearance feature extractor to obtain the appearance features of the target object.

11. The method according to claim 1, wherein determining the first sequence of the second motion features based on the first positional features and the audio feature sequence using a diffusion model comprises: Using a diffusion model, a first sequence of second motion features is determined based on the first positional feature, the audio feature sequence, and the sequence of random noise.

12. A method for training a diffusion model, comprising: Acquire the first speaking video of the target object, the first speaking video including an initial first frame and multiple second frames; Based on the first speaking video, the first position of multiple preset points of the target object in a preset state and the first motion features of each frame are generated. The first motion features include the first offset of the multiple preset points relative to the first position. Add noise to the sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence; An audio feature sequence is obtained based on the audio sequences corresponding to the multiple second frames; Using a diffusion model, based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, a sequence of second motion features corresponding to the plurality of second frames is determined, wherein the second motion feature includes the second offset of the plurality of preset points relative to the first position; The parameters of the diffusion model are adjusted based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

13. The method of claim 12, further comprising: A second speaking video is obtained for training the diffusion model, wherein each frame of the second speaking video is generated by horizontally flipping each frame of the first speaking video.

14. The method according to claim 12, wherein adjusting the parameters of the diffusion model based on the sequence of the first motion features corresponding to the plurality of second frames and the sequence of the second motion features includes: The dimensions of each second motion feature are grouped, weights are set for each group, and the dimensions in each group are weighted based on the weights of each group to obtain a sequence of third motion features. The parameters of the diffusion model are adjusted based on the sequence of the first motion features and the sequence of the third motion features.

15. The method of claim 12, further comprising: After training the diffusion model, a predicted speaking video is generated based on the appearance features of the target object, the first position, the first motion features of the first frame, and the sequence of the second motion features. A lip-sync score is determined based on the predicted speaking video and the audio sequence, and the diffusion model is evaluated based on the lip-sync score.

16. The method of claim 12, further comprising training the diffusion model based on at least one of the following losses: The loss between the first motion feature of the first frame and the first second motion feature of the sequence of the second motion features; The loss between the velocity and acceleration corresponding to the sequence of the first motion features of the plurality of second frames and the velocity and acceleration corresponding to the second motion features.

17. A device for predicting the motion characteristics of a target object, comprising: An acquisition unit is used to acquire a first image, wherein the first image includes the headshot of the target object; A generation unit is configured to generate a first position feature of the target object based on the first image. The first position feature includes a first position of a plurality of preset points of the target object in a preset state and a first motion feature of the plurality of preset points. The first motion feature includes a first offset of the plurality of preset points relative to the first position. The acquisition unit is further configured to acquire a first speech audio and acquire an audio feature sequence based on the first speech audio; The determining unit is configured to determine a first sequence of second motion features based on the first positional features and the audio feature sequence using a diffusion model, wherein the second motion features include a second offset of the plurality of preset points relative to the first position.

18. An apparatus for training a diffusion model, comprising: The acquisition unit is used to acquire a first speaking video of the target object, the first speaking video including an initial first frame and multiple second frames; The generation unit is configured to generate, based on each frame of the first speaking video, a first position of a plurality of preset points of the target object in a preset state and a first motion feature of each frame, wherein the first motion feature includes a first offset of the plurality of preset points relative to the first position; Add noise to the sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence; The acquisition unit is further configured to acquire an audio feature sequence based on the audio sequences corresponding to the plurality of second frames; The determining unit is configured to determine, through a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion features of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion features include a second offset of the plurality of preset points relative to the first position; The training unit is used to adjust the parameters of the diffusion model based on the sequence of the first motion features corresponding to the plurality of second frames and the sequence of the second motion features.

19. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-16.

Images