Method for predicting motion features of target object, and method and device for training diffusion model

Abstract

A method for predicting a motion feature of a target object, a method for training a diffusion model and a device, the method for predicting the motion feature of the target object comprising: acquiring a first image, the first image comprising a head portrait of the target object; generating a first position feature of the target object based on the first image, the first position feature including first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion feature comprises a first offset of the plurality of preset points relative to the first position; obtaining a first voice audio, and obtaining an audio feature sequence based on the first voice audio; and determining a first sequence of second motion features based on the first position features and the audio feature sequence through a diffusion model, the second motion features including second offsets of the plurality of preset points relative to the first position.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of machine learning technology, and more particularly, to a method for predicting motion characteristics of a target object, a method and device for training a diffusion model. Background Art

[0002] One-shot audio-driven talking-head synthesis has been a hot topic in recent years because it can minimize the need for a target image to a single photo and generate spoken content without relying on a video template. In early studies, methods based on generative adversarial networks (GAN) were the mainstream solution for this type of task. Related research has achieved relatively accurate lip synchronization, but it still lacks authenticity and diversity in the expression of speaking style, expression, and head movements.

[0003] Recently, the vividness and realism of facial expressions and head movements have been significantly improved through solutions based on diffusion models. The introduction of diffusion models has greatly improved the effect of audio-driven talking avatars. However, related methods directly use variational autoencoders (VAEs) pre-trained with general vinculum or video generation models to extract image representations. Limited by such a redundant and implicit representation, such methods have slow reasoning speed and the generated results cannot be finely controlled according to facial semantics.

[0004] Therefore, the current diffusion model-based solutions have the following problems. The first point is the lack of control over the generated results, including the control of the quality of visual details and the control of facial movements. At present, most methods can only affect the generated results through audio and other weak conditions, but cannot directly and accurately control the semantics of dimensions such as facial area, expression intensity, and head movement. On the one hand, it is impossible to effectively intervene when there are defects in the generated results, and on the other hand, it is difficult to splice with other undriven body areas, which limits the application in more diverse scenarios. In addition, the defects such as facial features, teeth, and hair that often appear in the portrait generation of this solution also limit the performance of this type of method. The second point is the slow reasoning performance. Most of the current diffusion model-based solutions are still far from real-time reasoning, which is the basic requirement of most interactive scenarios. Therefore, even if this type of method provides far more vividness and expressiveness than previous solutions, and is very suitable for scenarios such as AI assistants, it cannot be applied in such scenarios due to the limitation of reasoning speed. Summary of the invention

[0005] The object of the present invention is to provide a solution for predicting motion characteristics of a target object.

[0006] A first aspect of the present specification provides a method for predicting motion characteristics of a target object, comprising:

[0007] Acquire a first image, wherein the first image includes a head portrait of a target object;

[0008] generating a first position feature of the target object based on the first image, the first position feature comprising first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion feature comprising first offsets of the plurality of preset points relative to the first position;

[0009] Acquire a first voice audio, and acquire an audio feature sequence based on the first voice audio;

[0010] A first sequence of second motion features is determined based on the first position feature and the audio feature sequence through a diffusion model, wherein the second motion feature includes a second offset of the plurality of preset points relative to the first position.

[0011] In one embodiment, the method further includes: generating a speaking video based on a first sequence of the appearance feature of the target object, the first position feature, and the second motion feature.

[0012] In one embodiment, the first motion characteristic comprises an offset corresponding to at least one of: facial movement, head rotation, and head translation.

[0013] In one embodiment, the method further includes obtaining a correspondence between a dimension of a predetermined motion feature and facial semantics, adjusting a first sequence of the second motion feature based on the correspondence and the target action to obtain a second sequence,

[0014] The generating of a speaking video based on a first sequence of the appearance feature of the target object, the first position feature, and the second motion feature comprises:

[0015] A speaking video is generated based on the appearance feature of the target object, the first position feature and the second sequence.

[0016] In one implementation, obtaining a first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model includes:

[0017] Through the diffusion model, based on the first position feature, the audio feature sequence, the preset expression label, and the preset eye state sequence, a first sequence of the second motion features is obtained.

[0018] In one embodiment, the first voice audio is obtained by cutting from the second voice audio, and the duration of the first voice audio corresponds to a preset number of video frames.

[0019] The method of obtaining an audio feature sequence based on the first speech audio includes: inputting the first speech audio and a pre-stored third speech audio into an audio feature extraction model to obtain audio features of the first speech audio, wherein the third speech audio includes preset timbre and content information.

[0020] In one embodiment, the method of obtaining a first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model includes: outputting a third sequence of third motion features based on the first position feature and the audio feature sequence through a diffusion model, performing weighted calculation on each of the third motion features to obtain the first sequence, wherein the weight of each third motion feature is determined based on the distance of the third motion feature from the center position of the third sequence.

[0021] In one implementation, generating the first position feature of the target object based on the first image includes inputting the first image into a motion extractor to obtain the first position feature.

[0022] In one embodiment, the method further includes: inputting the first image into an appearance feature extractor to obtain appearance features of the target object.

[0023] In one embodiment, obtaining the first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model includes: obtaining the first sequence of second motion features based on the first position feature, the audio feature sequence and a sequence of random noise through a diffusion model.

[0024] A second aspect of the present specification provides a method for training a diffusion model, comprising:

[0025] Acquire a first speaking video, where the first speaking video includes an initial first frame and a plurality of second frames;

[0026] generating, based on each frame of the first speech video, first positions of a plurality of preset points of the target object in a preset state and first motion features of each frame, wherein the first motion features include first offsets of the plurality of preset points relative to the first positions;

[0027] Adding noise to the sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence;

[0028] Acquire an audio feature sequence based on the audio sequences corresponding to the multiple second frames;

[0029] Obtaining, by means of a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion feature includes second offsets of the plurality of preset points relative to the first position;

[0030] The parameters of the diffusion model are adjusted based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

[0031] In one embodiment, the method further includes: acquiring a second speaking video for training a diffusion model, wherein each frame in the second speaking video is generated by horizontally flipping each frame in the first speaking video.

[0032] In one embodiment, adjusting the parameters of the diffusion model based on the sequence of first motion features and the sequence of second motion features corresponding to the multiple second frames includes: grouping the dimensions in each second motion feature, weighting the dimensions of each group to obtain a sequence of third motion features, and adjusting the parameters of the diffusion model based on the sequence of the first motion features and the sequence of the third motion features.

[0033] In one embodiment, the method also includes: after completing the training of the diffusion model, generating a predicted speaking video based on the appearance features of the target object, the first position, the first motion features of the first frame, and the sequence of the second motion features, determining a lip synchronization score based on the predicted speaking video and the audio sequence, and evaluating the trained diffusion model based on the lip synchronization score.

[0034] In one embodiment, the method further comprises training the diffusion model based on at least one of the following losses:

[0035] a loss between a first motion feature of the first frame and a first second motion feature of a sequence of second motion features;

[0036] The loss between the speed and acceleration corresponding to the sequence of the first motion features corresponding to the plurality of second frames and the speed and acceleration corresponding to the second motion features.

[0037] A third aspect of the present specification provides a device for predicting motion characteristics of a target object, comprising:

[0038] An acquisition unit, configured to acquire a first image, wherein the first image includes a head portrait of a target object;

[0039] a generating unit, configured to generate a first position feature of the target object based on the first image, the first position feature comprising first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion feature comprising first offsets of the plurality of preset points relative to the first position;

[0040] The acquisition unit is further used to acquire a first speech audio, and acquire an audio feature sequence based on the first speech audio;

[0041] A determination unit is used to determine a first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model, wherein the second motion feature includes a second offset of the plurality of preset points relative to the first position.

[0042] A fourth aspect of the present specification provides a device for training a diffusion model, comprising:

[0043] An acquisition unit, configured to acquire a first speaking video, wherein the first speaking video includes an initial first frame and a plurality of second frames;

[0044] a generating unit, configured to generate first positions of a plurality of preset points of the target object in a preset state and first motion features of each frame based on each frame of the first speaking video, wherein the first motion features include first offsets of the plurality of preset points relative to the first positions; and to add noise to a sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence;

[0045] The acquisition unit is further configured to acquire an audio feature sequence based on the audio sequences corresponding to the plurality of second frames;

[0046] The generating unit is further configured to obtain, through a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion feature includes second offsets of the plurality of preset points relative to the first position;

[0047] A training unit is used to adjust the parameters of the diffusion model based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

[0048] A fifth aspect of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect or the second aspect.

[0049] A sixth aspect of the specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0050] A seventh aspect of the present specification provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect.

[0051] The embodiments of the present specification provide a method for predicting the motion features of a target object, wherein motion generation and facial rendering are decoupled by using an explicit, ID-independent motion space instead of a latent vector space in a diffusion model, thereby reducing the dimension of the generation space of the diffusion model and improving the speed of model prediction. In addition, by using the diffusion model to predict the motion features of the target object, precise control of facial motion can be achieved based on the predicted motion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 This is a schematic diagram of the process of training a diffusion model in an embodiment of this specification;

[0054] Figure 2 This is a flow chart of a method for training a diffusion model in an embodiment of this specification;

[0055] Figure 3 This is a schematic diagram of a process of outputting a second motion feature sequence through a diffusion model in an embodiment of this specification;

[0056] Figure 4 This is a schematic diagram of the process of obtaining a target object's speaking video by predicting through a diffusion model in an embodiment of this specification;

[0057] Figure 5 is a flow chart of a prediction method of a diffusion model in an embodiment of this specification;

[0058] Figure 6 This is a schematic diagram of the effect of adjusting facial expressions in the embodiments of this specification;

[0059] Figure 7 is a structural diagram of a prediction device for predicting motion characteristics of a target object in an embodiment of this specification;

[0060] Figure 8 Schematic diagram of a training device for training a diffusion model in an embodiment of this specification. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0062] The above problems with current diffusion model-based solutions are mainly because these methods train diffusion models in a redundant, uninterpretable high-dimensional space. Specifically, most current methods follow the EMO dual U-Net architecture and perform denoising in the latent vector space defined by the pre-trained VAE. This latent vector space mixes the representation of motion and texture, making the learning of the diffusion model and the rendering of the decoder more difficult.

[0063] To this end, the embodiments of this specification provide a method for generating a speaking video based on a diffusion model to achieve controllable and real-time speaking avatar generation. By using an explicit, ID-independent motion space instead of a general VAE-extracted latent vector space, motion generation and facial rendering are decoupled, the generation space dimension of the diffusion model is reduced, and precise control of facial motion is achieved.

[0064] Figure 1 Schematic diagram of the process of training a diffusion model in an embodiment of this specification.

[0065] like Figure 1 As shown in FIG. 1 , in the system architecture of the embodiment of the present specification, a pre-trained motion extractor is included. Audio Feature Extractor and the diffusion model to be trained

[0066] Among them, the motion extractor ConvNeXt-V2-Tiny is used as the backbone network, which combines the facial key point extractor, head pose estimation network and expression variation estimation network. When a single head image I is input, the motion extractor The position features c of K preset key points (hereinafter referred to as standard key points) of the subject object in the image in a preset state and the motion features m of the subject object can be output. The preset state is, for example, a state where the head faces forward vertically and the expression is neutral. The multiple preset key points include, for example, K=21 key points, such as the corners of the eyes, the tip of the nose, the corners of the mouth, etc. For example, the position feature c may include the three-dimensional coordinates of the K key points, thus including K×3 dimensional data in total, i.e. The motion feature m may include {δ, R, t}, where δ corresponds to facial motion, which may cause facial expression changes. represents the offset data of the position features c of the K preset key points relative to the standard key points caused by facial movement. R is the head posture, Indicates the rotation angle of the head. t is the translation value, Indicates the translation value of the head, which includes, for example, translation values ​​in three directions: x, y, and z.

[0067] Audio Feature Extractor For example, the HUBERT model. HUBERT (Hidden Unit BERT) is a self-supervised learning model mainly used for speech processing tasks. The HUBERT model uses a multi-layer transformer architecture as its encoder and is pre-trained through self-supervised learning. It aims to learn the hidden representation of audio in an unsupervised way, thereby improving the performance of speech recognition tasks.

[0068] In addition, in the embodiments of this specification, the diffusion model A diffusion model based on a transformer network can be used, which will be referred to below Figure 3 Describe the diffusion model in detail structure.

[0069] Combine the following Figure 1 and Figure 2 The method process of training a diffusion model according to an embodiment of the present specification is described.

[0070] Figure 2 1 is a flow chart of a method for training a diffusion model in an embodiment of this specification. The method can be executed by a server for model training.

[0071] like Figure 2 As shown, in step S201, a speaking video is obtained, the video including an initial first frame and a plurality of second frames.

[0072] refer to Figure 1 As shown, video clips can be obtained [i-1,i+L] indicates that the video clip includes L+2 frames (images) in total, and seq indicates that the video clip is a sequence of L+2 frames. The video clip includes the initial first frame and the rest of the multiple second frames The first frame The plurality of second frames are used as reference frames. With audio sequence correspond.

[0073] In one embodiment, in order to enable the diffusion model to generate speech videos in real-time streaming, during the training phase, multiple video clips with a preset number of frames can be captured from multiple long videos as training samples. The preset number of frames is, for example, 25 frames.

[0074] In one embodiment, since multiple training samples often show uneven distribution of head directions on the left and right sides, this imbalance may cause the head direction and facial expression corresponding to the motion features generated by the diffusion model to be biased to one side. In order to solve this problem, the facial images in the existing training samples can be horizontally flipped (i.e., swapped left and right) to obtain new training samples, and the new training samples are used together to train the diffusion model, so that the head direction and facial expression in the training samples are evenly distributed on the left and right sides.

[0075] In step S203, the first positions of a plurality of preset points of the target object in a preset state (ie, standard key points) and first motion features of each frame are generated based on the speech video. The first motion features include the offsets of the plurality of preset points relative to the first positions.

[0076] In one embodiment, reference Figure 1 , by taking the first frame and multiple second frames A randomly determined frame in Input Motion Extractor Can be output with the first frame The corresponding first motion feature and random frames The corresponding position features of the canonical key points The location features Includes the first position of each canonical key point. Among them, by obtaining position features from random frames This can increase the robustness of model prediction. In addition, by adding multiple second frames Input motion extractor A sequence of first motion features corresponding to a plurality of second frames may be output

[0077] In step S205, noise is added to the sequence of first motion features of multiple second frames to obtain a noise sequence.

[0078] refer to Figure 1 As shown, the sequence of the first motion feature can be Each motion feature m in j Perform multiple noise additions. Specifically, m j The corresponding {δ j ,R j ,t j Each data in} (such as δ j ,R j or j ) is noised multiple times in each dimension to obtain a noise sequence

[0079] In step S207, an audio feature sequence is obtained based on the audio sequences corresponding to the plurality of second frames.

[0080] In one embodiment, reference Figure 1 As shown, by Input Audio Feature Extractor Output audio feature sequence

[0081] In step S209, a sequence of second motion features corresponding to a plurality of second frames is determined based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence through a diffusion model.

[0082] refer to Figure 1 , the first position of the multiple standard key points obtained above The first motion feature of the first frame Noise sequence and audio feature sequence Inputting a diffusion model to output a sequence of predicted second motion features corresponding to a plurality of second frames

[0083] In one embodiment, if Figure 1 As shown in , other conditions can also be input into the diffusion model.

[0084] Figure 3 This is a schematic diagram of the process of outputting the second motion feature sequence through the diffusion model in the embodiment of this specification.

[0085] like Figure 3As shown, the diffusion model may include an upper neural network module for inputting an initial conditional signal (ICS) and a lower neural network module for inputting an enhanced conditional signal (ECS).

[0086] The ICS signal includes, for example, a reference motion feature sequence, which is obtained by splicing the motion feature m of the target head portrait image copied at a preset time step. ref (As mentioned above ), which time step is consistent with the time step of the audio feature sequence. The reference motion feature sequence is used as a guide for motion generation in each video clip, thereby enhancing the continuity between video clips and periodically guiding the motion to a specified target state, thereby reducing error accumulation in long video sequences (or long sequences of motion features).

[0087] The ECS signal includes, for example, a reference position sequence, an audio feature sequence a (such as the above ), and the expression label sequence and eye state sequence as other conditions. The reference position sequence includes the positions c of the canonical key points of the target avatar copied by splicing at a preset time step. ref (For example, the above ). The expression label sequence can be obtained by the following steps: input the target avatar image into a pre-trained expression classifier to obtain an expression label s, and obtain a sequence of expression labels by splicing the copied expression labels s at a preset time step. It can be understood that in the embodiments of the present specification, in order to perform real-time streaming output, the number of motion features in the motion feature sequence output by the diffusion model (corresponding to a video clip) is preset within a preset range (for example, 25), and therefore, a consistent expression label s can be set for the motion feature sequence. In one embodiment, a sequence of multiple expression labels can also be artificially set for the motion feature sequence. The eye state sequence includes pre-set eye states corresponding to each motion feature in the motion feature sequence, and each eye state includes an identifier for indicating whether the eyes are open or closed and pupil position information, etc. The diffusion model outputs a sequence of predicted motion features corresponding to the target image based on the input ECS and ICS. (For example, the above ).

[0088] The upper neural network module may include a multilayer perceptron (MLP) and multiple ( Figure 32) encoder layers are shown in the figure, each encoder layer includes a self-attention layer (SelfAtt) and an MLP layer. The above-mentioned lower neural network module may include an MLP layer as an input layer, multiple (8 are shown in the figure) encoder layers and an MLP layer as an output starting from the input on the left, wherein the encoder layer may sequentially include a Self Att layer, an element-wise linear modulation layer (Feature-wise Linear Modulation, FiLM), a cross attention layer (CrossAtt), a FiLM layer, an MLP layer and a FiLM layer. Among them, the output of the upper neural network module is connected to the Cross Att in the lower neural network module so that Cross Att fuses the features corresponding to ECS and the features corresponding to ICS.

[0089] Back to Figure 2 In step S211, based on the sequence of first motion features and the sequence of second motion features of the plurality of second frames, the parameters of the diffusion model are adjusted.

[0090] refer to Figure 1 , can be expressed by the loss function shown in the following formula (1) Train the diffusion model:

[0091]

[0092] in, is the mean square error loss, C represents the conditional signal, such as ICS, ECS, etc. mentioned above, and m t is a noise sequence obtained by performing t times of noise addition on the actual motion feature sequence m0. In the embodiment of this specification, m0 is, for example, the sequence of the first motion feature m t is the noise sequence is the sequence of the second motion feature

[0093]

[0094] In one embodiment, as described above, the motion feature m may include {δ, R, t}, where δ is used to represent facial movement, R is used to represent the head rotation angle, and t is used to represent the translation of the head. Since different people in practice will make different facial movements and head movements for the same audio, dynamic adjustment weights can be added to different dimensions in the motion feature during training to make the trained model more in line with the actual situation. Specifically, δ, R, t in the motion feature m can be divided into two groups, δ and R, t, to dynamically adjust the weights of each group. For example, in multiple trainings of the diffusion model, a weight w1 can be set for δ in the motion feature m in one training stage, and a weight w2 can be set for R and t to participate in the calculation of the loss function. In the next training stage, a weight w3 is set for δ, and a weight w4 is set for R and t to participate in the calculation of the loss function. Experiments have shown that the strategy of dynamically adjusting the weights of each group in the motion feature accelerates the convergence of the model and improves the quality of the prediction results of the diffusion model.

[0095] By adjusting the parameters of the diffusion model, the loss function The value of is minimized so that the sequence of the second motion features output by the diffusion model A sequence closer to the first motion feature of the actual video This makes the output of the diffusion model closer to the true motion representation.

[0096] In one embodiment, if Figure 1 As shown in , it is also possible to use the initial motion loss Train a diffusion model where the initial motion loss For reference motion features (such as the above ) and the sequence of the second motion features output by the diffusion model

[0097] The first one The loss between.

[0098] In one embodiment, in order to enhance the temporal stability, the loss of velocity and acceleration corresponding to the motion feature may also be calculated as shown in formula (2):

[0099]

[0100] in, Represents the predicted motion feature sequence output by the diffusion model (such as ), m ′ Represents the actual motion feature sequence (such as ), which can be calculated by dividing the distance difference between each motion feature in the motion feature sequence by time. represents the second-order derivative of the predicted motion feature sequence output by the diffusion model, and m″ represents the second-order derivative of the actual motion feature sequence, which can be calculated by dividing the speed difference corresponding to each motion feature in the motion feature sequence by time.

[0101] Combining the above loss items, we can get the total loss as shown in formula (3):

[0102]

[0103] After the diffusion model is trained through the above process, the model performance can be evaluated based on the existing lip synchronization model. Specifically, a verification sample is obtained, and the verification sample includes a head image of the target object and a target audio sequence; the system of the embodiment of this specification outputs a predicted speech video corresponding to the head image based on the head image and the target audio sequence, and the predicted speech video and the target audio are input into the lip synchronization model, and the lip synchronization model outputs a lip synchronization score, so that the performance of the diffusion model can be evaluated based on the lip synchronization score. The higher the lip synchronization score, the more accurate the prediction of the diffusion model. Among them, the verification sample can be a training sample for training the diffusion model. Specifically, after the diffusion model is trained, a predicted speech video is generated based on the appearance features of the target object, the first position corresponding to the training sample, the first motion feature, and the sequence of the second motion feature, and the lip synchronization score is determined based on the predicted speech video and the audio sequence corresponding to the training sample, and the trained diffusion model is evaluated based on the lip synchronization score.

[0104] In one embodiment, the Figure 1 and Figure 2 The training method shown trains multiple diffusion models, and evaluates the multiple diffusion models based on the evaluation method, for example, the diffusion model with the highest lip synchronization score can be selected as the model actually used. This method solves a common problem in the training of diffusion models, that is, the loss function cannot correctly reflect the performance of the model.

[0105] After the diffusion model is trained, the trained diffusion model may be used to predict a sequence of motion features of the target object based on a single portrait and audio of the target object.

[0106] Figure 4 This is a schematic diagram of the process of obtaining a target object's speaking video by predicting through a diffusion model in an embodiment of this specification.

[0107] like Figure 4 As shown in the figure, the system architecture of this scenario includes Figure 1 Motion Extractor in Audio Feature Extractor and the trained diffusion model In addition, it also includes a pre-trained appearance feature extractor and renderer

[0108] Among them, the appearance feature extractor Used to map the input avatar image s to 3D appearance features f s . Renderer For the appearance feature f based on the reference frame (i.e. a single head image) ref , the position x of the key point of the reference frame ref , the locations of the key points of the predicted frame generated by the diffusion model Synthesized Frames

[0109] Reference below Figure 5 Describe the prediction process of the diffusion model in detail. Figure 5 The illustrated method may be performed by a server for generating a speaking video based on a single avatar image and audio.

[0110] like Figure 5 As shown, in step S501, a head portrait image of a target object is obtained.

[0111] The server may receive a video generation request from a user device, and the video generation request may include an avatar image I for generating a speaking video. ref and voice audio audio.

[0112] In step S503, a first position feature of the target object is generated based on the avatar image.

[0113] refer to Figure 4 , by taking the avatar image I ref Input Motion Extractor Get a first position feature, the first position feature includes the position c of multiple standard key points ref (e.g. 3D coordinates) and avatar image I ref The first motion feature m ref .

[0114] In step S505, a first speech audio is obtained, and an audio feature sequence is obtained based on the first speech audio.

[0115] In one implementation, the first voice audio audio1 may be the voice audio audio received from the user equipment.

[0116] In another embodiment, when the received voice audio is long, in order to make Figure 4The system shown performs streaming output, and can divide the voice audio into a plurality of audio segments (chips) arranged in sequence according to a preset time step, each audio segment includes a preset number of audio frames, and the duration of the audio frame is consistent with the duration of the video frame. Currently, the first audio segment can be obtained as the first voice audio.

[0117] Input the first speech audio into the audio feature extractor Get the audio feature sequence a of the first speech audio seq . In a scenario where the first voice audio is relatively short, the embodiments of the present specification may splice a pre-stored designed voice sample with the first voice audio to provide pseudo-context information. Among them, the splicing method includes splicing the voice sample before the first voice audio, and the voice sample may include preset timbre and content information, etc., and the pseudo-context information is extracted from a fixed corpus. This method enables a model such as the HUBERT model pre-trained based on a longer voice audio to still output a high-quality audio feature sequence when a shorter first voice audio is input. This optimization method allows the diffusion model to process shorter (for example, 0.4 seconds) audio stream units in real time. It can be understood that this optimization method can also be used in the training process of the diffusion model. At the same time, in this optimization method, the audio feature extractor can be The structure is changed to that, for the input concatenated audio, for the speech sampling part, by using masking in the attention layer, only the speech sampling part and its subsequent audio features are paid attention to, that is, only forward attention calculation is performed on the speech sampling part to reduce the amount of calculation brought by the speech sampling part.

[0118] In step S507, based on the first position feature and the audio feature sequence a, a diffusion model is used to generate the first position feature and the audio feature sequence a. seq , determine a sequence of second motion features of multiple preset key points.

[0119] By transforming the first position feature (i.e., c ref and m ref ), audio feature sequence a seq and random noise input to the diffusion model, so that the diffusion model outputs a sequence of predicted second motion features The model structure of the diffusion model can be found in Figure 3 In one embodiment, the diffusion model can automatically generate random noise.

[0120] In one embodiment, reference Figure 3 and Figure 4 As shown, in addition to the first position feature (i.e. c ref and m ref ), audio feature sequence a seqIn addition to the random noise input diffusion model, other conditions can also be input into the diffusion model, such as reference Figure 3 The expression label s and eye state e described in are added to make the diffusion model output more accurate prediction results.

[0121] Optionally, after the second motion feature predicted by the diffusion model output is trained, in step S509 , a speaking video is generated based on a sequence of appearance features, first position features, and second motion features of the target object.

[0122] refer to Figure 4 , by taking the appearance features f of the target object obtained above ref , the first position feature (i.e. c ref and m ref ), and the sequence of the second motion feature Input Renderer Make the renderer Can output speaking video

[0123] In one embodiment, assuming m ref ={δ ref ,R ref ,t ref}, motion features of each predicted frame As shown in formulas (4) and (5), the position x of the reference key point can be generated based on the first position feature. ref and the positions of the key points of each predicted frame

[0124] x ref =c ref R ref +δ ref +t ref (4)

[0125]

[0126] Then f ref 、x ref and each Input Renderer Get each frame of the predicted video

[0127] In one embodiment, in the present specification, a direct and explicit mapping relationship between the dimension in the motion feature m and the facial semantics is pre-established through empirical values, thereby allowing The specific dimensions in the renderer can achieve accurate semantic control of the driving area, facial movement, head movement, etc. Output frames that conform to this semantic control

[0128] For example, by adjusting In To adjust the facial motion in the rendered frame, as described above, It is a 63-dimensional vector, which includes 21 preset key three-dimensional coordinates. Each dimension of affects a different facial region, for example, the 34th dimension controls the closing movement of the right eye, the 58th dimension controls the opening of the mouth, etc. Each dimension of can be adjusted by applying a positive or negative offset. Two types of control are performed: (1) Facial area control, by selecting The specific dimension of can limit the motion generation to a specific local area of ​​the face; (2) size control, by adjusting The size of a specific dimension of the face can be adjusted to adjust the amplitude of the facial movement, for example, by adjusting The size of a specific dimension is used to avoid predicting unnatural facial movements in videos.

[0129] Figure 6 In the embodiments of this specification, by adjusting Schematic diagram of the effect of adjusting facial expressions by the dimensions in . Figure 6 The avatar in the middle of the 9 avatars represents the avatar without applying an offset to any dimension. The mark in the upper left corner of each avatar represents the mark and coordinate mark of the key point. For example, the 15 in the "15-0" in the upper left corner of the avatar represents the 15th key point, and 0 represents the x-axis, that is, the avatar is obtained by adjusting the x-axis value of the 15th key point of the motion feature m of the avatar in the middle. According to the mapping relationship between the dimensions in the above-mentioned preset motion feature m and the facial semantics, the x-axis value of the 15th key point corresponds to the rotation movement of the eyes, so that by adjusting the x-axis value of the 15th key point, the eyes of the rendered avatar turn to the left. It can be understood that by adjusting the Multiple dimensions in the value to achieve complex expressions.

[0130] In one embodiment, in order to output the predicted video in a streaming manner, the first audio segment in the long audio is processed as described above, and the predicted video is output. Afterwards, you can The last frame of the image is used as the reference image I ref , for the second audio clip in the long audio, Figure 4 and Figure 5 The processing shown in the figure obtains the predicted video corresponding to the second audio segment. And so on, so that the sequence of predicted videos corresponding to the long video can be quickly output in a streaming manner This streaming processing method achieves low first frame latency and improves user experience.

[0131] In order to further improve the effect of the above-mentioned streaming output, in the embodiment of this specification, by The adjacent motion feature sequence and A preset length of overlap is introduced between the predicted videos, thereby introducing a predetermined length of overlap between the predicted videos. In addition, for the motion feature sequence Based on each motion feature in the motion feature sequence The distance (relative position) from the motion feature at the center determines each motion feature The weight of the motion feature closer to the center position The larger the weight, the greater the weight. By setting the motion feature sequence The various movement characteristics The weights make the generated predicted video sequence It is smoother and more natural when streaming.

[0132] In addition, in order to provide real-time display performance, the number of denoising steps when acquiring random noise can be reduced in the embodiments of this specification. Thanks to the space of motion features provided by the diffusion model in the embodiments of this specification, even if the number of denoising steps is reduced, video quality comparable to that in the related art can still be obtained.

[0133] Figure 7 The following is a schematic diagram of a prediction device for predicting motion characteristics of a target object according to an embodiment of the present specification, including:

[0134] An acquisition unit 71 is configured to acquire a first image, wherein the first image includes a head portrait of a target object;

[0135] a generating unit 72, configured to generate a first position feature of the target object based on the first image, the first position feature comprising first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion feature comprising first offsets of the plurality of preset points relative to the first position;

[0136] The acquisition unit 71 is further used to acquire the first speech audio, and acquire the audio feature sequence based on the first speech audio;

[0137] The determination unit 73 is used to determine a first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model, where the second motion feature includes a second offset of the plurality of preset points relative to the first position.

[0138] Figure 8 The following is an architecture diagram of a training device for training a diffusion model in an embodiment of the present specification, including:

[0139] An acquisition unit 81 is configured to acquire a first speaking video, where the first speaking video includes an initial first frame and a plurality of second frames;

[0140] A generating unit 82 is configured to generate first positions of a plurality of preset points of the target object in a preset state and first motion features of each frame based on each frame of the first speech video, wherein the first motion features include first offsets of the plurality of preset points relative to the first positions; and add noise to a sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence;

[0141] The acquisition unit 81 is further configured to acquire an audio feature sequence based on the audio sequences corresponding to the plurality of second frames;

[0142] a determining unit 83, configured to determine, by a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion feature includes second offsets of the plurality of preset points relative to the first position;

[0143] The training unit 84 is configured to adjust the parameters of the diffusion model based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

[0144] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0145] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0146] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0147] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0148] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0157] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0158] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.

Claims

1. A method for predicting motion characteristics of a target object, comprising: Acquire a first image, wherein the first image includes a head portrait of a target object; generating a first position feature of the target object based on the first image, the first position feature comprising first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion feature comprising first offsets of the plurality of preset points relative to the first position; Acquire a first voice audio, and acquire an audio feature sequence based on the first voice audio; A first sequence of second motion features is determined based on the first position feature and the audio feature sequence through a diffusion model, wherein the second motion feature includes a second offset of the plurality of preset points relative to the first position.

2. The method according to claim 1, further comprising: A speaking video is generated based on a first sequence of the appearance feature of the target object, the first position feature, and the second motion feature. 3 . The method of claim 1 , wherein the first motion feature comprises an offset corresponding to at least one of: a facial action, a head rotation, and a head translation.

4. The method according to claim 2 further comprises: obtaining a correspondence between a dimension of a predetermined motion feature and facial semantics, adjusting the first sequence of the second motion feature based on the correspondence and the target action to obtain a second sequence, The generating of a speaking video based on a first sequence of the appearance feature of the target object, the first position feature, and the second motion feature comprises: A speaking video is generated based on the appearance feature of the target object, the first position feature and the second sequence.

5. The method according to claim 1, wherein determining the first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model comprises: A first sequence of the second motion features is determined through a diffusion model based on the first position features, the audio feature sequence, a preset expression label, and a preset eye state sequence.

6. The method according to claim 1, wherein the first voice audio is obtained by cutting from the second voice audio, and the duration of the first voice audio corresponds to a preset number of video frames, The step of acquiring an audio feature sequence based on the first speech audio comprises: The pre-stored third voice audio and the first voice audio are spliced ​​and input into an audio feature extraction model to obtain the audio features of the first voice audio, and the third voice audio includes preset timbre and content information.

7. The method according to claim 6, wherein the third speech audio is spliced ​​before the first speech audio, and the splicing of the pre-stored third speech audio and the first speech audio is input into an audio feature extraction model to obtain the audio features of the first speech audio, comprising: The audio feature extraction model is configured to perform forward attention calculation on the third speech audio.

8. The method according to claim 6, wherein determining the first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model comprises: Through the diffusion model, based on the first position feature and the audio feature sequence, a third sequence of third motion features is output, and each of the third motion features is weightedly calculated to obtain the first sequence, wherein the weight of each third motion feature is determined based on the distance of the third motion feature from the center position of the third sequence.

9. The method according to claim 1, wherein generating the first position feature of the target object based on the first image comprises inputting the first image into a motion extractor to obtain the first position feature.

10. The method according to claim 2, further comprising: The first image is input into an appearance feature extractor to obtain the appearance features of the target object.

11. The method according to claim 1, wherein determining the first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model comprises: A first sequence of second motion features is determined based on the first position feature, the audio feature sequence and a sequence of random noise through a diffusion model.

12. A method for training a diffusion model, comprising: Acquire a first speaking video of a target object, wherein the first speaking video includes an initial first frame and a plurality of second frames; generating, based on the first speaking video, first positions of a plurality of preset points of the target object in a preset state and first motion features of each frame, wherein the first motion features include first offsets of the plurality of preset points relative to the first positions; Adding noise to the sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence; Acquire an audio feature sequence based on the audio sequences corresponding to the multiple second frames; Determining, by a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion feature includes second offsets of the plurality of preset points relative to the first position; The parameters of the diffusion model are adjusted based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

13. The method according to claim 12, further comprising: A second speaking video for training a diffusion model is obtained, wherein each frame in the second speaking video is generated by horizontally flipping each frame in the first speaking video.

14. The method according to claim 12, wherein adjusting the parameters of the diffusion model based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames comprises: The dimensions in each second motion feature are grouped, the weights of each group are set, and the dimensions in each group are weighted based on the weights of each group to obtain a sequence of third motion features, and the parameters of the diffusion model are adjusted based on the sequence of the first motion features and the sequence of the third motion features.

15. The method according to claim 12, further comprising: After completing the training of the diffusion model, a predicted speaking video is generated based on the appearance features of the target object, the first position, the first motion features of the first frame, and a sequence of the second motion features, a lip synchronization score is determined based on the predicted speaking video and the audio sequence, and the diffusion model is evaluated based on the lip synchronization score.

16. The method of claim 12, further comprising training the diffusion model based on at least one of the following losses: a loss between a first motion feature of the first frame and a first second motion feature of a sequence of second motion features; The loss between the speed and acceleration corresponding to the sequence of the first motion features corresponding to the plurality of second frames and the speed and acceleration corresponding to the second motion features.

17. A device for predicting motion characteristics of a target object, comprising: An acquisition unit, configured to acquire a first image, wherein the first image includes a head portrait of a target object; a generating unit, configured to generate a first position feature of the target object based on the first image, the first position feature comprising first positions of a plurality of preset points of the target object in a preset state and first motion features of the plurality of preset points, the first motion feature comprising first offsets of the plurality of preset points relative to the first position; The acquisition unit is further used to acquire a first speech audio, and acquire an audio feature sequence based on the first speech audio; A determination unit is used to determine a first sequence of second motion features based on the first position feature and the audio feature sequence through a diffusion model, wherein the second motion feature includes a second offset of the plurality of preset points relative to the first position.

18. A device for training a diffusion model, comprising: An acquisition unit, configured to acquire a first speaking video, wherein the first speaking video includes an initial first frame and a plurality of second frames; A generating unit, configured to generate first positions of a plurality of preset points of the target object in a preset state and first motion features of each frame based on each frame of the first speaking video, wherein the first motion features include first offsets of the plurality of preset points relative to the first positions; Adding noise to the sequence of the first motion features corresponding to the plurality of second frames to obtain a noise sequence; The acquisition unit is further configured to acquire an audio feature sequence based on the audio sequences corresponding to the plurality of second frames; a determining unit, configured to determine, by a diffusion model, a sequence of second motion features corresponding to the plurality of second frames based on the first position, the first motion feature of the first frame, the noise sequence, and the audio feature sequence, wherein the second motion feature includes second offsets of the plurality of preset points relative to the first position; A training unit is used to adjust the parameters of the diffusion model based on the sequence of the first motion features and the sequence of the second motion features corresponding to the plurality of second frames.

19. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 16 is implemented.

Images