Video generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring input information for generating a video, wherein the input information comprises a source image containing a target object and a driving condition at least containing a target audio; based on the source image, a three-dimensional facial representation and a source facial action of the target object are determined. And determining a driving facial action sequence synchronized with the target audio at least based on the target audio. A video of the target object is generated based on a sequence of differential actions between the sequence of source facial actions and the sequence of driving facial actions, and the three-dimensional facial representation of the target object, the video representing that the target object speaks in speech of the target audio. Therefore, the video generation quality and efficiency are improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, storage media, and program products for video generation. Background Art

[0002] With the continuous development of speech-driven face generation technology (Talking Face Generation, TFG), this technology has shown broad potential in application scenarios such as virtual character generation, video conferencing, and intelligent assistants.

[0003] How to generate videos of target objects that are both high-quality and reflect personalized characteristics still faces many challenges, and effective solutions are urgently needed. Summary of the invention

[0004] In a first aspect of the present disclosure, a method for video generation is provided. The method may include: obtaining input information for generating a video, the input information including a source image containing a target object, and a driving condition containing at least a target audio. Based on the source image, a three-dimensional facial representation and a source facial action of the target object are determined. Based at least on the target audio, a driving facial action sequence synchronized with the target audio is determined. Based on a difference action sequence between the source facial action and the driving facial action sequence, and the three-dimensional facial representation of the target object, a video of the target object is generated, the video representing that the target object speaks in the voice of the target audio.

[0005] In a second aspect of the present disclosure, a device for video generation is provided. The device may include: an input information acquisition module, configured to acquire input information for generating a video, the input information including a source image containing a target object, and a driving condition containing at least a target audio. A target object information determination module, configured to determine a three-dimensional facial representation and a source facial action of the target object based on the source image. A driving facial action sequence determination module, configured to determine a driving facial action sequence synchronized with the target audio based at least on the target audio. A video generation module, configured to generate a video of the target object based on a difference action sequence between the source facial action and the driving facial action sequence, and the three-dimensional facial representation of the target object, the video indicating that the target object speaks in the voice of the target audio.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, and when the computer executable instructions are executed by a processor, the method of the first aspect is implemented.

[0009] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 A flowchart of a method for video generation according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic diagram showing a method for video generation according to some embodiments of the present disclosure is shown;

[0014] Figure 4 An example diagram of a fine-tuning process of a video synthesis model according to some embodiments of the present disclosure is shown;

[0015] Figure 5A A schematic diagram of an audio-to-action model reasoning process according to some embodiments of the present disclosure is shown;

[0016] Figure 5B A schematic diagram of training an audio-to-motion model according to some embodiments of the present disclosure is shown;

[0017] Figure 5C A schematic diagram of training samples according to some embodiments of the present disclosure is shown;

[0018] Figure 6 A schematic structural block diagram of an apparatus for video generation according to some embodiments of the present disclosure is shown; and

[0019] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0022] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0029] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, environment 100 may include electronic device 110 .

[0030] In the example environment 100, the electronic device 110 can obtain input information 102. The input information 102 includes at least a target audio 114 and a source image 113 of a target object. Exemplarily, the target object may include humans, animals, cartoon characters, and virtual characters, etc. The electronic device 110 can generate a three-dimensional facial representation corresponding to the target object using a target model 115 based on the source image 113 of the target object. In addition, the electronic device 110 can also generate a facial action sequence synchronized with the target audio 114 based on the target audio 114 using the target model 115, and the facial action sequence is used to drive the action of the three-dimensional facial representation corresponding to the target object. The action of the three-dimensional facial representation is synchronized with the target audio, so a video of the target object speaking with the target audio can be generated. Figure 1 Only one target model 115 is shown as an example. In practice, multiple different target models 115 may collaborate to complete the generation of the video.

[0031] The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server device (not shown) can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. The server device can, for example, provide background services for the application of the electronic device 110.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0033] With the development of speech-driven facial generation technology, how to achieve personalized video generation has attracted more and more attention. The personalization of video generation emphasizes the identity-perceived similarity between the target person in the generated video and the real target person in terms of appearance and speaking style. Existing personalized video generation methods usually train an independent neural radiance field (NeRF) model for each (class) target person (or the type corresponding to the target task) to implicitly store the feature information of the target person. However, since each (class) target person needs to be trained separately and the training data is limited, this method is inefficient and lacks generalization.

[0034] In an embodiment of the present disclosure, an improved scheme for video generation is proposed. In this scheme, an electronic device obtains input information for generating a video, and the input information includes a source image containing a target object, and a driving condition containing at least a target audio. Based on the source image, a three-dimensional facial representation and a source facial action of the target object are determined. Based at least on the target audio, a driving facial action sequence synchronized with the target audio is determined. Based on a difference action sequence between the source facial action and the driving facial action sequence, and the three-dimensional facial representation of the target object, a video of the target object is generated, and the video represents that the target object speaks in the voice of the target audio.

[0035] Through the above process, a 3D facial representation of the target object is generated based on the source image, and a driving facial action sequence is generated synchronously according to the target audio, which can efficiently generate facial actions that match the audio. In addition, the difference action sequence between the source facial action and the driving facial action is used for adjustment to ensure that the generated facial actions are more natural and coherent. This difference-driven generation method reduces the dependence on a large amount of training data and makes the generation process more generalized, able to adapt to different target objects and audio inputs, greatly improving the efficiency and adaptability of video generation.

[0036] Figure 2 FIG. 2 shows an example process of a method 200 for video generation according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 In the environment 100, the video generation can be completed by the electronic device 110, but some operations can be performed by requesting a server device (not shown) (for example, the determination of the driving facial action sequence, the generation of the video, or the training process of part of the model can be implemented at the server device).

[0037] In block 201 , the electronic device 110 obtains input information for generating a video, where the input information includes a source image including a target object and a driving condition including at least a target audio.

[0038] The electronic device 110 may obtain input information for generating a video as a trigger condition for generating a video. Figure 3 A process diagram 300 of a method for video generation according to some embodiments of the present disclosure is shown. Input information may include at least a source image 301 containing a target object and a target audio 305. Exemplarily, the target object may include humans, animals, cartoon characters, and virtual characters, etc. The source image 301 of the target object may be a static image or a video image obtained by performing frame extraction from a video containing the target object.

[0039] The source image 301 containing the target object can be used to determine the three-dimensional facial representation of the target object. The driving condition containing the target audio 305 can be used to generate facial movements synchronized with the target object's speech content. Exemplarily, the driving condition may also include other additional information (such as a reference video for reflecting the speaking style) to further optimize the generated video content.

[0040] In box 202, the electronic device 110 can determine the three-dimensional facial representation 302 and source facial action of the target object based on the source image 301. The three-dimensional facial representation 302 of the target object is a three-dimensional geometric structure model generated by analyzing the source image 301 of the target object, which accurately describes the facial shape and structure of the target object to maintain the identity characteristics of the target object. The source facial action can be the initial facial posture or expression information of the target object in the source image, which is used to describe the state of the face. The source facial action can be a projected normalized coordinate code (PNCC), and the source facial action can be expressed in PNCC. src express.

[0041] Exemplarily, the source image 301 containing the target object may be taken as I src Indicates that the three-dimensional face representation can be expressed using P cano Indicates. Corresponding to P cano =FaceRecon(I src ). FaceRecon can represent a reconstruction model that implements an image to a three-dimensional face, which is used to convert a received image into a three-dimensional facial image.

[0042] In block 203, the electronic device 110 determines a driving facial action sequence synchronized with the target audio based at least on the target audio. Determining the driving facial action sequence may be accomplished using an audio-to-motion model 310. The audio-to-motion model 310 may generate a driving facial action sequence synchronized with the speech based at least on the target audio. The driving facial action sequence may be a plurality of frames, each of which may be synchronized using a PNCC. tgt This sequence represents the driving sequence of facial actions generated under the drive of the target audio, which reflects the dynamic changes of facial expressions and is synchronized with the target audio. The driving facial action sequence can be used to generate dynamic facial actions in the video.

[0043] In order to improve the accuracy of the driven facial action sequence, the input information may further include a reference video 304. The reference video may be a video containing a reference object speaking, and the video may reflect the speaking style of the reference object. The reference person may be a target object or another object other than the target object. Based on the action reference provided by the reference video 304, the driven facial action sequence may be made more realistic and fit the real person.

[0044] At block 204 , the electronic device 110 generates a video of the target object based on the difference motion sequence between the source facial motion and the driving facial motion sequence and the three-dimensional facial representation of the target object, the video representing the target object speaking in the voice of the target audio.

[0045] The source facial action can form an action pair 307 with each driving facial action in the driving facial action sequence. The electronic device 110 can determine a difference action sequence based on the difference between the action pairs. The electronic device 110 uses the video synthesis model 320 to combine the difference action sequence with the three-dimensional facial representation 302 of the target object to generate a dynamic facial video 308 of the target object, so that it is synchronized with the audio to show that the target object is speaking.

[0046] Exemplarily, generating a video of a target object can be expressed by the formula:

[0047] I raw =VolumeRenderer(P cano +MotionAdapter[PNCC src ,PNCC tgt ],cam tgt ) (1)

[0048] MotionAdapter[PNCC src ,PNCC tgt ] can represent the difference motion between the source facial motion and a given driving facial motion in the driving facial motion sequence. tgt It can represent the acquisition position of the driving camera that controls the head posture. VolumeRenderer can represent the processing process of the volume rendering model. raw It can represent the low-resolution image obtained by volume rendering model processing, and then the final high-resolution result is generated by super-resolution model processing: I pred =SuperResolution(I raw ). SuperResolution can represent the processing process of the super-resolution model.

[0049] Through the above process, combined with 3D facial representation and driven facial action sequences, the accuracy and personalized performance of video generation are effectively improved. The generated video can not only maintain the personalized appearance of the target object, but also ensure that the facial movements are highly synchronized with the speech content of the target audio, ultimately presenting natural and smooth facial movements. Through the calculation of differential action sequences, the matching degree of facial expressions and movements is further optimized, making the generated results more realistic and efficient.

[0050] Depending on the information contained in the driving conditions, the determination process of the driving facial action sequence may include two categories. The first category is based on the audio feature information of the target audio. The second category is based on the audio feature information of the target audio and the action feature information of the reference video. The first category is first introduced below: the electronic device 110 extracts the audio feature information of the target audio 305, and the audio feature information at least indicates the duration of the target audio 305 and the pronunciation characteristics of each sound unit. Based on the pronunciation characteristics of each sound unit, the driving facial action sequence corresponding to the duration of the target audio 305 is determined.

[0051] The electronic device 110 extracts audio feature information of the target audio 305, which includes at least the duration of the target audio 305 and the pronunciation features of each sound unit. The duration may indicate the total duration of the target audio 305, while the pronunciation features reflect the speech properties of each sound unit in the target audio 305, such as tone, speech speed, pitch, etc.

[0052] Based on the extracted pronunciation features of each sound unit, the electronic device 110 can use the audio-to-action model 310 to determine a driving facial action sequence corresponding to the duration of the target audio 305. By matching the audio features with the facial actions, it is ensured that the generated facial action sequence can accurately reflect the voice changes in the target audio 305. In this way, facial actions synchronized with the target audio 305 can be generated to achieve dynamic matching of the target object's facial actions with the target audio 305.

[0053] Let's continue with the second category: Combination Figure 3 As shown, in response to the driving condition also including the reference video 304, the electronic device 110 extracts the audio feature information of the target audio 305 and the action feature information of the reference video, the audio feature information at least indicates the duration of the target audio 305 and the pronunciation features of each sound unit, and the action feature information at least indicates the facial action features of the reference object. Based on the pronunciation features and facial action features of each sound unit, a driving facial action sequence that meets the duration is generated.

[0054] The audio feature information of the target audio 305 is the same as the above example, and may at least include the duration of the target audio and the pronunciation features of each sound unit in the target audio 305. The reference video 304 may be a video containing the speech content of the reference object, which can reflect the speaking style and facial movement features of the reference object. The reference object may be the target object itself, or other objects different from the target object. The reference video 304 may provide reference information of speaking style or facial expression. The movement feature information of the reference video may reflect the facial movement features of the reference object in the reference video, such as changes in facial expressions such as mouth opening and closing and eye blinking.

[0055] The electronic device 110 combines the pronunciation features of the target audio 305 with the facial action features in the reference video 304 to generate a driving facial action sequence that can match the duration of the target audio 305. The driving facial action sequence can not only accurately reflect the voice changes in the target audio 305, but also reflect the facial action changes of the reference object in the reference video 304, so that the finally generated video can have both audio synchronization and the facial expression style reflected in the reference video.

[0056] For the second category, the process of generating a driven facial action sequence may specifically include: iteratively executing the following steps until a preset condition is met: based on the pronunciation characteristics and facial action characteristics of each sound unit, determining the first facial action change speed corresponding to each time step in the duration. Based on the first facial action change speed and the initial facial action sequence, determining the first facial action sequence. Based on the first facial action sequence, the first facial action change speed, the pronunciation characteristics and facial action characteristics of each sound unit, determining the second facial action change speed corresponding to each time step. Based on the second facial action change speed and the first facial action sequence, determining the second facial action sequence. Based on the second facial action change speed and the first facial action sequence, generating a driven facial action sequence.

[0057] In the process of generating the driving facial action sequence, each iteration processes a complete action sequence of the same length as the entire target audio 305. The purpose of iteration is to improve the accuracy of the sequence through multiple optimizations. For each time step in the duration, the electronic device 110 can use the audio-to-action model 310 to determine the first facial action change speed within the corresponding duration based on the pronunciation characteristics of each sound unit (such as pitch, sound speed, etc.) and the facial action characteristics in the reference video. The first facial action change speed reflects the action change rate of the facial action in the driving facial action sequence in the time step, and combines the characteristics of the audio to ensure that the facial action is consistent with the sound rhythm of the target audio 305.

[0058] A first facial action sequence is generated by applying the first facial action change speed to the initial facial action sequence. If it corresponds to the first time step, the initial facial action sequence may be a full-noise facial action sequence having the same length as the target audio. If it is not the first time step, the initial facial action sequence is a facial action sequence determined at a time step that is adjacent to the current time step.

[0059] Based on the generated first facial action sequence, the first facial action change speed, the pronunciation characteristics of the sound unit and the facial action characteristics in the reference video, a new second facial action change speed can be iteratively determined. This process optimizes the facial action change speed and ensures that the facial action more accurately fits the expression changes in the audio and reference video.

[0060] Combining the second facial action change speed with the first facial action sequence, a more accurate second facial action sequence can be generated. The second facial action sequence is optimized based on the first facial action sequence, gradually improving the synchronization and refinement between facial actions and audio.

[0061] When the iteration reaches the preset condition, the optimized second facial action sequence can be used to generate a driving facial action sequence. This sequence is synchronized with the target audio at each time step, showing natural and smooth facial movements, and is ultimately used to generate a video of the target object. The preset condition can be that the action change rate is less than a change threshold (indicating convergence), the generated facial action sequence meets the expected quality standard, the iteration has reached a set number of iterations, and so on.

[0062] Through the above process, the audio-to-action model 310 can obtain an action sequence that can be synchronized with the target audio 305, and can also realize the facial action features in the reference video 304. Each iteration optimizes the entire action sequence, gradually approaching an accurate and natural driving facial action sequence from an initial rough action sequence.

[0063] In some embodiments of the present disclosure, the video of the target object is obtained using a trained video synthesis model 320, which includes a basic model and an additional model. The basic model is trained by pre-training, and the additional model is trained by fine-tuning, and the parameters of the pre-trained basic model remain unchanged during the fine-tuning of the additional model.

[0064] The video synthesis model 320 can be composed of a basic model and an additional model. The basic model is trained by pre-training, and is mainly used to provide the basic features and structures required in the video generation process. The additional model is trained by fine-tuning, and is responsible for processing the personalized features of the target object, such as facial expressions and dynamic changes. In the process of fine-tuning the additional model, the parameters of the basic model remain unchanged, so that the additional model can converge faster and require fewer training samples and time. This is because the basic model has learned a wealth of general features, and the additional model trained by fine-tuning only needs to focus on the detailed adjustment of personalized features, which can greatly improve the efficiency and effect of training.

[0065] This approach can improve efficiency and adapt to the personalized needs of different target objects by fixing the basic model and fine-tuning the design of the additional model.

[0066] The following takes the fine-tuning process performed by the electronic device 110 as an example to introduce the fine-tuning process of the additional model in detail. It should be noted that the fine-tuning process of the additional model can also be completed by other electronic devices or by a server device. The electronic device 110 generates a video prediction result of the object sample based on the difference action sequence sample between the source facial action sample of the object sample and the driving facial action sequence sample, and the three-dimensional facial representation of the object sample. The driving facial action sequence sample is determined from the audio contained in the sample video of the object sample, and the three-dimensional facial representation of the object sample is initialized to a three-dimensional facial representation determined from the reference frame of the sample video, and is updated during the fine-tuning process of the additional model. Based on the first difference between each video frame in the video prediction result and each corresponding video frame in the sample video, the parameters of the additional model are fine-tuned.

[0067] Figure 4 FIG4 shows a schematic diagram 400 of fine-tuning an additional model according to some embodiments of the present disclosure. Figure 4 As shown, the basic models of the video synthesis model 320 may include a motion adaptation model 411 -A, a volume rendering model 411 -B, and a super-resolution model 411 -C. The additional models of the video synthesis model 320 may include a low-rank adaptation (LoRA) model 421 .

[0068] The sample video of the object sample may be a sample video including different facial styles. By parsing the audio contained in the sample video of the object sample by the electronic device 110, a driving facial action sequence sample corresponding to the sample video of the object sample may be obtained. The source facial action sample of the object sample may form a sample pair 422 with each driving facial action in the driving facial action sequence sample. By processing the sample pair 422 using the pre-trained motion adaptation model 411-A, a difference action sequence sample 403 may be obtained.

[0069] Figure 4 The 3D reconstruction model 412 in is pre-trained. The 3D reconstruction model 412 processes the reference frame 421 of the sample video containing the object sample to obtain the initialization representation 404 of the 3D face of the object sample. The reference frame 421 of the sample video containing the object sample can be the first frame of the sample video or any other frame. The initialization representation 404 of the 3D face can be used as the 3D face representation of the object sample, and the 3D face representation can be used as the learnable 3D face representation 442.

[0070] The volume rendering model 411-B processes the three-dimensional difference action sequence sample 407 to obtain a low-resolution image. The three-dimensional difference action sequence sample 407 is obtained by fusing the learnable three-dimensional facial representation 442 and the difference action sequence sample 403. The low-resolution image is processed by the super-resolution model 411-C to generate a high-resolution result. The high-resolution result can correspond to the video prediction result of the object sample.

[0071] During the process of the motion adaptation model 411-A, the volume rendering model 411-B and the super-resolution model 411-C processing the input data, the parameters of these basic models remain unchanged. On the other hand, the low-rank adaptation model 421 will also participate in the processing of the input data by the above basic models, and adjust the output results through the parameters of the low-rank adaptation model 421 itself.

[0072] The video prediction result of the object sample obtained by adjusting the output result through the parameters of the low-rank adaptive model 421 itself can be expressed as:

[0073]

[0074] Can represent learnable 3D facial representation 442, MotionAdapter ([PNNC src ',PNCC tgt '] can represent the difference motion samples between the source facial motion samples based on the object samples and the driving facial motion sequence samples. drv ' can represent the acquisition position of the driving camera that controls the head posture. SR can represent the processing process of the super-resolution model. pred ' can represent the video frame in the video prediction result. It can represent adjusting the learnable parameters in the low-rank adaptive model 421.

[0075] Based on the first difference between each video frame 431 of the video prediction result and each corresponding video frame 432 in the sample video, fine-tune the parameters of the additional model and iteratively update the learnable three-dimensional facial representation 442. That is, after each adjustment, the learnable three-dimensional facial representation 442 can be adjusted based on the first difference, and the learnable parameters of the low-rank adaptive model 421 can be adjusted. After the adjustment, the training process continues until the first difference between each video frame 431 of the video prediction result and each corresponding video frame 432 in the sample video converges.

[0076] The first difference between each video frame 431 of the video prediction result and the corresponding video frames 432 in the sample video can indicate at least one of: pixel difference between each video frame in the video prediction result and the corresponding video frames in the sample video, structure and texture difference between each video frame in the video prediction result and the corresponding video frames in the sample video, identity feature difference between each video frame in the video prediction result and the corresponding video frames in the sample video, authenticity difference between each video frame in the video prediction result and the corresponding video frames in the sample video.

[0077] The pixel difference may correspond to the error at the pixel level between each video frame 431 in the video prediction result and each corresponding video frame 432 in the sample video, for example, may include absolute error loss (L1 loss), mean square error loss (L2Loss), and the like.

[0078] The structural and texture differences can correspond to the difference in the perceptual level of the two images in the human visual system (LPIPS loss) evaluated based on the feature representation of the video frames. By capturing the perceptual similarity of structural and texture details, the human visual system's understanding of image similarity is simulated.

[0079] The identity feature difference may correspond to the identity similarity (ID loss) of the objects appearing in each video frame 431 in the video prediction result and each corresponding video frame 432 in the sample video. That is, the training goal is not only to approximate facial movements, but also to satisfy the identity features that keep the target facial features consistent, such as facial contours, the shape and distance of eyes and eyebrows, and so on.

[0080] The authenticity difference may correspond to the authenticity evaluation (GAN loss) between each video frame 431 in the video prediction result and each corresponding video frame 432 in the sample video. Specifically, the authenticity difference may be used to measure the difference in "realism" between each video frame 431 in the video prediction result and each corresponding video frame 432 in the sample video, that is, whether the discriminator can distinguish whether the generated video frame comes from the sample video.

[0081] Exemplarily, by assigning corresponding weights (λ) to different losses, the parameters of the additional model can be fine-tuned based on the first differences in each dimension and the three-dimensional facial initialization representation of the object sample can be iteratively updated. The first difference can be expressed as:

[0082] L SD-Hybrid =λ1*L1+λ LPIPS *L LPIPS +λ ID *L ID +λ GAN *L GAN (3)

[0083] L SD-Hybrid L can represent the first difference, L1 and λ1 can represent the pixel difference and the corresponding weight respectively. LPIPS and λ LPIPS can represent the structural and texture differences and the corresponding weights respectively. ID and λ ID can represent the identity feature differences and the corresponding weights respectively. GAN and λ GAN The authenticity differences and the corresponding weights can be represented respectively.

[0084] The videos generated by the fine-tuned model are not only accurate at the pixel level, but also have high quality in structure, identity consistency and overall realism, and can better meet the multi-dimensional requirements of practical applications.

[0085] The above is based on Figure 4 The fine-tuning process of the video synthesis model 320 is introduced. In the inference stage, the 3D reconstruction model 412 processes the source image containing the target object to obtain a 3D facial representation. The motion adaptation model 411-A in the trained video synthesis model 320 processes the source facial action and the driving facial action sequence output by the audio-to-action model 310 to obtain a 3D difference action sequence between the source facial action and the driving facial action sequence. The volume rendering model 411-B processes the 3D difference action sequence to obtain a low-resolution image. The 3D difference action sequence is obtained by fusion of the 3D facial representation and the driving facial action sequence. The low-resolution image is processed by the super-resolution model 411-C to generate a high-resolution result. The high-resolution result can correspond to the video prediction result of the object sample. The low-rank adaptation model 421 will simultaneously participate in the processing of the input data by the waybill adaptation model 411-A, the volume rendering model 411-B and the super-resolution model 411-C.

[0086] The driving facial action sequence in the aforementioned example is obtained using the audio-to-motion model 310, which is trained in the following manner: performing masking processing on a reference sample video containing an object sample to obtain a processed reference sample video. The processed reference sample video and a given sample audio are combined to obtain an input sample. Based on each time step in the sample audio duration, the input sample is iteratively processed using the audio-to-motion model to obtain a driving facial action sequence prediction result corresponding to each time step, and the driving facial action sequence prediction result corresponding to each time step is generated based on the driving facial action sequence prediction result corresponding to the previous adjacent time. The parameters of the audio-to-motion model are adjusted based on the second difference between the driving facial action sequence prediction result corresponding to each time step and the driving facial action sequence sample, and the driving facial action sequence sample is obtained based on the reference sample video.

[0087] Before introducing the training process of the audio-to-motion model 310 , the inference process of the audio-to-motion model 310 is first introduced. Figure 5A A schematic diagram 500A of the reasoning process of the audio-to-action model 310 according to some embodiments of the present disclosure is shown. Using the audio content contained in the reference video 304, a Mel spectrum 501 corresponding to the audio can be obtained. Using the audio contained in the reference video 304, a driving facial action sequence 502 can be generated. The features of the reference video 304 can be expressed as [B, T1, Ca1+Cm1]. B can represent the amount of data, T1 can represent the duration, Ca1 can represent the Mel spectrum 501, and Cm1 can represent the features corresponding to the driving facial action sequence 502.

[0088] The features corresponding to the target audio 305 can be expressed as [B, T2, Ca2]. B can represent the amount of data, T2 can represent the duration, and Ca2 can represent the Mel spectrum 503 of the target audio 305. The features of the reference video 304 and the target audio 305 are combined to serve as input data of the audio-to-action model 310. The audio-to-action model 310 determines the driving facial action sequence 511 based on the input data.

[0089] Figure 5BA specific structural schematic diagram 500B of the audio-to-action model 310 according to some embodiments of the present disclosure is shown. The audio-to-action model 310 may include a forward reasoning submodule 310-A ​​and a differential equation solving submodule 310-B. In the initial state x0 of reasoning, the driving facial action sequence is blank or random noise. Thereafter, the forward reasoning submodule 310-A ​​may begin to determine the preliminary change state x1 of the facial action based on the input data [B, T1+T2, Ca1+Ca2+Cm1]. The input data may be the features (Ca1+Ca2+Cm1) corresponding to the reference video 304 and the target audio 305 in a duration of T (T1+T2), and the data volume is B. The differential equation solving submodule 310-B determines the action change rate dxt / dt at each time step based on the preliminary changes in the facial action.

[0090] Through the intermediate iteration (state x2 to state x tT-1 ), the differential equation solving submodule 310-B determines the action change rate dxt / dt corresponding to each iteration process, and the action change rate is combined with the change state output by the forward reasoning submodule 310-A ​​to gradually generate a more coherent and natural facial action sequence. After multiple iterations, the final complete action sequence (x tT ), the result is highly synchronized with the input audio and video features, and exhibits natural facial movements and expression changes.

[0091] The training process of the audio-to-motion model 310 is introduced below. Figure 5C A schematic diagram 500C of a training sample according to some embodiments of the present disclosure is shown. The training sample includes aligned sample audio 521 and sample video 522. Masking is performed on the sample video 522, and the masked portion can simulate the missing data to generate a processed reference sample video. The training goal of the audio-to-action model 310 is to reconstruct the masked action sequence. Through training, the audio-to-action model 310 can fill in the missing action sequence based on the sample audio and the unmasked action portion to generate natural facial action changes.

[0092] During the training process, the audio-to-motion model 310 generates an action sequence by calculating the rate of change of the action (dxt / dt) at each time step. For example, in the initial state (x0), the input sample only contains sample audio but no sample video, and the output of the model is a blank sequence of length T (T1+T2). In the first iteration (x1), the input sample includes sample audio and a blank sequence of the initial state, and the audio-to-motion model 310 can begin to generate preliminary changes in facial movements based on audio features (such as pitch, speaking rate, stress, etc.). The audio-to-motion model 310 determines the rate of change dxt / dt for the first time step, which is the predicted facial movement trend (such as the opening and closing of the mouth or the movement of facial muscles) based on the audio signal.

[0093] Based on the second difference between the driving facial action sequence prediction result corresponding to each time step and the driving facial action sequence sample, the parameters of the audio-to-action model are adjusted. Exemplarily, the second difference can at least indicate the difference in the action change trajectory. Adjusting the parameters of the audio-to-action model 310 based on the second difference can be expressed as:

[0094]

[0095] L CFM It can be expressed as: at different time steps (t), under the real data distribution q(x) and the conditional probability distribution p(x|x1), the real speed u is calculated t (x|x1) and the model prediction speed v t The difference (i.e., error) between (x; θ) and the expectation (mean) of this difference. t (x; θ) can represent the prediction speed (the result predicted at time step t based on the current state x and model parameters θ). t (x|x1) can represent the optimal path from the current state x to the target state x1 at time step t. t (x|x1) is the ideal action change rate (true value) determined by driving the facial action sequence samples, and the driving facial action sequence samples can be obtained based on the reference sample video.

[0096] For example, for a data point x in the true data distribution q(x) t The true velocity at continuous time steps can be expressed as u t =dxt / dt. The data points x0~p(x) obey a simple conditional probability distribution p(x) (such as Gaussian distribution), starting at t=0, and as the time step approaches t=1, the velocity field is pushed to the real data distribution q(x). This process can be expressed as a differential equation:

[0097] dxt / dt=u t≈v t (x;θ)stx o ~p(x)t∈[0,1] (5)

[0098] The goal of the entire training process is to continuously optimize v t (x; θ) to make it as close to dxt / dt as possible. Therefore, adjusting the parameters of the audio-to-motion model 310 based on the second difference can be performed based on the following loss function:

[0099]

[0100] Through training, we can obtain v t (x t ;θ)≈dxt / dt, thus the predicted result x1 can be obtained by expressing (5).

[0101] After multiple iterations of training, the input of the audio-to-motion model already contains rich facial motion information, and the rate of change dxt / dt gradually decreases because the generation of facial motions has become stable and the final changes are more about fine-tuning details. This indicates the end of training.

[0102] In the above example, the second difference may indicate the difference in the motion change trajectory (L CFM ). In addition, the second difference can also indicate the audio-visual synchronization difference (Lsync) between the driving facial action sequence prediction result and the driving facial action sequence sample. The audio-visual synchronization difference evaluates the degree of synchronization between the generated facial action sequence and the audio input, and this part of the difference is measured by synchronization loss. Synchronization loss can be used to ensure that facial actions and speech rhythms are synchronized in time to avoid the situation where the audio and picture are out of sync. Exemplarily, corresponding weights (λ) are assigned to the action change trajectory differences and the audio-visual synchronization differences, and the parameters of the audio-to-action model can be adjusted based on the second differences in the above dimensions. The second difference can be expressed in the form of the following loss function:

[0103] L ICS-A2M =λ CFM *L CFM+ =λsync*Lsync (7)

[0104] L ICS-A2M Can represent the second difference. CFM and λ CFM can represent the difference in action change trajectories and the corresponding weights, L CFM and λ CFM The difference in audio and video synchronization and the corresponding weights can be respectively represented.

[0105] Figure 6A schematic structural block diagram of an apparatus 600 for video generation according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented in or included in the electronic device 110, for example. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0106] like Figure 6 As shown, the device 600 may include an input information acquisition module 601, which is configured to acquire input information for generating a video, wherein the input information includes a source image containing a target object, and a driving condition containing at least a target audio. A target object information determination module 602 is configured to determine a three-dimensional facial representation and a source facial action of the target object based on the source image. A driving facial action sequence determination module 603 is configured to determine a driving facial action sequence synchronized with the target audio based at least on the target audio. A video generation module 604 is configured to generate a video of the target object based on a difference action sequence between the source facial action and the driving facial action sequence, and the three-dimensional facial representation of the target object, wherein the video represents that the target object speaks in the voice of the target audio.

[0107] In some embodiments of the present disclosure, the driven facial action sequence determination module 603 may be specifically configured to: extract audio feature information of the target audio, where the audio feature information at least indicates the duration of the target audio and the pronunciation features of each sound unit. Based on the pronunciation features of each sound unit, determine the driven facial action sequence corresponding to the duration of the target audio.

[0108] In some embodiments of the present disclosure, the driving condition also includes a reference video. Based on this, the driving facial action sequence determination module 603 can also be specifically configured to: extract the audio feature information of the target audio and the action feature information of the reference video, the audio feature information at least indicates the duration of the target audio and the pronunciation features of each sound unit, and the action feature information at least indicates the facial action features of the reference object. Based on the pronunciation features and facial action features of each sound unit, a driving facial action sequence that meets the duration is generated.

[0109] In some embodiments of the present disclosure, the driven facial action sequence determination module 603 can be specifically configured to: iteratively perform the following steps until a preset condition is met: based on the pronunciation characteristics and facial action characteristics of each sound unit, determine the first facial action change speed corresponding to each time step in the duration. Based on the first facial action change speed and the initial facial action sequence, determine the first facial action sequence. Based on the first facial action sequence, the first facial action change speed, the pronunciation characteristics and facial action characteristics of each sound unit, determine the second facial action change speed corresponding to each time step. Based on the second facial action change speed and the first facial action sequence, determine the second facial action sequence. Based on the second facial action change speed and the first facial action sequence, generate a driven facial action sequence.

[0110] In some embodiments of the present disclosure, the video of the target object is obtained using a trained video synthesis model, and the video synthesis model includes a basic model and an additional model. This also includes a model training module, which trains the basic model by pre-training and trains the additional model by fine-tuning. That is, the basic model is trained by pre-training, and the additional model is trained by fine-tuning, and the parameters of the pre-trained basic model remain unchanged during the fine-tuning process of the additional model.

[0111] In some embodiments of the present disclosure, the model training module can be configured to: generate a video prediction result of the object sample based on a difference action sequence sample between a source facial action sample of the object sample and a driving facial action sequence sample, and a three-dimensional facial representation of the object sample, wherein the driving facial action sequence sample is determined from the audio contained in the sample video of the object sample, and the three-dimensional facial representation of the object sample is initialized to a three-dimensional facial representation determined from a reference frame of the sample video and is updated during the fine-tuning process of the additional model. Fine-tune the parameters of the additional model based on the first difference between each video frame in the video prediction result and each corresponding video frame in the sample video.

[0112] In some embodiments of the present disclosure, the first difference indicates at least one of the following: pixel differences between each video frame in the video prediction result and each corresponding video frame in the sample video, structural and texture differences between each video frame in the video prediction result and each corresponding video frame in the sample video, identity feature differences between each video frame in the video prediction result and each corresponding video frame in the sample video, and authenticity differences between each video frame in the video prediction result and each corresponding video frame in the sample video.

[0113] In some embodiments of the present disclosure, the driven facial action sequence is obtained using an audio-to-action model. Based on this, the model training module can be configured to: perform masking processing on a reference sample video containing an object sample to obtain a processed reference sample video. Combine the processed reference sample video and a given sample audio to obtain an input sample. Based on each time step in the sample audio duration, the input sample is iteratively processed using the audio-to-action model to obtain a driven facial action sequence prediction result corresponding to each time step, and the driven facial action sequence prediction result corresponding to each time step is generated based on the driven facial action sequence prediction result corresponding to the previous adjacent time. Based on the second difference between the driven facial action sequence prediction result corresponding to each time step and the driven facial action sequence sample, the parameters of the audio-to-action model are adjusted, and the driven facial action sequence sample is obtained based on the reference sample video.

[0114] In some embodiments of the present disclosure, the second difference indicates at least one of the following: a difference in motion change trajectories between the driving facial motion sequence material and the driving facial motion sequence samples, and a difference in sound-image synchronization between the driving facial motion sequence material and the driving facial motion sequence samples.

[0115] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown may include or be implemented as Figure 1 Electronic device 110 or Figure 6 Device 600.

[0116] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0117] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0118] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0119] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0120] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0121] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0122] According to an exemplary implementation of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 2 The methods provided in the various optional methods are therefore not described in detail here.

[0123] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0125] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0126] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0127] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for video generation, comprising: Acquire input information for generating a video, the input information comprising a source image including a target object and a driving condition including at least a target audio; determining a three-dimensional facial representation and a source facial motion of the target object based on the source image; Based at least on the target audio, determining a driving facial action sequence synchronized with the target audio; and Based on a difference motion sequence between the source facial motions and the driving facial motion sequence, and the three-dimensional facial representation of the target object, a video of the target object is generated, the video representing the target object speaking in the voice of the target audio.

2. The method of claim 1, wherein determining a driving facial action sequence synchronized with the target audio comprises: Extracting audio feature information of the target audio, where the audio feature information at least indicates the duration of the target audio and pronunciation features of each sound unit; as well as Based on the pronunciation features of the sound units, the driving facial action sequence corresponding to the duration of the target audio is determined.

3. The method of claim 1 , wherein the driving condition further comprises a reference video, and wherein determining a driving facial action sequence synchronized with the target audio comprises: extracting audio feature information of the target audio and motion feature information of the reference video, wherein the audio feature information at least indicates the duration of the target audio and pronunciation features of each sound unit, and the motion feature information at least indicates facial motion features of the reference object; and The driven facial action sequence satisfying the duration is generated based on the pronunciation features of the sound units and the facial action features.

4. The method according to claim 3, wherein generating the driving facial action sequence based on the pronunciation features of the sound units and the facial action features comprises: Iteratively perform the following steps until the preset conditions are met: Determining a first facial action change speed corresponding to each time step in the duration based on the pronunciation features of the sound units and the facial action features; determining a first facial action sequence based on the first facial action change speed and an initial facial action sequence; Determining a second facial action change speed corresponding to each time step based on the first facial action sequence, the first facial action change speed, the pronunciation characteristics of each sound unit, and the facial action characteristics; determining a second facial action sequence based on the second facial action change speed and the first facial action sequence; and The driving facial action sequence is generated based on the second facial action sequence determined after the preset condition is met.

5. The method according to claim 1, wherein the video of the target object is obtained using a trained video synthesis model, wherein the video synthesis model includes a base model and an additional model; The basic model is trained by pre-training, and the additional model is trained by fine-tuning. During the fine-tuning of the additional model, the parameters of the pre-trained basic model remain unchanged.

6. The method of claim 5, wherein the fine-tuning of the additional model comprises: generating a video prediction result of the object sample based on a difference action sequence sample between a source facial action sample and a driving facial action sequence sample of the object sample, and a three-dimensional facial representation of the object sample, wherein the driving facial action sequence sample is determined from an audio contained in a sample video of the object sample, and the three-dimensional facial representation of the object sample is initialized to a three-dimensional facial representation determined from a reference frame of the sample video and is updated during the fine-tuning process of the additional model; as well as Based on the first difference between each video frame in the video prediction result and each corresponding video frame in the sample video, the parameters of the additional model are fine-tuned.

7. The method of claim 6, wherein the first difference indicates at least one of: The pixel difference between each video frame in the video prediction result and each corresponding video frame in the sample video, The structural and texture differences between each video frame in the video prediction result and each corresponding video frame in the sample video, The identity feature difference between each video frame in the video prediction result and each corresponding video frame in the sample video, The authenticity difference between each video frame in the video prediction result and each corresponding video frame in the sample video.

8. The method of claim 3, wherein the driven facial action sequence is obtained using an audio-to-action model, wherein the audio-to-action model is trained by: Performing masking processing on a reference sample video containing an object sample to obtain a processed reference sample video; Combining the processed reference sample video with a given sample audio to obtain an input sample; Based on each time step in the sample audio duration, iteratively process the input sample using the audio-to-action model to obtain a driving facial action sequence prediction result corresponding to each time step, wherein the driving facial action sequence prediction result corresponding to each time step is generated based on a driving facial action sequence prediction result corresponding to a previous adjacent time step; Adjust the parameters of the audio-to-motion model based on the second difference between the driving facial action sequence prediction result corresponding to each time step and the driving facial action sequence sample, wherein the driving facial action sequence sample is obtained based on the reference sample video.

9. The method of claim 8, wherein the second difference indicates at least one of: The difference between the movement change trajectory of the driving facial movement sequence material and the driving facial movement sequence sample, The audio and video synchronization difference between the driving facial action sequence material and the driving facial action sequence sample.

10. A device for video generation, comprising: An input information acquisition module, configured to acquire input information for generating a video, wherein the input information includes a source image including a target object and a driving condition including at least a target audio; a target object information determination module, configured to determine a three-dimensional facial representation and a source facial action of the target object based on the source image; A driving facial action sequence determination module, configured to determine a driving facial action sequence synchronized with the target audio based at least on the target audio; as well as A video generation module is configured to generate a video of the target object based on a difference motion sequence between the source facial motion and the driving facial motion sequence, and the three-dimensional facial representation of the target object, wherein the video represents the target object speaking in the voice of the target audio.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, comprising computer executable instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images