Video generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: performing masking on a predetermined area of a target object in a reference video to obtain a masked video; a first video feature representation of the reference video and a second video feature representation of the masked video are respectively determined. An audio feature representation of the target audio is determined. And at least based on the first video feature representation, the second video feature representation and the audio feature representation, utilizing a trained video generation model to generate a target video containing the target object, the target video representing that the target object speaks the target audio in a mouth shape matched with the target audio. Therefore, the video generation quality can be improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, storage media, and program products for video generation. Background Art

[0002] With the continuous development of speech-driven video motion synchronization technology, this technology has shown broad potential in application scenarios such as virtual character generation, dubbing and video conferencing. As an important branch of speech-driven video generation, lip synchronization technology has the core task of generating accurate lip movements based on the corresponding speech. How to achieve temporal consistency between lip movements and target language is a technical problem that needs to be solved. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for video generation is provided. The method may include: obtaining a masked video by performing masking on a predetermined area of ​​a target object in a reference video. Determining a first video feature representation of the reference video and a second video feature representation of the masked video respectively. Determining an audio feature representation of a target audio. Based at least on the first video feature representation, the second video feature representation, and the audio feature representation, a target video containing the target object is generated using a trained video generation model, wherein the target video represents that the target object speaks the target audio with a mouth shape that matches the target audio.

[0004] In a second aspect of the present disclosure, a device for video generation is provided. The device may include: a mask video determination module, configured to obtain a mask video by performing masking on a predetermined area of ​​a target object in a reference video. A video feature representation determination module, configured to respectively determine a first video feature representation of a reference video and a second video feature representation of a mask video. An audio feature representation determination module, configured to determine an audio feature representation of a target audio. A target video generation module, configured to generate a target video containing a target object using a trained video generation model based at least on the first video feature representation, the second video feature representation, and the audio feature representation, wherein the target video represents that the target object speaks the target audio with a mouth shape that matches the target audio.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, and when the computer executable instructions are executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart of a method for video generation according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of a video generation architecture according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram showing a process of processing a video frame in which a target object is tilted according to some embodiments of the present disclosure is shown;

[0014] Figure 5 An example diagram showing a training process of a video generation model according to some embodiments of the present disclosure;

[0015] Fig. 6A A schematic diagram showing a process of determining a time synchronization difference according to some embodiments of the present disclosure;

[0016] Figure 6B A schematic diagram of a training process of a synchronization network according to some embodiments of the present disclosure is shown;

[0017] Figure 7 A schematic structural block diagram of an apparatus for video generation according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0021] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0024] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0025] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0026] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0027] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0028] With the continuous development of speech-driven image generation technology, this technology has shown broad potential in application scenarios such as virtual character generation, video conferencing, and smart assistants. As an important branch of speech-driven image generation, lip synchronization technology has the core task of generating accurate lip movements based on the corresponding speech while maintaining the integrity of head posture and individual identity characteristics.

[0029] Currently, the most popular lip synchronization techniques are mainly based on generative adversarial networks (GANs). However, GANs face some limitations in practical applications, such as unstable training process, mode collapse, and difficulty in scaling to large and diverse datasets.

[0030] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, environment 100 may include electronic device 110 .

[0031] In the example environment 100, the electronic device 110 may obtain input information 102. The input information 102 includes at least a target audio 114 and a reference video 113 of a target object. As an example, the target object may include humans, animals, cartoon characters, and virtual characters, etc. The electronic device 110 may generate a target object speaking the target audio 114 in a mouth shape matching the target audio 114 based on the reference video 113 and the target audio 114 of the target object. Figure 1 Only one target model 115 is shown as an example. In practice, multiple different target models 115 may collaborate to complete the generation of the video.

[0032] The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server device (not shown) can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. The server device can, for example, provide background services for the application of the electronic device 110.

[0033] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0034] In an embodiment of the present disclosure, an improved scheme for video generation is proposed. In the scheme, an electronic device obtains a masked video by performing a mask on a predetermined area of ​​a target object in a reference video. A first video feature representation of the reference video and a second video feature representation of the masked video are determined respectively. An audio feature representation of a target audio is determined. Based at least on the first video feature representation, the second video feature representation, and the audio feature representation, a target video containing the target object is generated using a trained video generation model, wherein the target video represents that the target object speaks the target audio with a mouth shape that matches the target audio.

[0035] Through the above process, a masked video is generated by masking a predetermined area of ​​the target object in the reference video, so that the video generation process can focus on a specific area (such as the mouth) of the target object. This processing method enables the video generation model to generate the lip movements of the target object more accurately and avoid interference from irrelevant areas. By determining the feature representations of the reference video and the mask video respectively, and combining them with the audio feature representation, a close association between audio and video is achieved. By extracting video features and audio features, the audio and video synchronization problem in complex scenes can be effectively handled without relying on additional labeled data. Based on these feature representations, the target video is generated using the video generation model to ensure that the target object in the target video matches the target audio in terms of lip shape.

[0036] Figure 2 FIG. 2 shows an example process of a method 200 for video generation according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 In the environment 100, the video generation can be completed by the electronic device 110, but some operations can be performed by requesting a server device (not shown) (for example, the determination of feature representation, the determination of target video, or the training process of part of the model can be implemented at the server device). For ease of understanding, in the description of process 200, the video generation can be completed by the electronic device 110, but some operations can be performed by requesting a server device (not shown) (for example, the determination of feature representation, the determination of target video, or the training process of part of the model can be implemented at the server device). Figure 3 The video generation architecture 300 is discussed.

[0037] In block 201 , the electronic device 110 obtains a masked video 301 by performing masking on a predetermined area of ​​a target object in a reference video 113 .

[0038] The reference video 113 usually includes the activities or behaviors of the target object. The reference video 113 can be any video containing the target object, or a video extracted from other related media (such as film clips, user-generated content, etc.). The target object can be any object for which image generation is required, for example, the target object can include humans, animals, cartoon characters, virtual characters, etc.

[0039] For the task of generating the target video 104, the target video 104 is generated based on the facial expression of the target object in the reference video 113 (whether in speech or silence), and the target object's mouth shape and the target audio (the target audio is different from the speech in the reference video) are accurately matched in the target video 104. In order to achieve this goal, the target object in the reference video 113 needs to be masked first.

[0040] The mask processing may include identifying and calibrating a predetermined area of ​​the target object in the reference video 113, especially the mouth area or other facial feature areas that need to be generated or adjusted, and performing masking on the predetermined area to obtain the mask video 301. The mask can be implemented by an image processing algorithm to ensure that the predetermined area can be accurately located and processed in the subsequent generation process. By masking the predetermined area (e.g., the mouth area), the model can pay more attention to information in other areas outside the predetermined area from the mask video 301.

[0041] In block 202 , the electronic device 110 determines a first video feature representation of the reference video 113 and a second video feature representation of the mask video 301 , respectively.

[0042] In some embodiments, the first video feature representation of the reference video 113 is obtained by extracting features from the reference video 113 based on a trained video encoder model 305, and the second video feature representation of the mask video 301 is obtained by extracting features from the mask video 301 based on a trained video encoder model 305.

[0043] In some embodiments, in order to effectively process high-resolution images, the electronic device 110 converts the high-dimensional data of the original video into a lower-dimensional feature representation through a dimensionality reduction technique. The video encoder model 305 can use a variational autoencoder (VAE) encoder for feature extraction. The video encoder model 305 can convert the original data of the high-resolution video into a low-dimensional latent variable representation. In this process, the video encoder model 305 can not only effectively compress the facial movements and visual features in the video, but also retain important semantic information. By converting the visual information of the video into a representation in the latent space, the video encoder model 305 helps to reduce the amount of calculation, making the processing of high-resolution videos more efficient. In addition, the video encoder model 305 can learn the implicit distribution of the data, allowing more efficient generation and interpolation in the latent feature space, and further enhancing the quality and consistency of video generation. It should be understood that the model structure and configuration of the video encoder model can have a variety of options, and the embodiments of the present disclosure are not limited to this.

[0044] In box 203, the electronic device 110 determines the audio feature representation of the target audio 114. The target audio 114 may be audio content different from the speech in the reference video, for example, the content may be different, the language may be different, and so on. In some embodiments, the audio feature representation of the target audio 114 may be obtained by extracting the Mel-spectrogram of the target audio 114 using the trained audio encoder model 306. The Mel-spectrogram is the result of representing the target audio 114 on the Mel frequency scale after the target audio 114 is processed by short-time Fourier transform, and can effectively capture the frequency features in the target audio 114 and its dynamic information that changes over time. It can be understood that in addition to the Mel-spectrogram, the audio feature representation of the target audio 114 can also be extracted based on other acoustic information.

[0045] In box 204, the electronic device 110 generates a target video 104 containing a target object using the trained video generation model 307 based on at least the first video feature representation, the second video feature representation and the audio feature representation, wherein the target video 104 indicates that the target object speaks the target audio 114 with lip movements matching the target audio.

[0046] In some embodiments, feature representation 304 may be obtained by aggregating the first video feature representation of reference video 113 and the second video feature representation of mask video 301. Feature representation 304 may include a cascade representation of the two feature representations. Feature representation 304 and the audio feature representation of target audio 306 are input into video generation model 307 together.

[0047] The video generation model 307 can be constructed based on a diffusion model. As a generative model, the diffusion model generates new data by simulating the forward diffusion process (gradually adding noise) and the reverse diffusion process (gradually denoising). In the generation process, the diffusion model can start from pure noise, and based on the input information (video feature representation and audio feature identification in this article) as a condition, gradually remove the noise through a series of reverse denoising steps to restore the target video content that matches the target audio.

[0048] Specifically, the video generation model 307 generates a target video 104 synchronized with the target audio 114 through a step-by-step denoising back-diffusion process. Each step of the denoising process is adjusted based on the input features (including the first video feature representation, the second video feature representation, and the audio feature representation) to ensure that the target object's lip shape is synchronized with the target audio 114. In this process, each time step corresponds to a step-by-step conversion from noise to real data, reflecting the step-by-step matching of the audio-driven lip shape generation with the target video. Compared with related video generation methods, the advantage of the video generation model 307 is that it can more accurately control the generation details through a back-diffusion process of multiple time steps, and can stably generate the target video 104 at high resolution. The target video 104 not only ensures the lip synchronization of the target object, but also has more expressiveness and fluency in details.

[0049] Through the above process, the electronic device 110 extracts the feature representations of the reference video 113 and the mask video 301 respectively, and combines them with the audio feature representation of the target audio 114, effectively realizing the close association between audio and video. This feature extraction and fusion method enables the generated target video 104 to match the target audio 114 more accurately, ensuring that the target object in the target video 114 presents lip movements synchronized with the target audio 114. The diffusion process of the video generation model 307 not only improves the quality of the generated video, but also enhances the temporal consistency and detail expression in the generation process, so that the generated target video 104 is highly consistent in both visual and auditory effects.

[0050] Combination Figure 3As shown, the electronic device 110 performing masking on the predetermined area of ​​the target object in the reference video 113 may include using multiple mask images 302 to perform masking on the predetermined area of ​​the target object in each video frame of the reference video 113. Based on this, the electronic device 110 can determine the mask feature representation of the multiple mask images 302. The mask feature representation will also be input into the video generation model 307, so that the target video 104 containing the target object is generated by the video generation model 307. The mask feature representation can be aggregated (for example, by cascading) to the video feature representation of the reference video 113 and the mask video 301 to form a feature representation 304.

[0051] The electronic device 110 can use the mask map to perform masking on a predetermined area (e.g., a mouth area) of the target object in each video frame of the reference video 113. These mask maps not only mark the area that needs to be processed, but also provide a precise guidance information indicating the specific area that the video generation model 307 needs to focus on.

[0052] Based on multiple mask images 302, the input of the video generation model 307 therefore includes not only the feature representations of the reference video 113, the mask video 301, and the target audio 114, but also the mask feature representation of the mask image 302 corresponding to each frame of the reference video 113. The mask image 302 can be used as an additional input to help the video generation model 307 process the predetermined area more accurately during the generation of the target video 104, ensuring that the generated target video 104 can truly reflect the lip shape and facial expression of the target object.

[0053] By introducing the mask map 302 into the input of the video generation model, the electronic device 110 can rely on these visual indications to improve the accuracy and consistency of the generated results when generating the target video 104. The above improvements enable the generated target video 104 to not only accurately match the target audio 114, but also ensure that the facial features of the target object in dynamic changes are correctly expressed.

[0054] Figure 4 A schematic diagram of a process 400 for processing a video frame in which a target object is tilted according to some embodiments of the present disclosure is shown. Figure 4As shown, in some scenarios, the target object in each video frame 113-a of the reference video 113 is tilted. For such a situation, the electronic device 110 can perform an angle transformation on the target object in each video frame of the reference video to obtain a transformed reference video 113-b. Based on this, the electronic device 110 can determine the first video feature representation from the transformed reference video 113-b. Correspondingly, the electronic device 110 can generate an intermediate video based on at least the first video feature representation, the second video feature representation, and the audio feature representation, and perform an inverse angle transformation on the target object in each video frame of the intermediate video to obtain the target video.

[0055] The electronic device 110 can perform an affine transformation on the target object in each video frame of the reference video 113, adjust the angle of the target object to a preset standard angle, and obtain a transformed reference video 113-b. As an example, the standard angle can be 0°. If the tilt angle of the target object in the video frame is 0°, then the adjustment angle corresponding to the affine transformation is also 0°. If the tilt angle of the target object in the video frame is 5° to the left, then the angle corresponding to the affine transformation can be adjusted 5° to the right.

[0056] This process adjusts the spatial position of the target object through affine transformation to ensure that the angle of the target object in the transformed reference video 113-b is consistent with the preset standard angle, thereby providing standardized input for the subsequent target video generation process. As an example, during the affine transformation process, only a predetermined area of ​​the target object (such as the face, mouth, etc.) can be transformed to save computing power and improve efficiency.

[0057] Based on the transformed reference video 113-b, the electronic device 110 can extract the first video feature representation from it and provide more accurate feature information for the subsequent video generation step. Based on the transformed reference video 113-b, the electronic device 110 performs masking on the predetermined area of ​​the target object in the transformed reference video to obtain the masked video 301. Thus, the electronic device 110 will use the first video feature representation, the second video feature representation and the audio feature representation to generate an intermediate video, and the target object's lip shape and facial features adjusted by the affine transformation in the intermediate video match the target audio 114.

[0058] After generating the target video, the electronic device 110 can perform an inverse angle transformation on the target object in the intermediate video, adjust the angle of the target object back to the original tilt state, and ensure that the generated target video 104 is consistent with the posture of the target object in the original reference video 113. This process restores the target object in the target video 104 to the original video angle by applying an affine transformation, thereby ensuring that the final generated target video 104 can accurately reflect the lip shape and expression of the audio synchronization, and maintain the natural appearance and posture of the target object in the video.

[0059] Figure 5 A schematic diagram of a training process 500 of a video generation model according to some embodiments of the present disclosure is shown. Taking the training process performed by the electronic device 110 as an example, the training process of the video generation model 307 is introduced. Based on the first training sample, a first predicted video feature representation 506 is generated using the video generation model 307 to be trained. The first training sample may include a first video sample 501 of a first object sample, a first mask video sample 502, and a first audio sample 503 corresponding to the first video sample 501. The first mask video sample 502 is obtained by masking a predetermined area of ​​the first object sample in the first video sample 501. Based on the first predicted video feature representation 506, the video generation model 307 in the training process (which may also be referred to as to be trained) may generate a predicted video 509. The electronic device 110 may update the video generation model 307 based at least on the difference between the predicted video 509 and the first video sample 501.

[0060] During the training process, the electronic device 110 can perform a training process of the video generation model 307 based on the first training sample. The first training sample includes a first video sample 501 of a first object sample, a first mask video sample 502, and a first audio sample 503 corresponding to the first video sample 501. In addition, it can also include noise 505 and multiple mask map samples 504, and the multiple mask map samples 504 are obtained based on masking a predetermined area of ​​the first object sample in each video frame of the first video sample 501. The electronic device 110 can use the video generation model 307 to be trained, and generate a first predicted video feature representation 506 by inputting the first video sample 501, the first mask video sample 502, the first audio sample 503, the noise 505 and the multiple mask map samples 504 in the first training sample.

[0061] After generating the first predicted video feature representation 506, the U-Net model (U-Net) 307-a in the video generation model 307 can generate noise prediction results (Predicted noises) 507. These noise prediction results 507 can represent the noise part removed from the current latent variable, which is the key information used to restore the generated video. Based on the noise prediction results 507, the predicted video feature representation (Estimated clean latents) 508 can be obtained. The obtained predicted video feature representation 508 can be expressed as follows:

[0062]

[0063] in, The predicted video feature representation 508 may be represented. t It can represent the current latent variable, which represents the state after adding noise 505 through the forward diffusion process. ∈ θ (z t ) can represent the noise prediction result 507. It can represent the signal retention ratio during the diffusion process and the degree of information retention of the data during the diffusion process.

[0064] As an example, noise 505 may be represented as follows:

[0065]

[0066] Among them, ∈ shared We can denote shared noise, which is a global noise that is the same for all video frames. This part of noise ensures global consistency between video frames. It can represent the noise of a specific frame, which is the noise unique to each frame. Through this part of noise, the model can capture the unique changes of each frame without losing global consistency.

[0067] The predicted video feature representation 508 indicates the potential video features after the noise is removed. Through this process, the U-type network model 307-a can extract a latent variable representation close to the truth from the noise prediction. The predicted video feature representation 508 is processed by the video decoder model 307-b in the video generation model 307 to be trained, and the predicted video 509 can be decoded and generated. Next, the electronic device 110 can compare the difference between the predicted video 509 and the first video sample 501. Based on these differences, the electronic device 110 can update the parameters of the video generation model 307, thereby adjusting the model generation effect, gradually reducing the difference between the predicted video and the real video, and thus completing the training of the video generation model 307.

[0068] As an example, the electronic device 110 may update the parameters of the video generation model 307 in two stages. The first stage may be to compare the difference between the noise prediction result 507 and the noise 505, and update the parameters of the video generation model 307 based on the difference. The difference between the noise prediction result 507 and the noise 505 may be expressed as follows:

[0069]

[0070] in Can represent the expected operation, applied to video frame x, audio feature A, noise ∈, and time step t, its role is to calculate the average error of noise prediction. ∈ can represent noise 505. ∈ θ (z t ,t,τ θ (A)) can represent the noise prediction result 507, z t can represent the current latent variable, t can represent the time step, τ θ (A) can represent the noise feature representation corresponding to the noise prediction result 507.

[0071] The second stage may include comparing the time domain feature representation difference 511 and the perceptual spatial feature difference 512 between the predicted video 509 and the first video sample 501, and comparing the time synchronization difference 513 between the predicted video 509 and the first audio sample 503. The parameters of the video generation model 307 are updated according to the above differences. The specific comparison process of the second stage will be described in detail later.

[0072] By repeatedly executing this process, the video generation model 307 gradually learns how to accurately generate video outputs that match the video samples according to the audio features during the training process. This process not only enables the video generation model to better understand the synchronization relationship between audio and video, but also optimizes the generation capability of the model, improves the generation quality of the video, and ensures the synchronization between the target object's lip shape and the target audio in the target video.

[0073] The training process of the second stage is described below. The electronic device 110 uses the trained synchronization network (SyncNet) to determine the time synchronization difference 513 between the predicted video 509 and the first audio sample 503. The video generation model 307 is updated based on the time synchronization difference 513.

[0074] Fig. 6AA schematic diagram of a process 600A for determining a time synchronization difference according to some embodiments of the present disclosure is shown. The synchronization network 601 can be used to evaluate the synchronization between each video frame of the predicted video 509 and the first audio sample 503, and calculate the time synchronization difference 513 between them by analyzing the feature representations of each video frame of the predicted video 509 and the first audio sample 503. The synchronization evaluation may include determining whether the video frame of the predicted video 509 matches the content of the first audio sample 503 in terms of lip shape, etc. The time synchronization difference 513 can be expressed as follows:

[0075]

[0076] in, The desired operation can be represented by averaging the audio-video pairs of all training samples (i.e., the video frame x of the evaluated prediction video 509, the audio feature a of the first audio sample 503, the noise ∈, and the time step t) to calculate the time synchronization loss. It can represent a video frame sequence of the predicted video 509 (in pixel dimension) obtained based on the predicted video feature representation 508 (f:f+16 can be used as a time window of the frame sequence, usually 16 consecutive frames). f:f+16 A sequence of audio frames corresponding to a sequence of video frames may be represented.

[0077] According to the determined time synchronization difference 513, the electronic device 110 can update the parameters of the video generation model based on the difference. Through this process, the video generation model 307 can optimize the generation effect, optimize the synchronization between the predicted video 509 and the first audio sample 503, and reduce the time error between the audio and video. Through continuous feedback and optimization, the video generation model 307 will gradually improve in each training stage, thereby achieving more accurate and natural audio and video synchronization.

[0078] Regarding the time synchronization difference 513, the electronic device 110 may also extract a second predicted video feature representation of the predicted video 509. An audio feature representation of the first target audio sample 503 is determined. The time synchronization difference 513 is determined based on the second predicted video feature representation and the audio feature representation of the first target audio sample 503 using the trained synchronization network 601.

[0079] Based on the predicted video 509, the electronic device 110 can determine a second predicted video feature representation corresponding to the predicted video 509. The second predicted video feature representation can be a high-dimensional feature extracted from the predicted video 509, covering the structure, texture and other visual information related to the target audio synchronization of each video frame. In addition, the electronic device 110 can also determine the audio feature representation of the first target audio sample 503, which includes the time-frequency features in the first target audio sample 503, especially the key information such as the rhythm, pitch and duration of the audio.

[0080] After determining the second predicted video feature representation and the audio feature representation of the first target audio sample 503, the electronic device 110 can use the trained synchronization network 601 to determine the time synchronization difference 513. In this way, the electronic device 110 can accurately determine the time synchronization difference 513 between the predicted video 509 and the first target audio sample 503 to improve the synchronization of the generated target video and the target audio.

[0081] Since the second predicted video feature representation abstracts and compresses the key content of the predicted video 509 (reduces redundant information and noise), the alignment with the audio feature is more direct and effective. The feature space can better capture the high-level semantic information of the video (such as the mouth shape, facial expressions, etc. of the characters), which is directly related to the audio features (such as pronunciation, intonation, etc.) and can provide a more accurate synchronization signal.

[0082] The training process of the synchronization network 601 is described below. Figure 6B A schematic diagram of a training process 600B of a synchronization network 601 according to some embodiments of the present disclosure is shown. The electronic device 110 determines a time synchronization prediction result between a second video frame sample 602 and a second audio sample 604 using a synchronization network to be trained based on a second training sample, wherein the second training sample includes a video feature representation of the second video frame sample 602 and an audio feature representation of the second audio sample. The synchronization network 601 is trained based on the difference between the time synchronization prediction result and a true time synchronization result annotated for the second training sample, wherein the true time synchronization result indicates the degree of audio and video synchronization between the second video frame sample 602 and the second audio sample 604.

[0083] The second training sample may include a video feature representation 603 of a second video frame sample containing a second object and an audio feature representation of a second audio sample 604. The video feature representation 603 of the second video frame sample contains key visual information in the second video frame sample 602. The audio feature representation of the second audio sample 604 is generally expressed as a mel-spectrogram, reflecting the time-frequency characteristics of the audio.

[0084] The electronic device 110 may use the synchronization network to be trained 601 to determine a time synchronization prediction result between the second video frame sample 602 and the second audio sample 604. The synchronization network to be trained 601 may output a prediction result indicating whether the video frame and the corresponding audio are synchronized based on the association between the video frame and the audio features according to the video feature representation 603 of the input second video frame sample and the audio feature representation of the second audio sample 604.

[0085] During the training process, the electronic device 110 may compare the time synchronization prediction result with the true time synchronization result annotated by the second training sample. The true time synchronization result indicates the actual degree of synchronization between the second video frame sample and the second audio sample. By comparing the difference between the prediction result and the annotated result, the electronic device 110 may determine the synchronization loss and train the synchronization network 601 based on the synchronization loss. The goal of the training is to minimize the difference between the prediction result and the true value, thereby improving the audio and video synchronization accuracy of the synchronization network 601 in future tasks.

[0086] In some embodiments of the present disclosure, the electronic device 110 may also determine a first temporal feature representation between a plurality of consecutive video frames in the first video sample 501. Determine a second temporal feature representation between a plurality of consecutive predicted video frames in the predicted video 509. The electronic device 110 may also update the video generation model 307 based on the difference between the first temporal feature representation and the second temporal feature representation.

[0087] The electronic device 110 may determine a first temporal feature representation between a plurality of consecutive video frames in the first video sample 501. These temporal feature representations may indicate a temporal change pattern between video frames, that is, a temporal relationship of the video. Next, the electronic device 110 may determine a second temporal feature representation between a plurality of consecutive predicted video frames in the predicted video 509, which is used to indicate a temporal relationship between frames in the generated predicted video 509.

[0088] The electronic device 110 may determine a difference between the first temporal feature representation and the second temporal feature representation, which may be used as a temporal feature representation difference 511. The electronic device 110 may enhance the temporal consistency between video frames by determining the temporal feature representation difference 511, thereby ensuring that the generated video sequence can more accurately reflect temporal changes and avoid unnatural timing mismatches in the generated video. The temporal feature representation difference 511 may be represented as follows:

[0089]

[0090] in, The desired operation can be expressed as indicating that the video pairs (video frame x of the first video sample 501, noise ∈ and time step t) of all training samples are averaged to calculate the time synchronization loss. The second time domain feature representation obtained by performing time domain feature analysis on the video frame sequence (in pixel dimension) obtained based on the predicted video feature representation 508 may be represented. It can represent a first time domain feature representation obtained by performing time domain feature analysis on multiple consecutive video frames of the first video sample 501 corresponding to the video frame sequence.

[0091] The electronic device 110 measures the accuracy of the generated video in the time dimension by determining the difference between the first time domain feature representation and the second time domain feature representation. The determination of this difference helps the temporal consistency and visual coherence of the video generation model 307. Through the above process, the video generated by the trained video generation model 307 is not only consistent with the input video in the content of each frame, but also can be better aligned in the time domain, and finally achieves a more natural and realistic video generation effect.

[0092] In some embodiments of the present disclosure, the electronic device 110 may further select a target video frame sample that corresponds to the predicted video frame in time sequence from the first video sample 501 for the predicted video frame in the predicted video 509. Determine the perceived spatial feature difference 512 between the predicted video frame and the target video frame sample. The electronic device 110 may further update the video generation model 307 based on the perceived spatial feature difference 512.

[0093] For the predicted video frame in the predicted video 509, the electronic device 110 may select a target video frame sample that corresponds to the predicted video frame in time sequence from the first video sample 501. Next, the electronic device 110 may determine a perceptual spatial feature difference 512 between the predicted video frame and the target video frame sample. The perceptual spatial feature difference 512 may be expressed as follows:

[0094]

[0095] in, The desired operation can be expressed as indicating that the video pairs of all training samples (the video frame x of the first video sample 501, the noise ∈ and the time step t) are averaged to calculate the perceptual spatial feature difference. It can represent the result of feature extraction of the predicted video frame (in pixel dimension) obtained based on the predicted video feature representation 508 using the trained VGG network. It can represent the result of feature extraction of target video frame samples using the trained VGG network.

[0096] The electronic device 110 may update the video generation model 307 based on the perceived spatial feature differences. This process minimizes the perceived spatial feature differences to ensure that the generated video is closer to the target video in terms of perceived quality, thereby improving the visual effect and accuracy of the video generation model.

[0097] In combination with the foregoing, the update of the video generation model 307 can be based on the difference between the noise prediction result 507 and the noise 505 Time synchronization difference L sync , time domain feature representation difference and perceived spatial feature differences For each difference, a corresponding weight λ can be configured. Based on this, the updated video generation model 307 can be expressed as follows:

[0098]

[0099] Among them, λ1, λ2, λ3 and λ4 can correspond to different weights respectively.

[0100] Figure 7 A schematic structural block diagram of an apparatus 700 for video generation according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented in or included in the electronic device 110, for example. Each module / component in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0101] like Figure 7 As shown, the device 700 may include a mask video determination module 701, which is configured to obtain a mask video by performing masking on a predetermined area of ​​the target object in a reference video. A video feature representation determination module 702 is configured to respectively determine a first video feature representation of the reference video and a second video feature representation of the mask video. An audio feature representation determination module 703 is configured to determine an audio feature representation of the target audio. A target video generation module 704 is configured to generate a target video containing the target object using a trained video generation model based at least on the first video feature representation, the second video feature representation, and the audio feature representation, wherein the target video represents that the target object speaks the target audio with a mouth shape that matches the target audio.

[0102] In some embodiments of the present disclosure, masking includes masking a predetermined area of ​​a target object in each video frame of a reference video using a plurality of mask images, and the apparatus 700 may further include a feature extraction module. The feature extraction module may be configured to determine mask feature representations of the plurality of mask images. The target video generation module 704 may further be specifically configured to generate a target video containing the target object using a video generation model based on the mask feature representation.

[0103] In some embodiments of the present disclosure, the feature extraction module may be specifically configured to: perform angle transformation on the target object in each video frame of the reference video to obtain a transformed reference video. Determine the first video feature representation from the transformed reference video,

[0104] In some embodiments of the present disclosure, the target video generation module 704 may be specifically configured to: generate an intermediate video based on at least the first video feature representation, the second video feature representation, and the audio feature representation, and perform an inverse angle transformation on the target object in each video frame of the intermediate video to obtain the target video.

[0105] In some embodiments of the present disclosure, the mask video determination module 701 may be specifically configured to: perform masking on a predetermined area of ​​the target object in the transformed reference video to obtain a mask video.

[0106] In some embodiments of the present disclosure, the device 700 may further include a model training module. The model training module may be configured to: generate a first predicted video feature representation based on a first training sample using a video generation model to be trained, wherein the first training sample includes a first video sample of a first object sample, a first mask video sample, and a first audio sample corresponding to the first video sample, wherein the first mask video sample is obtained by masking a predetermined area of ​​the first object sample in the first video sample. Generate a predicted video based on the first predicted video feature representation. Update the video generation model based on at least the difference between the predicted video and the first video sample.

[0107] In some embodiments of the present disclosure, the model training module may be configured to: determine the time synchronization difference between the predicted video and the first audio sample using a trained synchronization network, and update the video generation model based on the time synchronization difference.

[0108] In some embodiments of the present disclosure, the model training module may be configured to: extract a second predicted video feature representation of the predicted video, determine an audio feature representation of the first target audio sample, and determine a time synchronization difference based on the second predicted video feature representation and the audio feature representation of the first target audio sample using a trained synchronization network.

[0109] In some embodiments of the present disclosure, the model training module may be configured to: determine a time synchronization prediction result between a second video frame sample and a second audio sample using a synchronization network to be trained based on a second training sample, wherein the second training sample includes a video feature representation of the second video frame sample and an audio feature representation of the second audio sample. Based on the difference between the time synchronization prediction result and a true time synchronization result annotated for the second training sample, the synchronization network is trained, and the true time synchronization result indicates the degree of audio and video synchronization between the second video frame sample and the second audio sample.

[0110] In some embodiments of the present disclosure, the model training module may be configured to: determine a first temporal feature representation between a plurality of consecutive video frames in a first video sample, determine a second temporal feature representation between a plurality of consecutive predicted video frames in a predicted video, and update the video generation model based on a difference between the first temporal feature representation and the second temporal feature representation.

[0111] In some embodiments of the present disclosure, the model training module may be configured to: for a predicted video frame in a predicted video, select a target video frame sample that corresponds to the predicted video frame in time sequence from the first video sample, determine the difference in perceptual spatial features between the predicted video frame and the target video frame sample, and update the video generation model based on the difference in perceptual spatial features.

[0112] In some embodiments of the present disclosure, the predetermined area at least includes a mouth of the target object.

[0113] Figure 8 8 is a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown may include or be implemented as Figure 1 Electronic device 110 or Figure 7 Device 700.

[0114] like Figure 8 As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0115] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.

[0116] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0117] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0118] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0119] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0120] According to an exemplary implementation of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 2 The methods provided in the various optional methods are therefore not described in detail here.

[0121] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0123] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0124] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0125] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for video generation, comprising: A masked video is obtained by performing masking on a predetermined area of ​​the target object in a reference video; Determining a first video feature representation of the reference video and a second video feature representation of the mask video respectively; determining an audio feature representation of the target audio; as well as Based at least on the first video feature representation, the second video feature representation and the audio feature representation, a target video containing the target object is generated using a trained video generation model, wherein the target video represents that the target object speaks the target audio with lip shapes that match the target audio.

2. The method according to claim 1, wherein the masking comprises masking the predetermined area of ​​the target object in each video frame of the reference video using a plurality of mask images, and the method further comprises: determining mask feature representations of the plurality of mask maps; and The generating of the target video including the target object includes: also based on the mask feature representation, generating the target video including the target object using the video generation model.

3. The method of claim 1 , wherein determining a first video feature representation of the reference video frame comprises: Performing angle transformation on the target object in each video frame of the reference video to obtain a transformed reference video; as well as determining the first video feature representation from the transformed reference video, The generating of the target video comprises: generating an intermediate video based at least on the first video feature representation, the second video feature representation, and the audio feature representation; as well as An inverse angle transformation is performed on the target object in each video frame of the intermediate video to obtain the target video.

4. The method according to claim 3, wherein obtaining the mask video comprises: Masking is performed on the predetermined area of ​​the target object in the transformed reference video to obtain the masked video.

5. The method according to claim 1, wherein the training of the video generation model comprises: Based on a first training sample, generating a first predicted video feature representation using the video generation model to be trained, wherein the first training sample includes a first video sample of a first object sample, a first mask video sample, and a first audio sample corresponding to the first video sample, wherein the first mask video sample is obtained by performing masking on a predetermined area of ​​the first object sample in the first video sample; generating a predicted video based on the first predicted video feature representation; as well as The video generation model is updated based at least on a difference between the predicted video and the first video sample.

6. The method of claim 5, wherein updating the video generation model comprises: Determining, using a trained synchronization network, a time synchronization difference between the predicted video and the first audio sample; as well as The video generation model is updated based on the time synchronization difference.

7. The method of claim 6, wherein determining a time synchronization difference between the predicted video and the first audio sample comprises: Extracting a second predicted video feature representation of the predicted video; determining an audio feature representation of the first target audio sample; as well as The temporal synchronization difference is determined based on the second predicted video feature representation and the audio feature representation of the first target audio sample using a trained synchronization network.

8. The method of claim 6, wherein the synchronization network is trained by: Determining a time synchronization prediction result between a second video frame sample and a second audio sample using a synchronization network to be trained based on a second training sample, wherein the second training sample includes a video feature representation of the second video frame sample and an audio feature representation of the second audio sample; and The synchronization network is trained based on a difference between the time synchronization prediction result and a true time synchronization result annotated for the second training sample, wherein the true time synchronization result indicates a degree of audio and video synchronization between the second video frame sample and the second audio sample.

9. The method according to claim 5, wherein updating the video generation model further comprises: determining a first temporal feature representation between a plurality of consecutive video frames in the first video sample; Determining a second temporal feature representation between a plurality of consecutive predicted video frames in the predicted video; as well as The video generation model is also updated based on the difference between the first time-domain feature representation and the second time-domain feature representation.

10. The method of claim 5, wherein updating the video generation model comprises: For a predicted video frame in the predicted video, selecting a target video frame sample corresponding to the predicted video frame in time sequence from the first video samples; determining a difference in perceptual spatial features between the predicted video frame and the target video frame samples; and The video generation model is also updated based on the perceived spatial feature differences. The method according to claim 1 , wherein the predetermined area includes at least a mouth of the target object.

12. A device for video generation, comprising: a mask video determination module, configured to obtain a mask video by performing masking on a predetermined area of ​​the target object in a reference video; a video feature representation determination module, configured to respectively determine a first video feature representation of the reference video and a second video feature representation of the mask video; An audio feature representation determination module, configured to determine an audio feature representation of a target audio; as well as The target video generation module is configured to generate a target video containing the target object using a trained video generation model based at least on the first video feature representation, the second video feature representation and the audio feature representation, wherein the target video represents that the target object speaks the target audio with lip shapes that match the target audio.

13. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product, comprising computer executable instructions, wherein when the computer executable instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images