Video generation method, training method and device of video generation model, equipment, storage medium and program product

Abstract

The invention provides a video generation method, a training method and device of a video generation model, equipment, a storage medium and a program product. The method comprises the steps that detection frames are recognized from multiple video frames of a first video, and the detection frame in each video frame surrounds the face of a target object in the video frame; determining a surrounding frame based on the detection frames of the plurality of video frames, the surrounding frame surrounding the face of the target object in the plurality of video frames; a mask area of each video frame is determined, and the mask area of each video frame surrounds the mouth of the target object in the video frame; based on the mask region, performing mask processing on a region in the surrounding frame in the video frame to obtain a mask video frame; the mask video frames form a mask video; a second video is generated based on the mask video, the first video and the audio data, and in the second video, the mouth action of the target object is synchronously matched with the audio data. According to the invention, the dubbing quality of visual dubbing can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, a training method, device, equipment, storage medium and program product of a video generation model. Background Art

[0002] Visual dubbing technology is a technology that uses artificial intelligence and computer vision technology to synchronize sound with the mouth shape, expression and other actions of the characters in the video. Visual dubbing technology usually involves multiple fields such as facial capture, speech synthesis, and image processing. It aims to make the dubbing video more natural and realistic, especially in the fields of animation production, film post-production, and virtual reality. Related technologies The videos generated by visual dubbing technology have the problem of inconsistency between audio and lips, resulting in differences between the generated videos and the videos with real people's pronunciation, lack of authenticity, and affect user perception. Summary of the invention

[0003] The embodiments of the present application provide a video generation method, a training method, an apparatus, a device, a storage medium and a program product of a video generation model, which can improve the dubbing quality of visual dubbing.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a video generation method, the method comprising:

[0006] identifying a detection frame from a plurality of video frames of a first video, wherein the detection frame in each of the video frames surrounds a face of a target object in the video frame;

[0007] Determine a bounding box based on the detection boxes of the multiple video frames, wherein the bounding box surrounds the face of the target object in the multiple video frames;

[0008] Determining a mask region of each of the video frames, wherein the mask region of each of the video frames surrounds a mouth of the target object in the video frame;

[0009] Based on the mask area, mask processing is performed on the area in the video frame that is within the bounding box to obtain a masked video frame; the masked video frame constitutes a masked video;

[0010] A second video is generated based on the mask video, the first video and the audio data, wherein the mouth movement of the target object in the second video is synchronously matched with the audio data.

[0011] In the above solution, identifying the detection frame from the multiple video frames of the first video includes:

[0012] Dividing the first video into a plurality of video segments, wherein each of the video segments includes a plurality of video frames;

[0013] The following processing is performed for each of the video segments:

[0014] Performing target detection processing on each of the video frames to obtain a detection frame in each of the video frames, wherein the detection frame includes the face of the target object in the video frame;

[0015] The determining of the bounding box based on the detection boxes of the multiple video frames includes:

[0016] The following processing is performed for each of the video segments:

[0017] A union of the detection boxes in each of the video frames is determined, and the union is used as the bounding box.

[0018] In the above solution, determining the mask area of ​​each video frame includes:

[0019] The following processing is performed for each of the video segments:

[0020] Taking the portion of the detection frame of each video frame including the mouth as a candidate mask frame, and taking the size of the candidate mask frame with the largest area as the target size;

[0021] The size of the candidate mask frame of each of the video frames in the video clip is set to the target size to obtain a target mask frame of each of the video frames, and the area surrounded by the target mask frame is used as the mask area.

[0022] In the above solution, performing mask processing on the area in the video frame located within the bounding box based on the mask area to obtain a masked video frame includes:

[0023] The following processing is performed on each of the video segments:

[0024] Performing edge blurring processing on the mask area of ​​each video frame to obtain a blurred mask area;

[0025] Setting the pixel values ​​in the fuzzy mask area to a preset value to obtain a masked video frame;

[0026] The mask video frame constitutes a mask video, including:

[0027] combining the masked video frames of each of the video frames into a masked segment;

[0028] The mask segments of each of the video segments are combined into the mask video.

[0029] In the above solution, the decoding is performed at least once based on the at least one first reference image feature to obtain at least one second reference image feature, including:

[0030] Extracting a second spatial feature based on the input of the current decoding to obtain a second spatial feature, wherein when the current decoding is the first decoding, the input of the current decoding is a first reference image feature outputted by the last encoding, and when the current decoding is not the first decoding, the input of the current decoding is a second reference image feature outputted by the last decoding;

[0031] A second attention processing is performed based on the second spatial feature and the image coding feature to obtain a second reference image feature of the current decoding output.

[0032] In the above solution, performing at least one decoding based on the at least one encoding time feature to obtain at least one decoding time feature includes:

[0033] Performing a fourth spatial feature extraction based on the input of the current decoding to obtain a fourth spatial feature, wherein when the current decoding is the first decoding, the input of the current encoding is the encoding time feature of the last encoding output, and when the current decoding is not the first decoding, the input of the current decoding is the decoding time feature of the last decoding output;

[0034] performing a fifth attention process on the fourth spatial feature and the second reference image feature to obtain a third attention feature;

[0035] Performing a sixth attention process on the third attention feature and the audio feature to obtain a fourth attention feature;

[0036] Perform a second time feature extraction on the fourth attention feature to obtain a decoding time feature of the current decoding output.

[0037] The present application embodiment provides a method for training a video generation model, the method comprising:

[0038] Identify a first detection box sample from a plurality of video frames of a first video sample, wherein the first detection box sample in each of the video frames surrounds a face of a first target object sample in the video frame;

[0039] Determining a first bounding box sample based on the first detection box samples of the multiple video frames, wherein the first bounding box samples all surround the face of the first target object sample in the multiple video frames;

[0040] Determine a first mask area sample of each of the video frames, wherein the first mask area sample of each of the video frames surrounds a mouth of the first target object sample in the video frame;

[0041] Based on the first mask area sample, mask processing is performed on the area in the video frame that is located within the first bounding box sample to obtain a masked video frame sample; the masked video frame sample constitutes a masked video sample;

[0042] generating a second video sample based on the mask video sample, the first video sample and the audio data sample, wherein the mouth movement of the first target object sample in the second video sample is synchronously matched with the audio data sample;

[0043] determining a first loss value based on the second video sample;

[0044] The first video generation model is updated based on the first loss value to obtain a second video generation model.

[0045] In the above solution, determining the second loss value based on the second image sample includes:

[0046] Determine a second predicted noise sample corresponding to the second image sample, wherein the second image sample is obtained by denoising the noisy mask image sample based on the second predicted noise sample, the second predicted noise sample is obtained by performing noise prediction processing on the noisy mask image sample and a second reference image sample, the noisy mask image sample is obtained by performing noise processing on the mask image sample and the first image sample, and the noise processing is implemented based on the second noise data sample;

[0047] determining a third sub-loss based on a difference between the second predicted noise sample and the second noise data sample;

[0048] Obtaining a product of the third sub-loss and the mask image sample as a fourth sub-loss;

[0049] The third sub-loss and the fourth sub-loss are weightedly summed according to a preset weight to obtain the second loss value.

[0050] The embodiment of the present application provides a video generation device, including:

[0051] A data processing module, configured to identify a detection frame from a plurality of video frames of a first video, wherein the detection frame in each of the video frames surrounds a face of a target object in the video frame;

[0052] The data processing module is further used to determine a bounding box based on the detection boxes of the multiple video frames, wherein the bounding box surrounds the face of the target object in the multiple video frames;

[0053] The data processing module is further used to determine a mask area of ​​each of the video frames, wherein the mask area of ​​each of the video frames surrounds the mouth of the target object in the video frame;

[0054] The data processing module is further used to perform mask processing on the area in the video frame located within the bounding box based on the mask area to obtain a masked video frame; the masked video frame constitutes a masked video;

[0055] A generation module is used to generate a second video based on the mask video, the first video and the audio data, wherein the mouth movement of the target object in the second video is synchronously matched with the audio data.

[0056] The present application embodiment provides a training device for a video generation model, including:

[0057] A data processing module, configured to identify detection frame samples from a plurality of video frames of a first video sample, wherein the detection frame sample in each of the video frames surrounds a face of a first target object sample in the video frame;

[0058] The data processing module is further used to determine a first bounding box sample based on the detection box samples of the multiple video frames, wherein the first bounding box sample surrounds the face of the first target object sample in the multiple video frames;

[0059] The data processing module is further used to determine a first mask area sample of each of the video frames, wherein the first mask area sample of each of the video frames surrounds the mouth of the first target object sample in the video frame;

[0060] The data processing module is further used to perform mask processing on the area in the video frame located within the first bounding box sample based on the first mask area sample, based on the mask video frame sample; the mask video frame sample constitutes a mask video sample;

[0061] A generating module, configured to generate a second video sample based on the mask video sample, the first video sample and the audio data sample, wherein the mouth movement of the first target object sample in the second video sample is synchronously matched with the audio data sample;

[0062] A training module, configured to determine a first loss value based on the second video sample;

[0063] The training module is also used to update the first video generation model based on the first loss value to obtain a second video generation model.

[0064] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0065] A memory for storing computer executable instructions or computer programs;

[0066] The processor is used to implement the video generation method provided in the embodiment of the present application, or implement the training method of the video generation model provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0067] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement the video generation method provided in the embodiment of the present application when executed by a processor, or to implement the training method of the video generation model provided in the embodiment of the present application.

[0068] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the video generation method provided in the embodiment of the present application is implemented, or the training method of the video generation model provided in the embodiment of the present application is implemented.

[0069] The embodiments of the present application have the following beneficial effects:

[0070] Based on the detection frames of multiple video frames, a bounding box that surrounds the face of the target object in multiple video frames is determined, so that one bounding box can be reused for mask processing of multiple video frames, that is, in the facial area enclosed by the bounding box in each video frame, the mask area corresponding to the mouth is masked. Since the position of the bounding box in multiple video frames is fixed, it is achieved to generate a mask video frame based on a fixed area in each video frame of the first video. Compared with the method of using a non-fixed bounding box in multiple video frames, the discontinuity or jump of facial information in different mask video frames due to the jump of the size or position of the bounding box is avoided, the continuity of facial information between mask video frames is improved, and then the continuity of facial information in the second video generated based on the mask video frame is improved, so that the user has a real and natural perception effect when watching the second video. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a structural diagram of the video generation system architecture provided by an embodiment of the present application;

[0072] Figure 2A is a first structural diagram of a server provided in an embodiment of the present application;

[0073] Figure 2B is a second structural diagram of a server provided in an embodiment of the present application;

[0074] Figure 3A It is a first flow chart of the video generation method provided in an embodiment of the present application;

[0075] Figure 3B is a second flow chart of the video generation method provided in an embodiment of the present application;

[0076] Figure 3C is a third flow chart of the video generation method provided in an embodiment of the present application;

[0077] Figure 3D is a fourth flow chart of the video generation method provided in an embodiment of the present application;

[0078] Figure 3E is a fifth flow chart of the video generation method provided in an embodiment of the present application;

[0079] Figure 3F is a sixth flow chart of the video generation method provided in an embodiment of the present application;

[0080] Figure 3G is a seventh flow chart of the video generation method provided in an embodiment of the present application;

[0081] Figure 3H is an eighth flow chart of the video generation method provided in an embodiment of the present application;

[0082] Fig. 3I is a ninth flow chart of the video generation method provided in an embodiment of the present application;

[0083] Figure 3J is a tenth flow chart of a video generation method provided in an embodiment of the present application;

[0084] Figure 3K This is a schematic diagram of the eleventh process of the video generation method provided in the embodiment of the present application;

[0085] Figure 3L is a twelfth flow chart of the video generation method provided in an embodiment of the present application;

[0086] Figure 3M is a thirteenth flow chart of the video generation method provided in an embodiment of the present application;

[0087] Figure 3N It is a fourteenth flow chart of the video generation method provided in an embodiment of the present application;

[0088] Fig.3Ois a fifteenth flow chart of the video generation method provided in an embodiment of the present application;

[0089] Figure 4A It is a first flow chart of the training method of the video generation model provided in the embodiment of the present application;

[0090] Figure 4B It is a second flow chart of the training method of the video generation model provided in the embodiment of the present application;

[0091] Figure 4C It is a third flow chart of the training method of the video generation model provided in the embodiment of the present application;

[0092] Figure 4D It is a fourth flow chart of the training method of the video generation model provided in the embodiment of the present application;

[0093] Figure 5 It is a flowchart of a method for generating a video in a movie dubbing scenario provided by an embodiment of the present application;

[0094] Fig. 6A is a first schematic diagram of an optional structure of a video generation model provided in an embodiment of the present application;

[0095] Figure 6B is a second schematic diagram of an optional structure of a video generation model provided in an embodiment of the present application;

[0096] Fig. 7A is a schematic diagram of a detection frame provided in an embodiment of the present application;

[0097] Figure 7B is a schematic diagram of a bounding box provided in an embodiment of the present application;

[0098] Figure 7C is a schematic diagram of a bounding box and a mask area provided in an embodiment of the present application;

[0099] Fig. 8A It is a schematic diagram of the first effect of the video generation method provided in an embodiment of the present application;

[0100] Figure 8B is a schematic diagram of a second effect of the video generation method provided in an embodiment of the present application;

[0101] Fig. 9A is a schematic diagram of a third effect of the video generation method provided in an embodiment of the present application;

[0102] Fig. 9B 4 is a schematic diagram of a fourth effect of the video generation method provided in an embodiment of the present application;

[0103] Fig. 10Ais a fifth effect schematic diagram of the video generation method provided in an embodiment of the present application;

[0104] Fig. 10B It is a schematic diagram of the sixth effect of the video generation method provided in an embodiment of the present application.

[0105] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0106] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0107] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0108] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0109] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0110] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0111] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0112] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0113] 1) Transformer refers to a time series model based on the self-attention mechanism. The encoder part can effectively encode time series information. Its processing ability of time series information is much better than that of the long short-term memory network (LSTM), and its speed is fast. It is widely used in natural language processing, computer vision, machine translation, speech recognition and other fields.

[0114] 2) State space model (Structured Space Model, SSM), is a mathematical model that describes the behavior of a dynamic system. It uses a set of first-order differential equations (continuous time system) or difference equations (discrete time system) to represent the evolution of the internal state of the system, and another set of equations to describe the relationship between the system state and output. These equations can be expressed in the form of matrices and vectors to handle multivariable systems.

[0115] 3) Mamba (Linear-Time Sequence Modeling with Selective StateSpaces) is a selective structural state space model that performs well in long sequence modeling tasks. Mamba alleviates the modeling constraints of convolutional neural networks through global receptive fields and dynamic weighting, and provides advanced modeling capabilities similar to Transformers.

[0116] 4) Digital human visual dubbing refers to changing the movement of the lower half of a given portrait video, including the chin and lips, according to a voice, to make it consistent with the content of the voice. Since this technology can generate realistic portrait videos, digital human visual dubbing is gaining more and more attention and has the potential to be widely used in scenarios such as film video dubbing, game character driving, and digital human live broadcasting.

[0117] 5) Diffusion model is a generative model based on the thermodynamic random diffusion process. This process includes a forward process of gradually adding noise to the sample from the data distribution, and a backward process of training the neural network to reverse the forward process by gradually removing the noise. In recent years, diffusion models have been widely used in various tasks such as image generation and natural language generation.

[0118] 6) Bounding Box: It is a rectangular box used to calibrate the face of the target object in the image in the target detection task. The detection box can be defined by four values, which represent the position and size of the detection box. For example, the detection box can be expressed as (x, y, width, height), where (x, y) is the coordinate of the upper left corner of the detection box, width is the width of the detection box, and height is the height of the detection box.

[0119] 7) The bounding box is obtained by taking the union of the detection boxes in multiple video frames, that is, the position of the bounding box in multiple video frames is fixed, so that the facial area in the video frame can be stably represented.

[0120] 8) Masking Region refers to the area in the image used to cover or protect certain parts (such as the mouth). The mask can be a binary image. For example, the pixel values ​​of the mask area can be uniformly set to 1, indicating that this part of the area is the area that needs to be processed or needs to be focused on.

[0121] Visual dubbing in related technologies is achieved by inserting a temporal layer into a pre-trained hidden layer diffusion two-dimensional image generation network and fine-tuning it on a high-quality video dataset. This series of methods usually predicts the intermediate representation of the image in a low-dimensional hidden layer, and then obtains the image through a decoder that retains the position relationship. This approach can effectively alleviate the consumption of computing resources in image generation. The approach of inserting a temporal layer can make full use of the powerful generalization ability of the two-dimensional image generation network to ensure the authenticity of the generated single-frame image. The temporal layer mainly captures the continuity of the image across frames.

[0122] In the field of visual dubbing, there are three key aspects that affect the quality of the generated videos:

[0123] (1) Visual quality, including image fidelity and video temporal consistency;

[0124] (2) audio-lip synchronization, that is, whether the generated mouth movements match the audio;

[0125] (3) Generalization capability refers to the ability to generalize to out-of-domain faces, audio, and complex scenes (especially when the mouth is occluded).

[0126] With the rapid development of text-to-image and video generation, more and more methods adopt end-to-end methods to complete video generation tasks. In order to improve visual quality, generalization ability and lip synchronization, special input designs and additional computationally intensive iterations are introduced in related technologies. These operations make them difficult to reproduce or cause a lot of additional training consumption, which hinders the generalization of complex training datasets and makes it difficult to ensure the above three key aspects of video generation in a simple and efficient framework.

[0127] Embodiments of the present application provide a video generation method, a video generation model training method, an apparatus, a device, a computer-readable storage medium, and a computer program product, which can improve the dubbing quality of visual dubbing.

[0128] The following describes exemplary applications of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as various types of terminal devices such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers.

[0129] See also Figure 1 , Figure 1 is a structural diagram of the video generation system architecture provided in an embodiment of the present application, Figure 1 The server 100, the terminal device 200 and the network 300 are involved. The terminal device 200 is connected to the server 100 via the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0130] In some embodiments, the embodiments of the present application can be implemented by a terminal device alone. For example, the terminal device 200 sends a request to the server 100, the server 100 receives the request, sends the second video generation model used to perform the video generation method provided in the embodiments of the present application to the terminal device 200, the terminal device 200 receives the second video generation model sent by the server and downloads it locally, and obtains the second video corresponding to the first video through the second video generation model.

[0131] In other embodiments, the embodiments of the present application can be implemented by the server and the terminal device in collaboration. For example, the server 100 obtains the second video generation model through the training method of the video generation model provided in the embodiment of the present application, the terminal device 200 sends the first video and audio data to the server 100, the server 100 obtains the second video through the video generation method provided in the embodiment of the present application, and sends the second video to the terminal device 200 for display.

[0132] Here, the server 100 may be a single server. In this case, the video generation method and the video generation model training method provided in the embodiment of the present application may be implemented by the same server. The server 100 may also be a server cluster. In the case where the server 100 is a server cluster, the video generation method and the video generation model training method provided in the embodiment of the present application may be implemented by different servers, which is not limited in the embodiment of the present application.

[0133] In some embodiments, the terminal device or server can implement the video generation method or the training method of the video generation model provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0134] Take the server for video generation as an example, see Figure 2A , Figure 2A is a first structural diagram of a server provided in an embodiment of the present application, Figure 2A The server 100-1 shown includes: at least one processor 110-1, a memory 130-1 and at least one network interface 120-1. The various components in the server 100-1 are coupled together via a bus system 140-1. It is understood that the bus system 140-1 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140-1 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140-1 is not described in detail. Figure 2A Various buses are labeled as bus system 140 - 1 .

[0135] Processor 110-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0136] Memory 130-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 130-1 may optionally include one or more storage devices that are physically remote from processor 110-1.

[0137] The memory 130-1 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 130-1 described in the embodiment of the present application is intended to include any suitable type of memory.

[0138] In some embodiments, memory 130 - 1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0139] Operating system 131-1, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0140] A network communication module 132-1, used to reach other electronic devices via one or more (wired or wireless) network interfaces 120-1, exemplary network interfaces 120-1 include: Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB), etc.;

[0141] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2A The video generation device 133 stored in the memory 130-1 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data processing module 1331 and a generation module 1332. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0142] Take the server for training the video generation model as an example, see Figure 2B , Figure 2B is a second structural diagram of a server provided in an embodiment of the present application, Figure 2B The server 100-2 shown includes: at least one processor 110-2, a memory 130-2 and at least one network interface 120-2. The various components in the server 100-2 are coupled together via a bus system 140-2. It is understood that the bus system 140-2 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140-2 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140-2 is not described in detail. Figure 2BVarious buses are labeled as bus system 140-2. For the detailed description of processor 110-2 and memory 130-2, please refer to the above, which will not be repeated here.

[0143] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2B The video generation model training device 134 stored in the memory 130-2 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data processing module 1341, a generation module 1342 and a training module 1343. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0144] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generation method or the video generation model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0145] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the video generation method provided in the embodiment of the present application. Figure 3A , Figure 3A is a first flow chart of the video generation method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0146] In step 101, a detection frame is identified from a plurality of video frames of a first video, wherein the detection frame in each video frame surrounds a face of a target object in the video frame.

[0147] In some embodiments, target detection processing is performed on multiple video frames in the first video to obtain a detection frame in each video frame, wherein the detection frame includes the face of a target object. As an example, the target object may be a person or an animal.

[0148] For example, see Fig. 7A , Fig. 7A is a schematic diagram of a detection frame provided in an embodiment of the present application, Fig. 7A Detection frames 1 to 4 corresponding to video frames 1 to 4 are shown in the figure, each detection frame includes a face corresponding to the target, and the positions and sizes of the detection frames are different, wherein the size of the detection frame (such as 3×4) is the ratio between the width and height of the detection frame.

[0149] For example, each video frame in the first video is preprocessed (such as adjusting the size of the video frame, etc.), and target detection processing is performed on each preprocessed video frame through a preset target detection model (such as a regional convolutional neural network (RCNN), MediaPipe, etc.) to obtain a target detection result for each video frame, where the detection result includes a detection box, and the detection box includes the location information of the face of the target object (such as the center point coordinates, width, and height) and the confidence level, where the confidence level is used to indicate the accuracy of the model's prediction that the detection box includes the face of the target object.

[0150] Taking the training of the deep learning-based target detection model (You Only Look Once, YOLO) as an example, images containing object samples and corresponding object bounding box annotations can be collected to construct training data. In order to improve the generalization ability of the target detection model, the training data can be rotated, scaled, flipped and other data enhancement operations. Next, the target detection model is trained using a loss function with category cross entropy loss and bounding box regression loss, and the performance of the model is evaluated on the test set using indicators such as accuracy, recall, average precision, etc. Finally, the target detection model is adjusted according to the test results, such as adjusting the learning rate, adding new training data, etc., to further improve the detection performance of the target detection model.

[0151] Here, the embodiments of the present application do not limit the target detection model used for target detection processing.

[0152] In other embodiments, the first video is divided into multiple video segments, wherein each video segment includes multiple video frames, and the following processing is performed on each video segment: target detection processing is performed on each video frame to obtain a detection box in each video frame, wherein the detection box includes the face of the target object in the video frame.

[0153] For example, a first video is divided into multiple video segments according to a preset number of frames (such as 16 frames), wherein each video segment includes a preset number of video frames, and target detection processing is performed on each video frame in the video segment to obtain a detection box in each video frame. The specific implementation method of the target detection processing can be found in the above description.

[0154] It should be noted that dividing the first video into multiple video segments is a logical division, that is, it is only logically divided into multiple video segments, that is, the start and end frame numbers of each video segment are recorded, and there is no need to divide the first video into multiple files.

[0155] In step 102, a bounding box is determined based on detection boxes of multiple video frames, wherein the bounding box surrounds the face of the target object in the multiple video frames.

[0156] In some embodiments, a bounding box is determined based on detection boxes of multiple video frames, wherein the bounding box surrounds the face of the target object in all video frames of the multiple video frames or in a portion of video frames corresponding to one video clip.

[0157] In some embodiments, a union of detection boxes in each video frame is determined, and the union is used as a bounding box.

[0158] For example, the detection frames of multiple video frames of the first video are unioned, and the union of the multiple detection frames is used as the bounding box.

[0159] For example, see Figure 7B , Figure 7B is a schematic diagram of a bounding box provided in an embodiment of the present application. Following the above example, Fig. 7A The detection frames 1 to 4 corresponding to the video frames 1 to 4 shown in the figure are respectively combined to obtain Figure 7B The bounding box shown in .

[0160] In other embodiments, the first video is divided into multiple video segments, and the union of the detection boxes in each video frame in the video segment is used as the bounding box corresponding to the video segment. The division of the video segment can be referred to the description in step 101 above, which will not be repeated here.

[0161] For example, when the duration of the first video is greater than a threshold (such as 1 minute), the first video is divided into multiple video segments, and the union of the detection boxes in each video frame in the video segment is used as the bounding box corresponding to the video segment. This is because the probability of the position of the face changing in a longer video is relatively high. Therefore, after dividing the video segment into multiple video segments, the bounding box of each video segment is calculated separately to avoid a situation where the bounding box is large and the subsequent calculation complexity is increased.

[0162] For example, the method for obtaining the bounding box can be determined according to the type of the first video. For example, for a first video in which the facial position is basically unchanged (such as a live broadcast, a lecture, etc.), the detection boxes of multiple video frames of the first video are unioned, and the union of the multiple detection boxes is used as the bounding box. For a first video in which the probability of facial position change is relatively high (such as a performance video, an animation video, etc.), the first video is divided into multiple video clips, and the union of the detection boxes in each video frame in the video clip is used as the bounding box corresponding to the video clip.

[0163] Continue to see Figure 3A In step 103, a mask region of each video frame is determined, wherein the mask region of each video frame surrounds a mouth of the target object in the video frame.

[0164] In some embodiments, see Figure 3B , Figure 3A The step 103 shown can be implemented by following the steps 1031 to 1032, which are described in detail below.

[0165] In step 1031, the portion of the detection frame of each video frame including the mouth is used as a candidate mask frame, and the size of the candidate mask frame with the largest area is used as the target size.

[0166] In some embodiments, since the mouth is located in the lower half of the face, the lower half of the detection frame including the mouth is used as a candidate mask frame of the video frame, and the size of the candidate mask frame with the largest area is used as the target size to ensure that the mask area includes the mouth area, and then through subsequent mask processing, the jaw movement and other speech-related information are removed from the area within the enclosing frame, thereby ensuring that the mouth movement synchronized with the speech can be generated.

[0167] In step 1032, the size of the candidate mask frame of each video frame is set to the target size to obtain the target mask frame of each video frame, and the area surrounded by the target mask frame is used as the mask area.

[0168] In some embodiments, the scale of the candidate mask frame of each video frame is uniformly set to the target size, so as to obtain the target mask frame of each video frame, and the area of ​​the target mask frame is used as the mask area.

[0169] For example, assuming that the size of the detection box of a certain video frame is 2.8×4 (here, 2.8 and 4 represent the ratio between the width and height of the detection box), the lower half of the detection box is used as the candidate mask box of the video frame, then the size of the candidate mask box is 2.8×2, assuming that the target size is 3×2, then the size of the candidate mask box of this video frame is changed from 2.8×2 to 3×2 as the target mask box.

[0170] For example, see Figure 7C , Figure 7C is a schematic diagram of a bounding box and a mask area provided in an embodiment of the present application, from Figure 7C It can be seen that for multiple consecutive video frames in the first video, the position of the bounding box is fixed, the size of the mask area is uniform (3×2), the mask area is masked for the area below the mouth, and the area within the bounding box is used as the input of the video generation model. During the video generation process, this area is the pixel rendering area.

[0171] In some embodiments, see Figure 3C , Figure 3A The step 103 shown can be implemented by executing the following steps 1033 to 1034 for each video segment, which are described in detail below.

[0172] In step 1033, the portion of the detection frame of each video frame including the mouth is used as a candidate mask frame, and the size of the candidate mask frame with the largest area is used as the target size.

[0173] In some embodiments, the portion including the mouth in the detection frame of each video frame in the video clip is used as a candidate mask frame, and the size of the candidate mask frame with the largest area is used as the target size. The specific implementation method can be found in the description of step 1031 above, which will not be repeated here.

[0174] In step 1034, the size of the candidate mask frame of each video frame in the video clip is set to the target size to obtain the target mask frame of each video frame, and the area surrounded by the target mask frame is used as the mask area.

[0175] In some embodiments, the size of the candidate mask box of each video frame in the video segment is set to the target size. The specific implementation method can be found in the description of step 1032 above, which will not be repeated here.

[0176] Continue to see Figure 3A In step 104, a mask process is performed on the area in the video frame within the bounding box based on the mask area to obtain a masked video frame, and the masked video frame constitutes a masked video.

[0177] In some embodiments, see Figure 3D , Figure 3A The step 104 shown can be implemented by following the steps 1041 to 1043, which are described in detail below.

[0178] In step 1041, edge blurring is performed on the mask region of each video frame to obtain a blurred mask region.

[0179] In some embodiments, edge blurring is performed on the edge of the mask region of each video frame to obtain a blurred mask region.

[0180] For example, the edge of the mask area is found by the Canny edge detection algorithm, and the detected edge is blurred, for example, by using a Gaussian blur or a bilateral filter, so that the blurred edge is applied to the original mask area to obtain a blurred mask area.

[0181] In step 1042, the pixel values ​​in the fuzzy mask area are set to preset values ​​to obtain a masked video frame.

[0182] In some embodiments, the pixel values ​​in the fuzzy mask area are set to a preset value (such as 0) to obtain a masked video frame, wherein the image area included in the masked video frame is the area within the bounding box.

[0183] In step 1043, the mask video frames of each video frame are combined into a mask video.

[0184] In some embodiments, the mask video frames corresponding to each video frame in the first video are combined into a mask video.

[0185] For example, the mask video frames corresponding to each video frame in the first video are combined into a video frame sequence, and the video frame sequence is played at a preset speed (such as 30 frames / second) to obtain the mask video.

[0186] For example, a mask video frame corresponding to each video frame in the first video is subjected to video encoding processing (such as H.264 encoding standard) to obtain a mask video.

[0187] In other embodiments, the pixel value of the mask area in the area within the bounding box is first set to a preset value to obtain a masked video frame (the image area included in the masked video frame is the area within the bounding box, which can be understood as cropping the video frame of the first video through the bounding box, retaining the area within the bounding box, and setting the pixel value of the mask area therein to a preset value); then, the edge of the mask area in the masked video frame is blurred to obtain an edge blurred masked video frame; finally, the edge blurred masked video frames of each video frame are combined into a masked video.

[0188] In some embodiments, see Figure 3E , Figure 3A The step 104 shown can be implemented by executing the following steps 1044 to 1047 for each video segment, which are described in detail below.

[0189] In step 1044, edge blurring is performed on the mask region of each video frame to obtain a blurred mask region.

[0190] Here, the specific implementation method can refer to the description of step 1041 above, which will not be repeated here.

[0191] In step 1045, the pixel values ​​in the fuzzy mask area are set to preset values ​​to obtain a masked video frame.

[0192] Here, the specific implementation method can refer to the description of step 1042 above, which will not be repeated here.

[0193] In step 1046, the mask video frames for each video frame are combined into mask segments.

[0194] In some embodiments, the mask video frames for each video frame in the video segment are combined into a mask segment.

[0195] For example, the mask video frames corresponding to each video frame in the video segment are combined into a video frame sequence, and the video frame sequence is played at a preset speed (such as 30 frames / second) to obtain the mask segment.

[0196] For example, a video encoding process (such as H.264 encoding standard) is performed on the mask video frame corresponding to each video frame in the video segment to obtain the video segment.

[0197] In step 1047, the mask segments of each video segment are combined into a mask video.

[0198] In some embodiments, the mask segments for each video segment in the first video are combined into a mask video.

[0199] Through steps 101 to 104, detection frames are identified from multiple video frames of the first video, and based on the detection frames of the multiple video frames, bounding frames that surround the face of the target object in the multiple video frames are determined, so that a bounding frame that can be reused in multiple video frames is obtained. By obtaining the mask area of ​​each video frame, mask processing is performed in the area surrounded by the bounding frame, that is, the position of the bounding frame in the multiple video frames is fixed, and the area in the bounding frame after mask processing is used as the mask video frame, so that cross-frame masking is achieved, that is, the mask video frame is generated from the fixed area of ​​each video frame of the first video. Compared with the use of an unfixed bounding frame method in multiple video frames, the discontinuity or jump of facial information contained in the mask video due to the size difference or position difference of the bounding frame is avoided, the continuity of facial information between frames in the mask video is guaranteed, and continuous and stable data input is provided for video generation, thereby improving the video continuity of the generated video, so that the generated video has a better viewing experience. For example, when the video continuity is poor, flickering may occur during video playback, affecting the user experience.

[0200] Continue to see Figure 3A In step 105, a second video is generated based on the mask video, the first video and the audio data, wherein the mouth movement of the target object in the second video is synchronously matched with the audio data.

[0201] In some embodiments, see Fig. 6A , Fig. 6A It is a first schematic diagram of an optional structure of a video generation model provided in an embodiment of the present application. The video generation model can be an end-to-end model based on a hidden layer diffusion model, which is composed of two U-shaped networks with similar structures, namely an appearance network and a denoising network. Among them, the input of the appearance network is the features extracted from the reference image by encoder-1 and encoder-2 (corresponding to the second encoding features and image encoding features below), and the output is the reference image features; the input of the denoising network includes a noisy mask video, a reference image feature and an audio feature, wherein the acquisition of the noisy mask video is achieved in the following manner: the first video is feature encoded by encoder-3 (corresponding to the first feature encoding process in step 10511 below), and the encoding feature is obtained (corresponding to the first encoding feature in step 10511 below), and the encoding feature is denoised based on a preset time step (T) to obtain a noisy video, and the product (Hadamard product) of the noisy video and the mask video is used as the noisy mask video, and input into the denoising network, and the denoising network is used to obtain the predicted noise, and denoise is performed based on the predicted noise, so as to obtain the second video.

[0202] Here, for each frame of the second video, the synchronous matching of the mouth movement of the target object and the audio data means that the lip shape (mouth movement) of the target object is synchronized with the audio data.

[0203] In some embodiments, the above-mentioned noise addition and denoising processes may be implemented based on a diffusion model.

[0204] In some embodiments, see Figure 3F , Figure 3A The step 105 shown can be implemented by following the steps 1051 to 1053, which are described in detail below.

[0205] In step 1051, noise adding processing is performed based on the mask video and the first video to obtain a noisy mask video.

[0206] In some embodiments, noise processing is performed based on the mask video, the first video and the noise data to obtain a noisy mask video. Specifically, the first video is noise processed based on the noise data (such as Gaussian noise) to obtain a noisy video (corresponding to steps 10511 to 10513 below), and pixels at the same position in the noise video and the mask video are fused to obtain a noisy mask video (corresponding to step 10514 below).

[0207] In some embodiments, see Figure 3G , Figure 3FThe step 1051 shown can be implemented by performing the following steps 10511 to 10514 on each video frame in the first video, which is described in detail below.

[0208] In step 10511, a first feature encoding process is performed on the video frame to obtain a first encoding feature.

[0209] In some embodiments, a variational auto-encoder (VAE) can be used to perform a first feature encoding process on the video frame to obtain a first encoding feature. For example, the first feature encoding is performed on the video frame through a cascaded convolutional layer (Convolutional Layer, ConvLayer) and a fully connected layer (Fully Connected Layer, FC) in the encoder of the VAE to obtain the first encoding feature.

[0210] In other embodiments, the first feature encoding process can be performed on the video frame by other pre-configured deep learning models (such as Vision Transformer (ViT) etc.) to obtain the first encoding feature. The embodiment of the present application does not limit the encoder (corresponding to Fig. 6A Encoder-3).

[0211] In step 10512, noise data of a preset time step number is added to the first coding feature to obtain a noisy video frame.

[0212] In some embodiments, according to a preset time step, Gaussian noise (noise data) is gradually added to the first coding feature to obtain a noisy video frame, that is, the first coding feature is gradually converted into approximate Gaussian pure noise as a noisy video frame, where the preset time step represents the noise degree of the added noise data.

[0213] In step 10513, the noisy video frames of each video frame are combined into a noisy video.

[0214] In some embodiments, the resulting noisy video frames are combined into a noisy video.

[0215] For example, the noise video frames of each video frame are combined into a video frame sequence, and the video frame sequence is played at a preset speed (such as 30 frames / second) to obtain a noise video.

[0216] For example, a video encoding process (such as H.264 encoding standard) is performed on the noise video frame of each video frame to obtain a mask video.

[0217] In step 10514, pixels at the same position in the noise video and the mask video are fused to obtain a noisy mask video.

[0218] In some embodiments, the values ​​of pixels at the same position in the video frames that correspond one-to-one to the noise video and the mask video are bitwise multiplied element by element (or Hadamard product operation, etc.) to obtain the noisy mask video.

[0219] Continue to see Figure 3F In step 1052, noise prediction processing is performed on the noise masked video, the reference image and the audio data to obtain predicted noise, wherein the reference image includes the target object.

[0220] In some embodiments, the first video and the reference image both include a target object with the same appearance, and the mouth movements or head movements of the target object in the first video and the reference image are different. For example, the target object in the first video and the reference image are the same person with the same clothing, hairstyle, and background, but different mouth movements (lip shape) and / or head movements.

[0221] In some embodiments, see Figure 3H , Figure 3F The step 1052 shown can be implemented by following the steps 10521 to 10523, which are described in detail below.

[0222] In step 10521, feature extraction processing is performed on the reference image to obtain reference image features.

[0223] In some embodiments, see Fig. 3I , Figure 3H The step 10521 shown can be implemented by following the steps 201 to 205, which are described in detail below.

[0224] In step 201, a second feature encoding process is performed on a reference image to obtain a second encoding feature.

[0225] In some embodiments, the deep learning model (corresponding to Fig. 6A The encoder -1) performs a second feature encoding process on the reference image to obtain a second encoding feature. The specific implementation method can be found in the description of step 10511 above and will not be repeated here.

[0226] In step 202, the reference image is divided into a plurality of pixel blocks, and the plurality of pixel blocks are sorted into a sequence of pixel blocks to be encoded.

[0227] In some embodiments, the reference image is divided into a plurality of pixel blocks (Patches) according to a preset pixel block division size.

[0228] For example, assuming that the size of the reference image is 224×224, and it is divided into 16×16 pixel blocks, the result is (224 / 16) 2 =196 Patches (pixel blocks).

[0229] In some embodiments, a plurality of pixel blocks are arranged (eg, in order from left to right and from top to bottom) as a sequence of pixel blocks to be encoded, and the sequence of pixel blocks to be encoded is used as input units (Input Tokens) for image encoding processing.

[0230] In step 203, image coding processing is performed on the sequence of pixel blocks to be coded to obtain image coding features.

[0231] In some embodiments, an embedded coding process is performed on a sequence of pixel blocks to be encoded to obtain embedded features; an attention coding process (such as a self-attention mechanism, a multi-head attention mechanism, etc.) is performed on the embedded features to obtain attention coding features; and a mapping process is performed on the attention coding features to obtain image coding features.

[0232] For example, an embedding layer (Embedding Layer) can be used to perform embedded coding processing on a sequence of pixel blocks to be encoded to obtain embedded features. For example, a convolution operation is performed on the sequence of pixel blocks to be encoded in the embedding layer, and position coding (Position Embedding) is performed on the sequence of pixel blocks to be encoded after the convolution processing to obtain embedded features. Before the embedded coding processing is performed, each pixel block to be encoded in the sequence of pixel blocks to be encoded can be flattened into a one-dimensional vector. For example, for a 16×16×3 pixel block to be encoded, the length of the flattened vector is 768. The position coding processing can be generated by a fixed algorithm, for example, it can be implemented using a combination of sine and cosine functions. For each position (for example, the pixel position of each pixel point of each pixel block to be encoded) and the dimension of the feature, the position coding of the even dimension uses the sine function, while the position coding of the odd dimension uses the cosine function. The embodiment of the present application does not limit the specific implementation method of the position coding.

[0233] For example, before attention encoding, the embedded features can also be normalized (such as layer normalization) to obtain normalized features. For the input normalized features, a linear transformation is first performed to generate three matrices: query vector (Query, Q), key vector (Key, K) and value vector (Value, V). The linear transformation is composed of a learnable weight matrix W Q , W K and W VImplementation, Next, for each attention head (Head Attention), calculate the dot product of Q and K to obtain the attention score. Next, normalize the attention score, for example, apply a normalization function (such as a softmax function) to convert the attention score into a probability distribution, and use the attention probability distribution to weight V to generate a new feature representation. Finally, connect the outputs of all attention heads and perform a linear transformation to obtain a rich representation that contains the correlations at different positions in the first normalized feature, namely, the attention encoding feature.

[0234] For example, the attention encoding features can be mapped through a feed-forward neural network (FFN) layer to obtain image encoding features. For example, the feed-forward neural network layer can adopt a multilayer perceptron (MLP) structure. First, the attention encoding features are linearly transformed through the first linear layer. The first linear layer can be a weight matrix W 1 , the bias vector is b 1 The fully connected layer (FC) is then used to perform a nonlinear transformation on the attention coding features after the linear transformation through an activation function (such as ReLU, Sigmoid, etc.) to obtain nonlinear transformation features. Finally, the nonlinear transformation features are linearly transformed through the second linear layer to obtain image coding features. The second linear layer can be a weight matrix W 2 , the bias vector is b 2 The fully connected layer.

[0235] Among them, the encoder used for image encoding processing (corresponding to Fig. 6A The encoder in -2) can be constructed based on a Transformer network (such as ViT, the image encoder in the contrastive language-image pre-training model (Contrastive Language-ImagePre-Training, CLIP), etc.), for example, the input pixel block sequence to be encoded can be encoded through the TransformerEncoder structure to obtain image coding features, and here, multiple layers of Transformer Encoder structures can be superimposed for encoding. The embodiment of the present application does not limit the specific encoder structure.

[0236] In step 204, encoding is performed at least once based on the second encoding feature and the image encoding feature to obtain at least one first reference image feature.

[0237] In some embodiments, by Fig. 6AThe appearance network shown (including a spatial feature extraction layer and an attention layer-1) performs at least one encoding based on the second encoding feature and the image encoding feature to obtain at least one first reference image feature.

[0238] In some embodiments, see Figure 3J , Fig. 3I The step 204 shown can be implemented by following the steps 2041 to 2042, which are described in detail below.

[0239] In step 2041, a first spatial feature is extracted based on the input of the current encoding to obtain the first spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is the second encoding feature, and when the current encoding is not the first encoding, the input of the current encoding is the first reference image feature of the previous encoding output.

[0240] In some embodiments, see Fig. 6A , when the current encoding is the first encoding, the input of the current encoding is the second encoding feature (corresponding to Fig. 6A The input of the current encoding is the first reference image feature output by the previous encoding. For details, see Figure 6B , Figure 6B is a second schematic diagram of an optional structure of a video generation model provided in an embodiment of the present application, from Figure 6B It can be seen that the appearance network can be constructed by a U-Net, which includes multiple encoding blocks, an intermediate block and multiple decoding blocks, where each encoding block, intermediate block and decoding block includes Fig. 6A In the spatial feature extraction layer and attention layer-1, each encoding block outputs a first reference image feature, the feature output by the intermediate block is also called the first reference image feature, and the feature output by the decoding block is called the second reference image feature.

[0241] For example, see Fig. 6A , the first spatial feature extraction can be achieved by Fig. 6A The spatial feature extraction layer in the appearance network shown is implemented, and the spatial feature extraction layer and the temporal feature extraction layer both include a feature extraction module, and the feature extraction module includes a linear layer, a convolutional layer and a state-space model. For example, the spatial feature extraction layer and the temporal feature extraction layer can be implemented based on models such as Mamba and Transformer, and the temporal feature extraction layer can also be implemented based on timing models such as long short-term memory networks (LSTM). The embodiments of the present application do not limit the specific network structures of the spatial feature extraction layer and the temporal feature extraction layer.

[0242] In some embodiments, see Figure 3K , Figure 3JThe step 2041 shown can be implemented by following the steps 20411 to 20415, which are described in detail below.

[0243] In step 20411, a first linear transformation is performed on the input of the current encoding to obtain a first linear feature.

[0244] In some embodiments, a first linear transformation process may be performed on the input of the current encoding through a linear layer (Linear Layer) or a linear projection layer (LinearProjection Layer) to obtain a first linear feature. For example, the linear layer may be a weight matrix W 3 , the bias vector is b 3 The fully connected layer (FC) of the linear layer is different from the linear layer in that the number of rows of the weight matrix in the linear projection layer is less than the number of columns, thereby realizing the mapping of data from high dimension to low dimension. That is, if the number of rows of the weight matrix in the linear projection layer is equal to the number of columns, then the linear projection layer is equivalent to a standard linear layer.

[0245] For example, the first linear transformation process can be performed by Fig. 6A The linear layer-1 implementation of the feature extraction module in the appearance network is shown.

[0246] In step 20412, convolution processing is performed on the first linear feature to obtain a convolution feature.

[0247] In some embodiments, a preset convolution kernel (such as a convolution kernel of size 3×3) is used to slide on the feature map of the first linear feature according to a preset step size. After each slide, the grayscale value of the pixel on the feature map of the first linear feature is multiplied by the value on the corresponding convolution kernel, and then all the multiplied values ​​are added to obtain the grayscale value of the pixel on the convolution feature. After the sliding is completed, the feature map composed of the multiple grayscale value results is used as the convolution feature.

[0248] For example, convolution can be done by Fig. 6A The convolutional layer implementation of the feature extraction module in the appearance network is shown.

[0249] In step 20413, feature mapping is performed on the convolution features through matrix parameters to obtain mapping features.

[0250] In some embodiments, the convolutional features are mapped to mapped features by matrix parameters (eg, state matrix, input matrix, and output matrix) of a state space model (SSM) learned during a training phase.

[0251] For example, the continuous-time state-space model can be expressed as a differential equation in the form of: ′(t) = A(t)x(t) + B(t)u(t), y(t) = C(t)x(t), where x(t) represents the state vector, u(t) represents the input convolution feature, y(t) represents the output mapping feature, and x ′ (t) represents the derivative (the differential of x(t) with respect to time, representing the rate of change of the system state), A(t) represents the state matrix, B(t) represents the input matrix, and C(t) represents the output matrix.

[0252] In other embodiments, the convolution features can also be mapped to mapping features through the matrix parameters (such as state matrix, input matrix and output matrix) of the selective state space model (SSS) or the structured state space model (S4) of sequences learned in the training phase. The embodiments of the present application do not limit the specific state space model used.

[0253] For example, feature mapping can be processed by Fig. 6A State-space model implementation of the feature extraction module in the appearance network shown.

[0254] In step 20414, a second linear transformation is performed on the input of the current encoding to obtain a second linear feature.

[0255] In some embodiments, a second linear transformation process can be performed on the currently encoded input through a linear layer or a linear projection layer to obtain a second linear feature. The specific implementation method can be found in the description of step 20411 above, which will not be repeated here.

[0256] For example, the second linear transformation process can be performed by Fig. 6A The linear layer-2 implementation of the feature extraction module in the appearance network is shown.

[0257] In step 20415, the sum of the second linear feature and the mapping feature is obtained, and the sum is subjected to a third linear transformation to obtain the first spatial feature.

[0258] In some embodiments, a third linear transformation is performed on the sum of the second linear feature and the mapping feature to obtain a first spatial feature. Here, the third linear transformation can be implemented through a linear layer or a linear projection layer. The specific implementation method can be found in the description of step 20411 above, which will not be repeated here.

[0259] For example, the third linear transformation process can be performed by Fig. 6A The linear layer-3 implementation of the feature extraction module in .

[0260] Continue to see Figure 3JIn step 2042, a first attention processing is performed based on the first spatial feature and the image coding feature to obtain a first reference image feature of the current encoding output.

[0261] In some embodiments, the first spatial feature is used as a query vector (Q), and the image encoding feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the first reference image feature.

[0262] In other embodiments, the image encoding feature is used as a query vector (Q), and the first spatial feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the first reference image feature.

[0263] For example, the first attention process can be Fig. 6A Implementation of attention layer-1 in the appearance network shown in Fig.

[0264] Continue to see Fig. 3I In step 205, at least one decoding is performed based on at least one first reference image feature to obtain at least one second reference image feature.

[0265] In some embodiments, see Figure 3L , Fig. 3I The step 205 shown can be implemented by following the steps 2051 to 2052, which are described in detail below.

[0266] In step 2051, a second spatial feature is extracted based on the input of the current decoding to obtain a second spatial feature, wherein when the current decoding is the first decoding, the input of the current decoding is the first reference image feature of the previous encoding output, and when the current decoding is not the first decoding, the input of the current decoding is the second reference image feature of the previous decoding output.

[0267] In some embodiments, extracting the second spatial feature based on the input of the current decoding and obtaining the second spatial feature can be achieved in the following manner: performing a fourth linear transformation on the input of the current decoding to obtain a fourth linear feature (see the description of step 20411 above); performing convolution on the fourth linear feature to obtain a convolution feature (see the description of step 20412 above); performing feature mapping on the convolution feature through matrix parameters to obtain a mapping feature (see the description of step 20413 above); performing a fifth linear transformation on the input of the current decoding to obtain a fifth linear feature (see the description of step 20414 above); obtaining the sum of the fifth linear feature and the mapping feature, and performing a sixth linear transformation on the sum to obtain the second spatial feature (see the description of step 20415 above).

[0268] For example, the second spatial feature extraction can be done by Fig. 6A The spatial feature extraction layer implementation shown in .

[0269] In step 2052, a second attention processing is performed based on the second spatial features and the image coding features to obtain the second reference image features of the current decoding output.

[0270] In some embodiments, the second spatial feature is used as a query vector (Q), and the image encoding feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the second reference image feature.

[0271] In other embodiments, the image encoding feature is used as a query vector (Q), and the second spatial feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the second reference image feature.

[0272] For example, the second attention process can be Fig. 6A Attention layer-1 implementation of the appearance network shown.

[0273] Continue to see Figure 3H In step 10522, audio feature extraction is performed on the audio data to obtain audio features.

[0274] In some embodiments, audio features can be obtained by calling an audio feature extraction model (such as TitaNet, wav2vec, etc.) to perform audio feature extraction processing on audio data.

[0275] For example, the feature extraction of audio data through the TitaNet model can be achieved in the following way: first, the audio data is converted into a digital signal, for example, it is converted into a discrete time signal by sampling, and the audio signal is standardized to ensure that all audio data have similar amplitude levels. Next, the preprocessed audio data is input into the TitaNet model, and feature maps are obtained through the convolution layer. These feature maps represent multiple levels of representation of audio data, which represent various features of audio data, such as frequency, timing, etc. The feature maps can also be converted into more compact features, such as global average pooling, etc., so as to obtain the audio features corresponding to the audio data (corresponding to Fig. 6A audio features of the audio data in ).

[0276] In step 10523, the predicted noise is determined based on the reference image features, the audio features, and the noisy masked video.

[0277] In some embodiments, the predicted noise can be determined based on the reference image features, the audio features, and the noise mask video. Fig. 6A The denoising network implementation shown in , see Figure 6B The denoising network can be built based on the U-Net. The denoising network includes multiple encoding blocks, an intermediate block and a decoding block. Each encoding block, intermediate block and each decoding block includes Fig. 6A The spatial feature extraction layer (as), attention layer-1, attention layer-2 and temporal feature extraction layer shown in .

[0278] In some embodiments, see Figure 3M , Figure 3H The step 10523 shown can be implemented by following the steps 301 to 303, which are described in detail below.

[0279] In step 301, encoding is performed at least once based on the first reference image feature, the audio feature and the noise masked video to obtain at least one encoded time feature.

[0280] In some embodiments, see Figure 6B , each encoding block in the denoising network performs encoding once based on the first reference image feature, audio feature and noise mask video to obtain an encoded time feature, Figure 6B The features output by the intermediate blocks of the denoising network are also called encoded temporal features.

[0281] In some embodiments, see Figure 3N , Figure 3M The step 301 shown can be implemented by following the steps 3011 to 3014, which are described in detail below.

[0282] In step 3011, a third spatial feature is extracted based on the input of the current encoding to obtain the third spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is a noisy masked video, and when the current encoding is not the first encoding, the input of the current encoding is the encoding time feature of the previous encoding output.

[0283] In some embodiments, extracting the third spatial feature based on the input of the current encoding can be achieved in the following manner: performing a seventh linear transformation on the input of the current encoding to obtain a seventh linear feature (see the description of step 20411 above); performing convolution on the seventh linear feature to obtain a convolution feature (see the description of step 20412 above); performing feature mapping on the convolution feature through matrix parameters to obtain a mapping feature (see the description of step 20413 above); performing an eighth linear transformation on the input of the current encoding to obtain an eighth linear feature (see the description of step 20414 above); obtaining the sum of the eighth linear feature and the mapping feature, and performing a ninth linear transformation on the sum to obtain the third spatial feature (see the description of step 20415 above).

[0284] Here, when the current encoding is the first encoding, the input of the current encoding is a noisy masked video. The third spatial feature extraction based on the input of the current encoding can be implemented separately for each video frame in the noisy masked video, that is, the third spatial feature corresponding to each frame in the noisy masked video is output, and then the third attention processing is performed based on the third spatial feature corresponding to each frame and the first reference image feature to obtain the first attention feature of each frame.

[0285] For example, the third space feature extraction can be done by Fig. 6A Implementation of the spatial feature extraction layer of the denoising network.

[0286] In step 3012, a third attention processing is performed on the third spatial feature and the first reference image feature to obtain a first attention feature.

[0287] In some embodiments, the third spatial feature is used as a query vector (Q), and the first reference image feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the first attention feature.

[0288] In other embodiments, the first reference image feature is used as a query vector (Q), and the third spatial feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the first attention feature.

[0289] For example, the third attention process can be Fig. 6A Attention layer-1 implementation of the denoising network in [5].

[0290] In step 3013, the first attention feature and the audio feature are subjected to fourth attention processing to obtain a second attention feature.

[0291] In some embodiments, the first attention feature is used as a query vector (Q), and the audio feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely the second attention feature.

[0292] In other embodiments, the audio feature is used as a query vector (Q), and the first attention feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the second attention feature.

[0293] For example, the fourth attention process can be Fig. 6A Attention layer-2 implementation of the denoising network in [5].

[0294] In step 3014, the first time feature is extracted from the second attention feature to obtain the encoding time feature of the current encoding output.

[0295] In some embodiments, performing the first time feature extraction on the second attention feature to obtain the encoding time feature of the current encoding output can be achieved in the following manner: performing the tenth linear transformation processing on the second attention feature to obtain the tenth linear feature (see the description of step 20411 above); performing convolution processing on the tenth linear feature to obtain the convolution feature (see the description of step 20412 above); performing feature mapping processing on the convolution feature through matrix parameters to obtain the mapping feature (see the description of step 20413 above); performing the eleventh linear transformation processing on the second attention feature to obtain the eleventh linear feature (see the description of step 20414 above); obtaining the sum of the eleventh linear feature and the mapping feature, and performing the twelfth linear transformation processing on the sum to obtain the encoding time feature of the current encoding output (see the description of step 20415 above).

[0296] It should be noted that the network structure used for spatial feature extraction and temporal feature extraction is the same. The difference between the two is that spatial feature extraction performs feature extraction from the dimension of spatial information of video frames, and temporal feature extraction performs feature extraction from the dimension of temporal information of video frames, that is, the network structures of the two are the same, but the dimensions of input and output are different. Here, spatial information can be understood as the information of pixels in the width and height dimensions of the video frame. The information of each pixel contains color or grayscale values. These pixels are arranged in a certain spatial structure to form a video frame. Among them, spatial structure refers to the arrangement and organization of pixels in the image. Temporal information can be understood as a spatial information sequence composed of multiple consecutive video frames. There is a temporal dependency between multiple consecutive video frames.

[0297] For example, the input of the first spatial feature extraction can be a feature corresponding to a video frame (such as a second encoded feature or a first reference image feature), and the input of the first temporal feature extraction can be multiple consecutive video frames (such as a second attention feature corresponding to each video frame in a noisy masked video), resulting in different input and output dimensions between the first spatial feature extraction and the first temporal feature extraction.

[0298] For example, the first time feature extraction can be done by Fig. 6A Implementation of the temporal feature extraction layer of the denoising network.

[0299] Continue to see Figure 3M In step 302, at least one decoding is performed based on at least one encoding time feature to obtain at least one decoding time feature, wherein the last obtained decoding time feature among the at least one decoding time feature is the target decoding time feature.

[0300] For example, see Figure 6B,Each decoding block in the denoising network performs decoding once based on at least one encoded time feature to obtain a decoded time feature.

[0301] In some embodiments, see Fig.3O , Figure 3M The step 302 shown can be implemented by following the steps 3021 to 3024, which are described in detail below.

[0302] In step 3021, a fourth spatial feature is extracted based on the input of the current decoding to obtain a fourth spatial feature, wherein when the current decoding is the first decoding, the input of the current encoding is the encoding time feature of the previous encoding output; when the current decoding is not the first decoding, the input of the current decoding is the decoding time feature of the previous decoding output.

[0303] In some embodiments, a fourth spatial feature is extracted based on the input of the current decoding to obtain the fourth spatial feature, which can be achieved in the following manner: performing a thirteenth linear transformation on the input of the current decoding to obtain a thirteenth linear feature (see the description of step 20411 above); performing a convolution process on the thirteenth linear feature to obtain a convolution feature (see the description of step 20412 above); performing feature mapping on the convolution feature through matrix parameters to obtain a mapping feature (see the description of step 20413 above); performing a fourteenth linear transformation on the input of the current encoding to obtain a fourteenth linear feature (see the description of step 20414 above); obtaining the sum of the fourteenth linear feature and the mapping feature, and performing a fifteenth linear transformation on the sum to obtain the fourth spatial feature (see the description of step 20415 above).

[0304] For example, the fourth spatial feature extraction can be achieved by Fig. 6A Implementation of the spatial feature extraction layer of the denoising network.

[0305] In step 3022, the fourth spatial feature and the second reference image feature are subjected to fifth attention processing to obtain a third attention feature.

[0306] In some embodiments, the fourth spatial feature is used as a query vector (Q), and the second reference image feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the third attention feature.

[0307] In some embodiments, the second reference image feature is used as a query vector (Q), and the fourth spatial feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely, the third attention feature.

[0308] For example, the fifth attention process can be Fig. 6A Attention layer-1 implementation of the denoising network in [5].

[0309] In step 3023, the third attention feature and the audio feature are subjected to sixth attention processing to obtain a fourth attention feature.

[0310] In some embodiments, the third attention feature is used as a query vector (Q), and the audio feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely the fourth attention feature.

[0311] In some embodiments, the audio feature is used as a query vector (Q), and the third attention feature is used as a key vector (K) and a value vector (V). The attention score is calculated by the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution. The attention probability distribution is used to weight V to generate a new feature representation, namely the fourth attention feature.

[0312] For example, the sixth attention process can be Fig. 6A Attention layer-2 implementation of the denoising network in [5].

[0313] In step 3024, a second time feature is extracted from the fourth attention feature to obtain a decoding time feature of the current decoding output.

[0314] In some embodiments, the second time feature extraction is performed on the fourth attention feature to obtain the decoding time feature of the current decoding output, which can be achieved in the following manner: performing the sixteenth linear transformation processing on the fourth attention feature to obtain the sixteenth linear feature (see the description of step 20411 above); performing convolution processing on the sixteenth linear feature to obtain the convolution feature (see the description of step 20412 above); performing feature mapping processing on the convolution feature through matrix parameters to obtain the mapping feature (see the description of step 20413 above); performing the seventeenth linear transformation processing on the second attention feature to obtain the seventeenth linear feature (see the description of step 20414 above); obtaining the sum of the seventeenth linear feature and the mapping feature, and performing the eighteenth linear transformation processing on the sum to obtain the decoding time feature of the current decoding output (see the description of step 20415 above).

[0315] For example, the second time feature extraction can be done by Fig. 6A Implementation of the temporal feature extraction layer of the denoising network.

[0316] Continue to see Figure 3M , in step 303, the target decoding time feature is mapped to the prediction noise.

[0317] In some embodiments, the target decoded temporal features may be mapped to predicted noise corresponding to each video frame in the noise masked video through a fully connected layer.

[0318] Continue to see Figure 3F In step 1053, the noisy masked video is denoised based on the predicted noise to obtain a second video.

[0319] In some embodiments, the predicted noise is subtracted from the noisy masked video to obtain a second video. Here, an iterative denoising process of a preset time step can be performed, that is, noise prediction processing and denoising processing are performed once at each time step, and the second video is obtained after iterating the preset time step.

[0320] In some embodiments, noise prediction and denoising are performed once at each time step to obtain denoising features. Fig. 6A The decoder shown in decodes the denoising features to generate a second video.

[0321] For example, the decoder includes multiple convolutional layers and multiple transposed convolutional layers (Deconvolutional Layers), each convolutional layer or transposed convolutional layer includes an activation function (such as ReLU) and a normalization layer (such as Batch Normalization), and the last layer of the decoder outputs a feature map of the same size as the original image (the video frame in the first video). After a series of reverse operations, the feature map is restored to obtain the pixel value of the image, thereby obtaining the second video.

[0322] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the training method of the video generation model provided in the embodiment of the present application. Figure 4A , Figure 4A is a first flow chart of the training method of the video generation model provided in the embodiment of the present application, which will be combined with Figure 4A The steps shown are explained.

[0323] In step 401, a first detection box sample is identified from a plurality of video frames of a first video sample, wherein the first detection box sample in each video frame surrounds the face of a first target object sample in the video frame.

[0324] In some embodiments, target detection processing is performed on multiple video frames in the first video sample to obtain a first detection frame sample (corresponding to the detection frame above) in each video frame, wherein the first detection frame sample includes the face of the first target object sample.

[0325] In other embodiments, the first video sample is divided into multiple video segments, wherein each video segment includes multiple video frames; the following processing is performed on each video segment: target detection processing is performed on each video frame to obtain a first detection box sample in each video frame, wherein the first detection box sample includes the face of the first target object sample in the video frame.

[0326] Here, the specific implementation method can refer to the description of step 101 above.

[0327] In step 402, a first bounding box sample is determined based on first detection box samples of multiple video frames, wherein the first bounding box sample surrounds the face of the first target object sample in the multiple video frames.

[0328] In some embodiments, a union of first detection box samples in each video frame in the first video sample is determined, and the union is used as the first bounding box sample (corresponding to the detection box above).

[0329] In some other embodiments, the first video sample is divided into a plurality of video segments, and the union of the first detection box samples in each video frame in the video segments is used as the first bounding box samples corresponding to the video segments.

[0330] In step 403 , a first mask area sample of each video frame is determined, wherein the first mask area sample of each video frame surrounds a mouth of a first target object sample in the video frame.

[0331] In some embodiments, the portion including the mouth in the first detection frame sample of each video frame is taken as the first candidate mask frame sample (corresponding to the candidate mask frame), and the size of the first candidate mask frame sample with the largest area is taken as the first target size sample (see the description of step 1031 above); the size of the first candidate mask frame sample of each video frame is set to the first target size sample to obtain the first target mask frame sample of each video frame, and the area enclosed by the first target mask frame sample is taken as the first mask area sample (see the description of step 1032 above).

[0332] In other embodiments, the first video sample is divided into multiple video segments, and the following processing is performed on each video segment: the portion including the mouth in the first detection frame sample of each video frame is taken as the first candidate mask frame sample, and the size of the first candidate mask frame sample with the largest area is taken as the first target size sample; the size of the first candidate mask frame sample of each video frame in the video segment is set to the first target size sample to obtain the first target mask frame sample of each video frame, and the area enclosed by the first target mask frame sample is taken as the first mask area sample.

[0333] In step 404, a mask process is performed on an area in the video frame located within the first bounding box sample based on the first mask area sample to obtain a masked video frame sample, and the masked video frame sample constitutes a masked video sample.

[0334] In some embodiments, edge blurring is performed on the first mask area sample of each video frame in the first video sample to obtain a first blurred mask area sample (see the description in step 1041 above); the pixel value in the first blurred mask area sample is set to a preset value to obtain a masked video frame sample (see the description in step 1042 above); and the masked video frame samples of each video frame are combined into a masked video sample (see the description in step 1043 above).

[0335] In other embodiments, the first video sample is divided into multiple video segments, and the following processing is performed on each video segment: the first mask area sample of each video frame is edge blurred to obtain a first blurred mask area sample (see the description in step 1041 above); the pixel value in the first blurred mask area sample is set to a preset value to obtain a masked video frame sample (see the description in step 1042 above); the masked video frame samples of each video frame are combined into a masked segment sample (see the description in step 1043 above); and the masked segment samples of each video segment are combined into a masked video sample.

[0336] In step 405, a second video sample is generated based on the mask video sample, the first video sample and the audio data sample, wherein the mouth movement of the first target object sample in the second video sample is synchronously matched with the audio data sample.

[0337] In some embodiments, noise addition is performed based on the mask video sample and the first video sample to obtain a noisy masked video sample (see the description of step 1051 above); noise prediction is performed on the noisy masked video sample, the first reference image sample and the audio data sample to obtain a first predicted noise sample, wherein the first reference image sample includes a first target object sample (see the description of step 1052 above); and denoising is performed on the noisy masked video sample based on the first predicted noise sample to obtain a second video sample (see the description of step 1053 above).

[0338] In some embodiments, noise addition is performed based on the mask video sample and the first video sample to obtain a noisy mask video sample, which can be achieved by performing the following processing on each video frame in the first video sample: performing a first feature encoding process on the video frame to obtain a first encoded feature sample (see the description of step 10511 above); adding first noise data of a preset time step to the first encoded feature sample to obtain a noisy video frame sample (see the description of step 10512 above); combining the noisy video frame samples of each video frame into a noisy video sample (see the description of step 10513 above); and fusing pixels at the same position in the noisy video sample and the mask video sample to obtain a noisy mask video sample (see the description of step 10514 above).

[0339] In some embodiments, noise prediction processing is performed on the noise-added masked video sample, the first reference image sample and the audio data sample to obtain the first predicted noise sample, which can be achieved in the following manner: feature extraction processing is performed on the first reference image sample to obtain the first reference image feature sample (see the description of step 10521 above); audio feature extraction processing is performed on the audio data sample to obtain the audio feature sample (see the description of step 10522 above); the first predicted noise sample is determined based on the first reference image feature sample, the audio feature sample and the noise-added masked video sample (see the description of step 10523 above).

[0340] In some embodiments, the first reference image feature samples include reference image encoding feature samples and reference image decoding feature samples. Feature extraction processing is performed on the first reference image samples to obtain the first reference image feature samples, which can be achieved in the following manner: second feature encoding processing is performed on the first reference image samples to obtain second encoding feature samples (see the description of step 201 above); the first reference image samples are divided into multiple pixel blocks, and the multiple pixel blocks are sorted into pixel block sequence samples to be encoded (see the description of step 202 above); image encoding processing is performed on the pixel block sequence samples to be encoded to obtain image encoding feature samples (see the description of step 203 above); encoding is performed at least once based on the second encoding feature samples and the image encoding feature samples to obtain at least one reference image encoding feature sample (see the description of step 204 above, corresponding to the first reference image feature above); decoding is performed at least once based on at least one reference image encoding feature sample to obtain at least one reference image decoding feature sample.

[0341] In some embodiments, at least one encoding is performed based on the second encoding feature sample and the image encoding feature sample to obtain at least one reference image encoding feature sample, which can be achieved in the following manner: first spatial feature extraction is performed based on the input of the current encoding to obtain the first spatial feature sample, wherein when the current encoding is the first encoding, the input of the current encoding is the second encoding feature sample, and when the current encoding is not the first encoding, the input of the current encoding is the reference image encoding feature sample outputted by the previous encoding (see the description of step 2041 above); first attention processing is performed based on the first spatial feature sample and the image encoding feature sample to obtain the reference image encoding feature sample outputted by the current encoding (see the description of step 2042 above).

[0342] In some embodiments, at least one decoding is performed based on at least one reference image coding feature sample to obtain at least one reference image decoding feature sample, which can be achieved in the following manner: a second spatial feature is extracted based on the input of the current decoding to obtain a second spatial feature sample, wherein when the current decoding is the first decoding, the input of the current decoding is the reference image coding feature output of the previous encoding, and when the current decoding is not the first decoding, the input of the current decoding is the reference image decoding feature output of the previous decoding (see the description of step 2051 above); a second attention processing is performed based on the second spatial feature sample and the image coding feature sample to obtain the reference image decoding feature of the current decoding output (see the description of step 2052 above).

[0343] In some embodiments, determining a first predicted noise sample based on a first reference image feature sample, an audio feature sample, and a noise masked video sample can be achieved in the following manner: performing encoding at least once based on the first reference image feature sample, the audio feature sample, and the noise masked video sample to obtain at least one encoded time feature sample (see the description of step 301 above); performing decoding at least once based on at least one encoded time feature sample to obtain at least one decoded time feature sample, wherein the last obtained decoded time feature sample among the at least one decoded time feature sample is a target decoded time feature sample (see the description of step 302 above); mapping the target decoded time feature sample to a first predicted noise sample (see the description of step 303 above).

[0344] In some embodiments, at least one encoding is performed based on the first reference image feature sample, the audio feature sample and the noise mask video sample to obtain at least one encoded time feature sample, which can be achieved in the following manner: a third spatial feature extraction is performed based on the input of the current encoding to obtain a third spatial feature sample, wherein when the current encoding is the first encoding, the input of the current encoding is the noise mask video sample, and when the current encoding is not the first encoding, the input of the current encoding is the encoded time feature sample outputted by the previous encoding (see the description of step 3011 above); a third attention processing is performed on the third spatial feature sample and the first reference image feature sample to obtain a first attention feature sample (see the description of step 3012 above); a fourth attention processing is performed on the first attention feature sample and the audio feature sample to obtain a second attention feature sample (see the description of step 3013 above); a first time feature extraction is performed on the second attention feature sample to obtain the encoded time feature sample outputted by the current encoding (see the description of step 3014 above).

[0345] In step 406, a first loss value is determined based on the second video sample.

[0346] In some embodiments, see Figure 4B , Figure 4A The step 406 shown can be implemented by following the steps 4061 to 4064, which are described in detail below.

[0347] In step 4061, the first predicted noise sample corresponding to the second video sample is determined, wherein the second video sample is obtained by denoising the noisy masked video sample based on the first predicted noise sample, the first predicted noise sample is obtained by performing noise prediction processing on the noisy masked video sample, the first reference image sample and the audio data sample, the noisy masked video sample is obtained by performing noise processing on the masked video sample and the first video sample, and the noise processing is implemented based on the first noise data sample.

[0348] In some embodiments, noise prediction processing is performed on the noise masked video sample, the first reference image sample and the audio data sample to determine the first predicted noise sample. The specific implementation method can be found in the description of step 405 above and will not be repeated here.

[0349] In step 4062, a first sub-loss is determined based on a difference between the first predicted noise sample and the first noise data sample.

[0350] In some embodiments, based on the difference between the first predicted noise sample and the first noise data sample, determining the first sub-loss can be expressed by formula (1):

[0351]

[0352] Among them, z 0 1 : N =E(x 0 1 : N ), where E represents the encoder (such as the VAE encoder, corresponding to Fig. 6A Encoder-3), here represents bitwise multiplication or Hadamard product, z 0 1 : N represents the features of multiple video frames (1:N) in the first video sample without noise processing (corresponding to the first encoded feature sample), z t 1 : N Indicates based on z 0 1 : N Join 2 The features after the first noise data sample of time steps (corresponding to the noise video frame samples), represents the mask video sample, x 0 1:N Represents a sequence of multiple original video frames (first video sample) as input, ∈ 1 represents the Gaussian noise (first noise data sample) that follows the Gaussian distribution (∈~N(0,1)) added in the noise addition process, represents the first predicted noise sample, t 2 represents the preset time step in the noise addition process, a 1 : N Represents an audio data sample.

[0353] In step 4063, the product of the first sub-loss and the masked video sample is obtained as the second sub-loss.

[0354] In some embodiments, the product of the first sub-loss and the mask video sample is obtained as the second sub-loss, which can be expressed by formula (2):

[0355]

[0356] in, represents the first child loss, represents the mask video sample, Represents a bitwise multiplication or Hadamard product.

[0357] The second sub-loss is introduced because it is difficult to achieve accurate control of the local position if only the first sub-loss (mean square error) is used. In order to ensure that the video generation model can generate speech-synchronous mouth movements and reduce additional training costs, the local area of ​​the mouth is constrained (through the first sub-loss and Specifically, in order to reduce the optimization problem caused by the estimation error in the target detection process, a coarse-grained face mask (corresponding to the masked video sample) is obtained by performing an erosion operation on the fine-grained face mask (corresponding to the edge blurring process mentioned above), thereby helping to optimize the video generation model through the coarse-grained face constraint shown in formula (2). Without the need to introduce an additional optimization module, the beneficial effect of improving the synchronization between audio and lip is achieved. At the same time, this design has a higher versatility for diverse data.

[0358] In step 4064, the first sub-loss and the second sub-loss are weightedly summed according to preset weights to obtain a first loss value.

[0359] For example, the first loss value can be expressed by formula (3):

[0360] L total-1 =λ 1 L D-2 +λ 2 L LCF-2 (3)

[0361] Among them, λ 1 and λ 2 Indicates the preset weight, L total-1 Represents the first loss value, L D-2 represents the first sub-loss, L LCF-2 Indicates the second child loss.

[0362] Continue to see Figure 4A In step 407, the first video generation model is updated based on the first loss value to obtain a second video generation model.

[0363] In some embodiments, first gradient information is obtained through a first loss value, and parameters of a first video generation model are updated according to the first gradient information to obtain a second video generation model.

[0364] For example, the first gradient information of each parameter of the first video generation model for the first loss value is obtained through the back-propagation algorithm, and the parameters of the first video generation model are updated using the obtained first gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the first video generation model converges, thereby obtaining the second video generation model.

[0365] For example, Fig. 6A The encoder-1, encoder-2, encoder-3 and decoder shown in the figure can be pre-trained, and no parameter update is performed during the training process.

[0366] Through steps 401 to 407, it is realized that on the basis of the first video generation model that can extract identity features (corresponding to the second reference image feature sample below) and restore images (corresponding to the noise mask image sample below), the image fidelity can be improved (the training of the first generation model is shown in the description of steps 501 to 506 below), by introducing a speech attention layer (corresponding to Fig. 6A The potential connection between the hidden features (corresponding to the first attention feature sample above) and the speech feature sample (corresponding to the fourth attention process above) is established by introducing the temporal feature extraction layer (see Fig. 6A The time feature extraction layer in the video is used to fuse the information of adjacent frames in the time dimension (corresponding to the first time feature extraction above), thereby achieving the beneficial effect of generating multiple frames of images (second video samples) with consistent time sequence.

[0367] Since the first video generation model has learned how to restore the image, the current training stage (i.e., the stage of training the second video generation model) only needs to capture the correspondence between speech and image and the temporal continuity, which simplifies the training task and achieves the beneficial effect of improving training efficiency. Fig. 6A ), and a phased training method (i.e., training the first video generation model and the second video generation model in stages), to achieve the beneficial effect of improving the generalization ability of the video generation model.

[0368] In some embodiments, see Figure 4C Before updating the first video generation model based on the first loss value to obtain the second video generation model, the following steps 501 to 506 may also be performed, which are described in detail below.

[0369] In step 501, a second detection frame sample is identified from a first image sample, and the second detection frame sample is used as a second bounding frame sample, wherein the second bounding frame sample surrounds the face of the second target object sample in the first image sample.

[0370] In some embodiments, target detection processing is performed on the first image sample to obtain a detection frame of the first image sample (see the description of step 101 above), and the detection frame is used as the second bounding box sample, wherein the detection frame includes the face of the second target object sample.

[0371] In step 502 , a second mask area sample of the first image sample is determined, wherein the second mask area sample surrounds a mouth of the second target object sample in the first image sample.

[0372] In some embodiments, the lower half of the second bounding box sample (including the mouth of the second target object sample) is used as the mask box, and the area surrounded by the mask box is used as the second mask area sample.

[0373] In step 503, mask processing is performed on the area of ​​the first image sample located within the second bounding box sample based on the second mask area sample to obtain a mask image sample.

[0374] In some embodiments, edge blurring is performed on the second mask area sample to obtain a second blurred mask area sample (see the description of step 1041 above); the pixel value in the second blurred mask area sample is set to a preset value to obtain a mask image sample (see the description of step 1042 above).

[0375] In step 504, second image samples are generated based on the mask image samples and the first image samples.

[0376] In some embodiments, noise addition is performed based on the mask image samples and the first image samples to obtain noisy mask image samples (see the description of step 1051 above); noise prediction is performed on the noisy mask image samples and the second reference image samples to obtain second predicted noise samples, wherein the second reference image samples include second target object samples; and denoising is performed on the noisy mask image samples based on the second predicted noise samples to obtain mask image samples (see the description of step 1053 above).

[0377] In some embodiments, noise adding processing is performed based on the mask image sample and the first image sample to obtain a noisy mask image sample, which can be achieved in the following manner: performing a first feature encoding processing on the first image sample to obtain a second encoded feature sample (see the description of step 10511 above); adding a second noise data sample with a preset time step number to the second encoded feature sample to obtain a noise image sample (see the description of step 10512 above); fusing pixels at the same position in the noise image sample and the mask image sample to obtain a noisy mask image sample (see the description of step 10514 above).

[0378] In some embodiments, noise prediction processing is performed on the noisy mask image samples and the second reference image samples to obtain the second predicted noise samples, which can be achieved in the following manner: feature extraction processing is performed on the second reference image samples to obtain second reference image feature samples (see the description of step 10451 above); the second predicted noise samples are determined based on the second reference image feature samples and the noisy mask image samples.

[0379] In some embodiments, determining the second predicted noise sample based on the second reference image feature sample and the noise mask image sample can be achieved in the following manner: performing at least one encoding based on the second reference image feature sample and the noise mask image sample to obtain at least one encoded feature sample; performing at least one decoding based on the at least one encoded feature sample to obtain at least one decoded feature sample, wherein the last obtained decoded feature sample among the at least one decoded feature sample is the target decoded feature sample; and mapping the target decoded feature sample to the second predicted noise sample.

[0380] For example, at least one encoding is performed based on the second reference image feature sample and the noise mask image sample to obtain at least one encoded time feature sample, which can be achieved in the following way: based on the input of the current encoding, the fifth spatial feature is extracted to obtain the fifth spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is the noise mask image sample, and when the current encoding is not the first encoding, the input of the current encoding is the encoded feature sample output by the previous encoding; the fifth spatial feature and the second reference image feature sample are subjected to attention processing to obtain the encoded feature sample output by the current encoding, where the implementation method of the fifth spatial feature extraction can be found in the description of step 3011 above and will not be repeated here.

[0381] For example, performing at least one decoding based on at least one encoded feature sample to obtain at least one decoded feature sample can be implemented in the following manner: performing a sixth spatial feature extraction based on the input of the current decoding to obtain the sixth spatial feature, wherein when the current decoding is the first decoding, the input of the current encoding is the encoded feature sample output by the previous encoding, and when the current decoding is not the first decoding, the input of the current decoding is the decoded feature sample output by the previous decoding; performing attention processing on the sixth spatial feature and the second reference image feature sample to obtain a decoded feature sample output by the current decoding, wherein the implementation method of the sixth spatial feature extraction can refer to the description of step 3011 above, which will not be repeated here.

[0382] In step 505, a second loss value is determined based on the second image sample.

[0383] In some embodiments, see Figure 4D , Figure 4C The step 505 shown can be implemented by following the steps 5051 to 5054, which are described in detail below.

[0384] In step 5051, the second predicted noise sample corresponding to the second image sample is determined, wherein the second image sample is obtained by denoising the noisy mask image sample based on the second predicted noise sample, the second predicted noise sample is obtained by performing noise prediction processing on the noisy mask image sample and the second reference image sample, the noisy mask image sample is obtained by performing noise processing on the mask image sample and the first image sample, and the noise processing is implemented based on the second noise data sample.

[0385] In some embodiments, the noise mask image samples and the second reference image samples are subjected to noise prediction processing to determine the second predicted noise samples. The specific implementation method can be found in the description of step 504 above and will not be described again here.

[0386] In step 5052, a third sub-loss is determined based on the difference between the second predicted noise sample and the second noise data sample.

[0387] In some embodiments, the third sub-loss is determined based on the difference between the second predicted noise sample and the second noise data sample, which can be expressed by formula (4):

[0388]

[0389] Among them, z 0 =E(x 0 ), where E represents the encoder, here represents bitwise multiplication or Hadamard product, z 0 represents the features of the first image sample without noise processing (corresponding to the second encoded feature sample), z t Indicates based on z 0 Join 1 The features after the second noise data sample of time steps (corresponding to the noise image sample), M u represents the mask image sample, x 0 Represents the first image sample of the input, ∈ 2 represents the Gaussian noise (second noise data sample) that follows the Gaussian distribution (∈~N(0,1)) added in the noise addition process, represents the second prediction noise sample, t 1 Represents the preset time step in the noise addition process.

[0390] In step 5053, the product of the third sub-loss and the mask image sample is obtained as the fourth sub-loss.

[0391] In some embodiments, the product of the third sub-loss and the mask image sample is obtained as the fourth sub-loss, which can be expressed by formula (5):

[0392]

[0393] in, represents the third child loss, M u represents a mask image sample, Represents a bitwise multiplication or Hadamard product.

[0394] In step 5054, the third sub-loss and the fourth sub-loss are weightedly summed according to preset weights to obtain a second loss value.

[0395] In some embodiments, the third sub-loss and the fourth sub-loss are weighted and summed according to a preset weight to obtain a second loss value, which can be expressed by formula (6):

[0396] Ltotal-2 =λ 3 L D-1 +λ 4 L LCF-1 (6)

[0397] Among them, λ 3 and λ 4 Indicates the preset weight, L total-2 Represents the second loss value, L D-1 represents the third child loss, L LCF-1 Indicates the loss of the fourth child.

[0398] Continue to see Figure 4C In step 506, the initialized video generation model is updated based on the second loss value to obtain a first video generation model.

[0399] In some embodiments, second gradient information is obtained through a second loss value, and parameters of an initialized video generation model are updated according to the second gradient information to obtain a first video generation model.

[0400] For example, the second gradient information of each parameter of the initialized video generation model for the second loss value is obtained through the back-propagation algorithm, and the parameters of the initialized video generation model are updated using the obtained second gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the video generation model converges, thereby obtaining the first video generation model.

[0401] For example, see Fig. 6A In the figure, the 1 marked in the network layer of the appearance network and the denoising network indicates that the layer participates in the training of the first video generation model, and the 2 marked indicates that the layer participates in the training of the second video generation model, that is, the training of the second video generation model is achieved by adding a network layer marked as 2 on the basis of the first video generation model.

[0402] Through steps 501 to 506, by training the initialized video generation network, the first video generation model (or image restoration model) has the ability to extract identity features (second reference image feature samples) and restore noisy masked image samples to obtain second image samples. This training stage can be pre-trained on a large-scale image dataset to ensure strong generalization, thereby achieving the beneficial effect of improving image fidelity.

[0403] It should be noted that the embodiments of the present application can be applied to various video generation scenarios, such as digital human visual dubbing (such as movie dubbing scenarios), game character driving, human-computer interaction and other scenarios. In the movie dubbing scenario, the video generation method provided by the embodiment of the present application can intelligently and efficiently modify the mouth shape of the characters in the original film, meet the needs of modifying the script of the lines or translating the movie into different languages, thereby reducing the production cycle and labor costs. In the human-computer interaction scenario, the video generation method provided by the embodiment of the present application can generate real and natural digital human videos, thereby improving the experience of user-machine interaction. In the game character driving scenario, the video generation method provided by the embodiment of the present application can not only generate videos for 2D game characters, but also generate characters for 3D games, and obtain high-quality game video data.

[0404] Next, an exemplary application of the embodiment of the present application in a movie dubbing scenario will be described. Figure 5 , Figure 5 is a flow chart of a method for generating a video in a movie dubbing scenario provided by an embodiment of the present application, which will be combined with Figure 5 The steps shown are explained.

[0405] In step 601, an original video, an appearance reference image and a driving audio are obtained, wherein the original video and the appearance reference image both include an object to be dubbed with the same appearance.

[0406] In some embodiments, if the original video is copyrighted content, it needs to be obtained through legal channels, such as purchasing copyright, co-producing, or using free licensed video materials. If it is self-made content, it can be obtained by shooting or editing existing video materials. The original video and the appearance reference image both include objects to be dubbed with the same appearance (such as a character in a film or television work).

[0407] In step 602, a bounding box is determined from an original video including the object to be dubbed, wherein the bounding box surrounds the face of the object to be dubbed in multiple video frames of the original video.

[0408] In some embodiments, target detection processing is performed on multiple video frames in the original video to obtain a detection frame in each video frame, wherein the detection frame includes the face of the object to be dubbed (see the description of step 101 above); the union of the detection frames in each video frame is determined, and the union is used as the bounding box (see the description of step 102 above).

[0409] In step 603, a mask area of ​​each video frame is determined, wherein the mask area surrounds the mouth of the object to be dubbed in the video frame.

[0410] In some embodiments, the portion of the detection frame of each video frame including the mouth is taken as a candidate mask frame, and the size of the candidate mask frame with the largest area is taken as the target size (see the description of step 1031 above); the size of the candidate mask frame of each video frame is set to the target size to obtain the target mask frame of each video frame, and the area enclosed by the target mask frame is taken as the mask area (see the description of step 1032 above).

[0411] In step 604, mask processing is performed on the area in the bounding box in the video frame based on the mask area to obtain a masked video.

[0412] In some embodiments, the mask area of ​​each video frame is edge blurred to obtain a blurred mask area (see the description of step 1041 above); the pixel value in the blurred mask area is set to a preset value to obtain a masked video frame (see the description of step 1042 above); the masked video frames of each video frame are combined into a masked video (see the description of step 1043 above).

[0413] In step 605, a dubbing video is generated based on the mask video, the original video, the appearance reference image and the driving audio, wherein the mouth movement of the object to be dubbed in the dubbing video is synchronously matched with the driving audio.

[0414] In some embodiments, noise processing is performed based on the mask video and the original video (corresponding to the first video) to obtain a noisy masked video (see the description of step 1051 above); noise prediction processing is performed on the noisy masked video, the appearance reference image (corresponding to the reference image) and the driving audio (for audio data) to obtain predicted noise (see the description of step 1052 above); and denoising processing is performed on the noisy masked video based on the predicted noise to obtain a dubbed video (see the description of step 1053 above).

[0415] Through steps 601 to 605, the dubbing video generation in the movie dubbing scene is realized, and the dual-branch framework of the appearance network and the denoising network (see Fig. 6A ) not only ensures the quality of video generation, but also takes into account the efficiency of model training. The cross-frame mask (corresponding to step 602 to step 604) effectively improves the timing consistency and speech-lip synchronization. The following is an explanation of the effect diagram of video generation and experimental data.

[0416] See also Fig. 8A and Figure 8B , Figure 8B for Fig. 8A The corresponding color drawings ( Fig. 8A Yes Figure 8B Simplified line drawing), Fig. 8A and Figure 8B A qualitative comparison between the video generation method provided in the embodiment of the present application and other methods is shown in FIG. Fig. 8A and Figure 8B The first row shows the words being pronounced in these image frames, and the first four columns are schematic diagrams of the effect of reconstructing the video (that is, masking the lower half of the original video and reconstructing the video. The audio data of the original video and the reconstructed video are consistent). Fig. 8A and Figure 8B It can be seen from the figure that for video reconstruction (or video self-reconstruction task), the video generation method provided in the embodiment of the present application ensures the visual quality of the reconstructed video, has a high degree of speech synchronization, and also shows stronger generalization for the case where an object blocks the mouth (such as the microphone blocking the mouth in the third and fourth columns). For video dubbing (or cross-audio dubbing task), since the video is dubbed across audio, that is, the original video is dubbed using audio data different from the audio of the original video, for example, the original video is dubbed in English, and the generated dubbing video is dubbed in Chinese, so the video dubbing has no true value, so the lower half of the face corresponding to the true value is blocked, from Fig. 8A and Figure 8B It can be seen that the video generation method provided by the present application ensures the visual quality of the dubbing video and has a high degree of voice synchronization.

[0417] The following describes a comparison of the experimental results of the video generation method provided in an embodiment of the present application with those of the related art. The test data sets include a high-resolution audio-visual data set (High Definition Talking Face, HDTF), a lip reading recognition data set (Lip Reading Sentences 2, LRS2), and a more challenging high-resolution and difficult scene (for example, the mouth is blocked by an object) video clip selected from a high-quality data set for video face super-resolution (AHigh-Quality Dataset and Benchmark for Video Face Super-Resolution, VFHQ) to test the high-resolution video quality and generalization ability, referred to as VFHQ-C.

[0418] Six evaluation indicators are selected as quantitative evaluation methods. Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID) are used to evaluate image quality; Lip Sync Error Confidence (LSE-C), Lip Sync Error Distance (LSE-D) and Word Error Rate (WER) are used to evaluate speech lip synchronization; Fréchet Video Distance (FVD) is used to evaluate video multi-frame consistency. Among them, the higher the LSE-C value, the better the performance of its corresponding aspect; for LIPIPS, FID, FVD, LSE-D and WER, the lower the value, the better the performance of its corresponding aspect. Among them, the WER indicator is only evaluated on the LRS2 dataset.

[0419] See Table 1, which shows the evaluation results of the video generation method provided in the embodiments of the present application and the methods in the related technology under the above-mentioned video reconstruction task. The WER indicator only evaluates the LRS2 data set, and other indicators evaluate the VFHQ-C data set.

[0420] Table 1

[0421] method LPIPS LSE-C LSE-D FID FVD WER Wav2Lip 0.1317 7.2094 6.9966 18.4620 94.3733 75.9 DINet 0.0955 5.2746 9.0510 12.3246 115.8291 78.4 IP-LAP 0.0795 6.9510 7.1015 10.7277 63.8823 66.5 TalkLip 0.1040 6.7658 7.6372 14.4779 100.0698 72.2 Diff2Lip 0.0931 7.5891 6.5525 10.6108 64.4021 57.9 The present application embodiment method 0.0543 7.1272 6.4747 7.0619 35.5144 50.7

[0422] See Table 2, which shows the evaluation results of the video generation method provided in the embodiment of the present application and the method in the related technology under the above-mentioned video dubbing task. The WER indicator only evaluates the LRS2 data set, and other indicators evaluate the VFHQ-C data set.

[0423] Table 2

[0424] method LSE-C LSE-D FID FVD WER Wav2Lip 6.4223 7.8924 18.5791 110.8434 76.3 DINet 4.6266 9.5506 12.5959 122.7919 80.4 IP-LAP 6.2773 8.7868 11.3303 73.0300 65.6 TalkLip 4.8630 9.5676 14.8831 111.7875 75.8 Diff2Lip 6.3466 7.5899 11.3086 82.5527 68.6 The present application embodiment method 6.7416 7.4033 8.1377 46.0894 55.8

[0425] As can be seen from Tables 1 and 2, the video generation method provided in the embodiment of the present application has achieved the lowest LPIPS and FID scores on the VFHQ-C test set, whether it is video reconstruction or video dubbing tasks, which shows that the video generation method provided in the embodiment of the present application has obtained better generalization ability and generates higher quality images. The video generation method provided in the embodiment of the present application also obtained the lowest FVD score, which shows that the video generation method provided in the embodiment of the present application effectively maintains time consistency. For audio-lip synchronization, the video generation method provided in the embodiment of the present application obtained LSE-C and LSE-D scores comparable to or even better than Wav2Lip, DINet, TalkLip and Diff2Lip. The lower WER score also confirms the excellent audio lip synchronization of the video generation method provided in the embodiment of the present application. It is worth noting that the video generation method provided in the embodiment of the present application shows more significant advantages in the video dubbing task, proving its effectiveness in practical applications.

[0426] See also Fig. 9A and Fig. 9B , Fig. 9B for Fig. 9A The corresponding color drawings ( Fig. 9A Yes Fig. 9B Simplified line drawing), Fig. 9A and Fig. 9B It shows that in the video dubbing task, when there is an object occlusion (such as Fig. 9A and Fig. 9B The microphones in the first two rows block the mouth) and the lighting is uneven (such as Fig. 9A and Fig. 9B When the side face of the face in the third column is not illuminated sufficiently), the video generation method provided by the embodiment of the present application is compared with other methods. It can be seen that the video generation method provided by the embodiment of the present application has better generalization ability and generates higher quality images ( Fig. 9A The dotted line of the second column of microphones indicates that the image is blurred here).

[0427] See also Fig. 10A and Fig. 10B , Fig. 10B for Fig. 10A The corresponding color drawings ( Fig. 10A Yes Fig. 10B Simplified line drawing), Fig. 10A and Fig. 10B The first row in is the original video (English dubbing), Fig. 10A and Fig. 10BThe second row in the figure is a visual dubbing video (Chinese dubbing) generated by the video generation method provided by the embodiment of the present application. It can be seen that the lip movements of the visual dubbing video obtained by the video generation method provided by the embodiment of the present application match the audio of Chinese, which shows that the video generation method provided by the embodiment of the present application can perform beyond realistic character portraits (including Fig. 10A and Fig. 10B It can also be used for video translation or dubbing of cartoon characters in videos (i.e., foreign faces), and can provide realistic and natural detail enhancement in areas such as teeth.

[0428] In order to further illustrate the effect of the video generation method provided in the embodiment of the present application, a user study was conducted to evaluate the scores of the video generation method provided in the embodiment of the present application and related technologies in three aspects: visual quality (Visual Quality, VQ), audio-lip synchronization (Sync) and generalization ability (Generalization). Among them, a total of 30 users with professional English skills participated in this study, and the mean opinion score (Mean Opinion Score, MOS) was obtained by asking users to score each video from 1 (worst) to 5 (best) in three aspects. As shown in Table 3, IP-LAP and Diff2Lip achieved relatively high scores in visual quality and audio synchronization. The video generation method provided in the embodiment of the application is much better than other methods in these three aspects, which proves the superiority of the video generation method provided in the embodiment of the application.

[0429] Table 3

[0430] method Visual Quality Audio - Lip Sync Generalization Wav2Lip 3.26±023 3.59±0.29 3.70±0.19 DINet 2.48±0.24 2.96±0.24 2.62±0.22 IP-LAP 3.98±0.23 3.87±0.17 3.61±0.20 TalkLip 3.15±0.28 3.20±0.29 3.34±0.29 Diff2Lip 3.58±0.21 3.74±0.23 3.77±0.21 True value 4.87±0.10 4.82±0.15 4.94±0.09 The present application embodiment method 4.71±0.18 4.67±0.20 4.81±0.14

[0431] In summary, the video generation method provided in the embodiment of the present application can achieve higher single-frame image clarity and multi-frame consistency at the visual quality level, further improving the visual sensory experience of the generated video; without the need to introduce an additional optimization module, the constraints on the lip movement of speech synchronization are completed in the latent space, and a higher degree of synchronization is achieved. This design is more versatile for diverse data; it has stronger generalization, and can generate more natural, realistic, and stable results in the case of mouth occlusion and on out-of-domain face data (such as cartoon characters).

[0432] The following is a description of an exemplary structure of the video generation device 133 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the video generating device 133 of the memory 130-1 may include:

[0433] The data processing module 1331 is used to identify a detection frame from multiple video frames of the first video, wherein the detection frame in each of the video frames surrounds the face of the target object in the video frame.

[0434] In some embodiments, the data processing module 1331 is further used to determine a bounding box based on the detection boxes of the multiple video frames, wherein the bounding box surrounds the face of the target object in the multiple video frames.

[0435] In some embodiments, the data processing module 1331 is further used to determine a mask area of ​​each of the video frames, wherein the mask area of ​​each of the video frames surrounds the mouth of the target object in the video frame.

[0436] In some embodiments, the data processing module 1331 is further used to perform mask processing on the area in the video frame located in the bounding box based on the mask area to obtain a masked video frame; the masked video frame constitutes a masked video.

[0437] The generating module 1332 is used to generate a second video based on the mask video, the first video and the audio data, wherein the mouth movement of the target object in the second video is synchronously matched with the audio data.

[0438] In some embodiments, the data processing module 1331 is further used to perform target detection processing on multiple video frames in the first video to obtain a detection frame in each of the video frames, wherein the detection frame includes the face of the target object; determine the union of the detection frames in each of the video frames, and use the union as the bounding box.

[0439] In some embodiments, the data processing module 1331 is further used to use the part of the detection frame of each of the video frames that includes the mouth as a candidate mask frame, and use the size of the candidate mask frame with the largest area as the target size; set the size of the candidate mask frame of each of the video frames to the target size, obtain the target mask frame of each of the video frames, and use the area enclosed by the target mask frame as the mask area.

[0440] In some embodiments, the data processing module 1331 is also used to perform edge blurring on the mask area of ​​each of the video frames to obtain a blurred mask area; set the pixel value in the blurred mask area to a preset value to obtain a mask video frame; and combine the mask video frames of each of the video frames into the mask video.

[0441] In some embodiments, the data processing module 1331 is also used to divide the first video into multiple video segments, wherein each of the video segments includes multiple video frames; perform the following processing on each of the video segments: perform target detection processing on each of the video frames to obtain a detection frame in each of the video frames, wherein the detection frame includes the face of the target object in the video frame; determine the union of the detection frames in each of the video frames, and use the union as the bounding box.

[0442] In some embodiments, the data processing module 1331 is further used to perform the following processing for each of the video clips: taking the part of the detection frame of each of the video frames including the mouth as a candidate mask frame, and taking the size of the candidate mask frame with the largest area as the target size; setting the size of the candidate mask frame of each of the video frames in the video clip to the target size, obtaining the target mask frame of each of the video frames, and taking the area enclosed by the target mask frame as the mask area.

[0443] In some embodiments, the data processing module 1331 is also used to perform the following processing on each of the video segments: perform edge blurring on the mask area of ​​each of the video frames to obtain a blurred mask area; set the pixel value in the blurred mask area to a preset value to obtain a masked video frame; combine the masked video frames of each of the video frames into a masked segment; and combine the masked segments of each of the video segments into the masked video.

[0444] In some embodiments, the generation module 1332 is also used to perform noise processing based on the mask video and the first video to obtain a noisy mask video; perform noise prediction processing on the noisy mask video, a reference image and the audio data to obtain predicted noise, wherein the reference image includes the target object; and perform denoising processing on the noisy mask video based on the predicted noise to obtain the second video.

[0445] In some embodiments, the generation module 1332 is also used to perform the following processing on each video frame in the first video: perform a first feature encoding process on the video frame to obtain a first encoding feature; add noise data of a preset time step number to the first encoding feature to obtain a noisy video frame; combine the noisy video frames of each of the video frames into a noisy video; and fuse the pixels at the same position in the noisy video and the mask video to obtain a noisy mask video.

[0446] In some embodiments, the generation module 1332 is further used to perform feature extraction processing on the reference image to obtain reference image features; perform audio feature extraction processing on the audio data to obtain audio features; and determine predicted noise based on the reference image features, the audio features and the noisy masked video.

[0447] In some embodiments, the generation module 1332 is also used to perform a second feature encoding process on the reference image to obtain a second encoding feature; divide the reference image into multiple pixel blocks, and sort the multiple pixel blocks into a sequence of pixel blocks to be encoded; perform image encoding on the sequence of pixel blocks to be encoded to obtain an image encoding feature; perform at least one encoding based on the second encoding feature and the image encoding feature to obtain at least one first reference image feature; perform at least one decoding based on the at least one first reference image feature to obtain at least one second reference image feature.

[0448] In some embodiments, the generation module 1332 is also used to extract a first spatial feature based on the input of the current encoding to obtain a first spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is the second encoding feature, and when the current encoding is not the first encoding, the input of the current encoding is the first reference image feature of the previous encoding output; and perform a first attention processing based on the first spatial feature and the image encoding feature to obtain the first reference image feature of the current encoding output.

[0449] In some embodiments, the generation module 1332 is also used to perform a first linear transformation on the input of the current encoding to obtain a first linear feature; perform a convolution process on the first linear feature to obtain a convolution feature; perform feature mapping on the convolution feature through matrix parameters to obtain a mapping feature; perform a second linear transformation on the input of the current encoding to obtain a second linear feature; obtain the sum of the second linear feature and the mapping feature, and perform a third linear transformation on the sum to obtain the first spatial feature.

[0450] In some embodiments, the generation module 1332 is also used to extract a second spatial feature based on the input of the current decoding to obtain a second spatial feature, wherein when the current decoding is the first decoding, the input of the current decoding is the first reference image feature of the previous encoding output, and when the current decoding is not the first decoding, the input of the current decoding is the second reference image feature of the previous decoding output; and a second attention processing is performed based on the second spatial feature and the image encoding feature to obtain the second reference image feature of the current decoding output.

[0451] In some embodiments, the generation module 1332 is also used to perform at least one encoding based on the first reference image feature, the audio feature and the noise masked video to obtain at least one encoding time feature; perform at least one decoding based on the at least one encoding time feature to obtain at least one decoding time feature, wherein the last obtained decoding time feature among the at least one decoding time feature is the target decoding time feature; and map the target decoding time feature to predicted noise.

[0452] In some embodiments, the generation module 1332 is also used to extract a third spatial feature based on the input of the current encoding to obtain a third spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is the noisy masked video, and when the current encoding is not the first encoding, the input of the current encoding is the encoding time feature of the previous encoding output; perform a third attention processing on the third spatial feature and the first reference image feature to obtain a first attention feature; perform a fourth attention processing on the first attention feature and the audio feature to obtain a second attention feature; perform a first time feature extraction on the second attention feature to obtain the encoding time feature of the current encoding output.

[0453] In some embodiments, the generation module 1332 is also used to extract a fourth spatial feature based on the input of the current decoding to obtain a fourth spatial feature, wherein when the current decoding is the first decoding, the input of the current encoding is the encoding time feature of the previous encoding output, and when the current decoding is not the first decoding, the input of the current decoding is the decoding time feature of the previous decoding output; perform fifth attention processing on the fourth spatial feature and the second reference image feature to obtain a third attention feature; perform sixth attention processing on the third attention feature and the audio feature to obtain a fourth attention feature; perform second time feature extraction on the fourth attention feature to obtain a decoding time feature of the current decoding output.

[0454] The following is a description of an exemplary structure of a video generation model training device 134 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2B As shown, the software modules stored in the training device 134 of the video generation model stored in the memory 130-2 may include:

[0455] The data processing module 1341 is used to identify a first detection frame sample from multiple video frames of the first video sample, wherein the first detection frame sample in each of the video frames surrounds the face of the first target object sample in the video frame.

[0456] In some embodiments, the data processing module 1341 is further used to determine a first bounding box sample based on the first detection box samples of the multiple video frames, wherein the first bounding box sample surrounds the face of the first target object sample in the multiple video frames.

[0457] In some embodiments, the data processing module 1341 is further used to determine a first mask area sample for each of the video frames, wherein the first mask area sample for each of the video frames surrounds the mouth of the first target object sample in the video frame.

[0458] In some embodiments, the data processing module 1341 is further used to perform mask processing on the area in the video frame located within the first bounding box sample based on the first mask area sample to obtain a masked video frame sample; the masked video frame sample constitutes a masked video sample.

[0459] The generating module 1342 is used to generate a second video sample based on the mask video sample, the first video sample and the audio data sample, wherein the mouth movement of the first target object sample in the second video sample is synchronously matched with the audio data sample.

[0460] The training module 1343 is configured to determine a first loss value based on the second video sample.

[0461] In some embodiments, the training module 1343 is further used to update the first video generation model based on the first loss value to obtain a second video generation model.

[0462] In some embodiments, the training module 1343 is also used to determine a first predicted noise sample corresponding to the second video sample, wherein the second video sample is obtained by denoising the noisy masked video sample based on the first predicted noise sample, the first predicted noise sample is obtained by performing noise prediction processing on the noisy masked video sample, the first reference image sample and the audio data sample, the noisy masked video sample is obtained by performing noise addition processing on the masked video sample and the first video sample, and the noise addition processing is implemented based on the first noise data sample; based on the difference between the first predicted noise sample and the first noise data sample, a first sub-loss is determined; the product of the first sub-loss and the masked video sample is obtained as the second sub-loss; and the first sub-loss and the second sub-loss are weightedly summed according to preset weights to obtain the first loss value.

[0463] In some embodiments, the training module 1343 is also used to identify a second detection frame sample from the first image sample, and use the second detection frame sample as a second bounding frame sample, wherein the second bounding frame sample surrounds the face of the second target object sample in the first image sample; determine a second mask area sample of the first image sample, wherein the second mask area sample surrounds the mouth of the second target object sample in the first image sample; based on the second mask area sample, mask the area in the first image sample that is located within the second bounding frame sample to obtain a mask image sample; based on the mask image sample and the first image sample, generate a second image sample; determine a second loss value based on the second image sample; and update the initialized video generation model based on the second loss value to obtain the first video generation model.

[0464] In some embodiments, the training module 1343 is also used to determine a second predicted noise sample corresponding to the second image sample, wherein the second image sample is obtained by denoising the noisy mask image sample based on the second predicted noise sample, the second predicted noise sample is obtained by performing noise prediction processing on the noisy mask image sample and the second reference image sample, the noisy mask image sample is obtained by performing noise processing on the mask image sample and the first image sample, and the noise processing is implemented based on the second noise data sample; based on the difference between the second predicted noise sample and the second noise data sample, a third sub-loss is determined; the product of the third sub-loss and the mask image sample is obtained as a fourth sub-loss; and the third sub-loss and the fourth sub-loss are weightedly summed according to preset weights to obtain the second loss value.

[0465] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the video generation method or the video generation model training method described in the embodiment of the present application.

[0466] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the training method of the speech synthesis model or the speech synthesis method provided in the present application embodiment, for example, Figure 3A The video generation method shown or Figure 4A The video generation model training method is shown.

[0467] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0468] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0469] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0470] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0471] To summarize, through the embodiments of the present application, a bounding box that surrounds the face of the target object in multiple video frames is determined based on the detection box of multiple video frames, so that one bounding box can be reused for mask processing of multiple video frames, that is, in the facial area enclosed by the bounding box in each video frame, the mask area corresponding to the mouth is masked. Since the position of the bounding box in multiple video frames is fixed, the mask video frame is generated based on the fixed area in each video frame of the first video. Compared with the method of using a non-fixed bounding box in multiple video frames, the discontinuity or jump of facial information in different mask video frames due to the jump of the size or position of the bounding box is avoided, the continuity of facial information between mask video frames is improved, and then the continuity of facial information in the second video generated based on the mask video frame is improved, so that the user has a real and natural perception effect when watching the second video.

[0472] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: identifying a detection frame from a plurality of video frames of a first video, wherein the detection frame in each of the video frames surrounds a face of a target object in the video frame; Determine a bounding box based on the detection boxes of the multiple video frames, wherein the bounding box surrounds the face of the target object in the multiple video frames; Determining a mask region of each of the video frames, wherein the mask region of each of the video frames surrounds a mouth of the target object in the video frame; Based on the mask area, mask processing is performed on the area in the video frame that is within the bounding box to obtain a masked video frame; the masked video frame constitutes a masked video; A second video is generated based on the mask video, the first video and the audio data, wherein the mouth movement of the target object in the second video is synchronously matched with the audio data.

2. The method according to claim 1, characterized in that The step of identifying a detection frame from a plurality of video frames of the first video includes: Performing target detection processing on multiple video frames in the first video to obtain a detection frame in each of the video frames, wherein the detection frame includes the face of the target object; The determining of the bounding box based on the detection boxes of the multiple video frames includes: A union of the detection boxes in each of the video frames is determined, and the union is used as the bounding box.

3. The method according to claim 1, characterized in that Determining the mask area of ​​each of the video frames includes: Taking the portion of the detection frame of each video frame including the mouth as a candidate mask frame, and taking the size of the candidate mask frame with the largest area as the target size; The size of the candidate mask frame of each of the video frames is set to the target size to obtain a target mask frame of each of the video frames, and the area surrounded by the target mask frame is used as the mask area.

4. The method according to claim 1, characterized in that The step of performing mask processing on the area in the video frame within the bounding box based on the mask area to obtain a masked video frame includes: Performing edge blurring processing on the mask area of ​​each video frame to obtain a blurred mask area; Setting the pixel values ​​in the fuzzy mask area to a preset value to obtain a masked video frame; The mask video frame constitutes a mask video, including: The mask video frame of each of the video frames is combined into the mask video.

5. The method according to any one of claims 1 to 4, characterized in that: The step of generating a second video based on the mask video, the first video and the audio data comprises: Performing noise addition processing based on the mask video and the first video to obtain a noisy mask video; Performing noise prediction processing on the noise masked video, the reference image and the audio data to obtain predicted noise, wherein the reference image includes the target object; The noisy masked video is denoised based on the predicted noise to obtain the second video.

6. The method according to claim 5, characterized in that The performing noise adding processing based on the mask video and the first video to obtain the noisy mask video includes: Perform the following processing on each video frame in the first video: Performing a first feature encoding process on the video frame to obtain a first encoding feature; Adding noise data of a preset time step number to the first coding feature to obtain a noisy video frame; Combining the noisy video frames of each of the video frames into a noisy video; Pixels at the same position in the noise video and the mask video are fused to obtain a noisy mask video.

7. The method according to claim 5, characterized in that The step of performing noise prediction processing on the noise masked video, the reference image and the audio data to obtain predicted noise includes: Performing feature extraction processing on the reference image to obtain reference image features; Performing audio feature extraction processing on the audio data to obtain audio features; Predicted noise is determined based on the reference image features, the audio features, and the noisy masked video.

8. The method according to claim 7, characterized in that The step of performing feature extraction processing on the reference image to obtain reference image features includes: Performing a second feature encoding process on the reference image to obtain a second encoding feature; Dividing the reference image into a plurality of pixel blocks, and sorting the plurality of pixel blocks into a sequence of pixel blocks to be encoded; Performing image coding processing on the pixel block sequence to be coded to obtain image coding features; Perform at least one encoding based on the second encoding feature and the image encoding feature to obtain at least one first reference image feature; At least one decoding is performed based on the at least one first reference image feature to obtain at least one second reference image feature.

9. The method according to claim 8, characterized in that The performing at least one encoding based on the second encoding feature and the image encoding feature comprises: Extracting a first spatial feature based on an input of a current encoding to obtain a first spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is the second encoding feature, and when the current encoding is not the first encoding, the input of the current encoding is the first reference image feature outputted by the previous encoding; A first attention processing is performed based on the first spatial feature and the image coding feature to obtain a first reference image feature of the current encoding output.

10. The method according to claim 9, characterized in that The first spatial feature extraction is performed based on the input of the current encoding to obtain the first spatial feature, including: Performing a first linear transformation process on the input of the current encoding to obtain a first linear feature; Performing convolution processing on the first linear feature to obtain a convolution feature; Performing feature mapping processing on the convolution feature through matrix parameters to obtain mapping features; Performing a second linear transformation process on the currently encoded input to obtain a second linear feature; A sum of the second linear feature and the mapping feature is obtained, and a third linear transformation is performed on the sum to obtain the first spatial feature.

11. The method according to claim 8, characterized in that The step of determining the predicted noise based on the reference image feature, the audio feature and the noise masked video comprises: Perform at least one encoding based on the first reference image feature, the audio feature, and the noise masked video to obtain at least one encoding time feature; Perform at least one decoding based on the at least one encoding time feature to obtain at least one decoding time feature, wherein the last obtained decoding time feature among the at least one decoding time feature is a target decoding time feature; The target decoding time feature is mapped to prediction noise.

12. The method according to claim 11, characterized in that The performing encoding at least once based on the first reference image feature, the audio feature and the noise masked video to obtain at least one encoding time feature comprises: Extracting a third spatial feature based on the input of the current encoding to obtain a third spatial feature, wherein when the current encoding is the first encoding, the input of the current encoding is the noise masked video, and when the current encoding is not the first encoding, the input of the current encoding is the encoding time feature of the last encoding output; performing a third attention processing on the third spatial feature and the first reference image feature to obtain a first attention feature; Performing a fourth attention process on the first attention feature and the audio feature to obtain a second attention feature; Perform the first time feature extraction on the second attention feature to obtain the encoding time feature of the current encoding output.

13. A video generation model training method, characterized in that: The method comprises: Identify a first detection box sample from a plurality of video frames of a first video sample, wherein the first detection box sample in each of the video frames surrounds a face of a first target object sample in the video frame; Determining a first bounding box sample based on the first detection box samples of the multiple video frames, wherein the first bounding box samples all surround the face of the first target object sample in the multiple video frames; Determine a first mask area sample of each of the video frames, wherein the first mask area sample of each of the video frames surrounds a mouth of the first target object sample in the video frame; Based on the first mask area sample, mask processing is performed on the area in the video frame that is located within the first bounding box sample to obtain a masked video frame sample; the masked video frame sample constitutes a masked video sample; generating a second video sample based on the mask video sample, the first video sample and the audio data sample, wherein the mouth movement of the first target object sample in the second video sample is synchronously matched with the audio data sample; determining a first loss value based on the second video sample; The first video generation model is updated based on the first loss value to obtain a second video generation model.

14. The method according to claim 13, characterized in that The determining a first loss value based on the second video sample comprises: Determine a first predicted noise sample corresponding to the second video sample, wherein the second video sample is obtained by denoising a noisy masked video sample based on the first predicted noise sample, the first predicted noise sample is obtained by performing noise prediction processing on the noisy masked video sample, a first reference image sample and the audio data sample, the noisy masked video sample is obtained by performing noise addition processing on the masked video sample and the first video sample, and the noise addition processing is implemented based on the first noise data sample; determining a first sub-loss based on a difference between the first predicted noise sample and the first noise data sample; Obtaining a product of the first sub-loss and the mask video sample as a second sub-loss; The first sub-loss and the second sub-loss are weightedly summed according to a preset weight to obtain the first loss value.

15. The method according to claim 13 or 14, characterized in that Before updating the first video generation model based on the first loss value to obtain the second video generation model, the method further includes: Identify a second detection frame sample from the first image sample, and use the second detection frame sample as a second bounding frame sample, wherein the second bounding frame sample surrounds the face of the second target object sample in the first image sample; determining a second mask area sample of the first image sample, wherein the second mask area sample surrounds a mouth of the second target object sample in the first image sample; performing mask processing on an area in the first image sample that is within the second bounding box sample based on the second mask area sample to obtain a mask image sample; generating a second image sample based on the mask image sample and the first image sample; determining a second loss value based on the second image sample; The initialized video generation model is updated based on the second loss value to obtain the first video generation model.

16. A video generating device, characterized in that: The device comprises: A data processing module, configured to identify a detection frame from a plurality of video frames of a first video, wherein the detection frame in each of the video frames surrounds a face of a target object in the video frame; The data processing module is further used to determine a bounding box based on the detection boxes of the multiple video frames, wherein the bounding box surrounds the face of the target object in the multiple video frames; The data processing module is further used to determine a mask area of ​​each of the video frames, wherein the mask area of ​​each of the video frames surrounds the mouth of the target object in the video frame; The data processing module is further used to perform mask processing on the area in the video frame located within the bounding box based on the mask area to obtain a masked video frame; the masked video frame constitutes a masked video; A generation module is used to generate a second video based on the mask video, the first video and the audio data, wherein the mouth movement of the target object in the second video is synchronously matched with the audio data.

17. A training device for a video generation model, characterized in that: A data processing module, configured to identify detection frame samples from a plurality of video frames of a first video sample, wherein the detection frame sample in each of the video frames surrounds a face of a first target object sample in the video frame; The data processing module is further used to determine a first bounding box sample based on the detection box samples of the multiple video frames, wherein the first bounding box sample surrounds the face of the first target object sample in the multiple video frames; The data processing module is further used to determine a first mask area sample of each of the video frames, wherein the first mask area sample of each of the video frames surrounds the mouth of the first target object sample in the video frame; The data processing module is further used to perform mask processing on the area in the video frame located within the first bounding box sample based on the first mask area sample to obtain a masked video frame sample; the masked video frame sample constitutes a masked video sample; A generating module, configured to generate a second video sample based on the mask video sample, the first video sample and the audio data sample, wherein the mouth movement of the first target object sample in the second video sample is synchronously matched with the audio data sample; A training module, configured to determine a first loss value based on the second video sample; The training module is also used to update the first video generation model based on the first loss value to obtain a second video generation model.

18. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, for implementing the video generation method described in any one of claims 1 to 12, or implementing the video generation model training method described in any one of claims 13 to 15, when executing the computer executable instructions or computer program stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the video generation method described in any one of claims 1 to 12 is implemented, or the training method of the video generation model described in any one of claims 13 to 15 is implemented.

20. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the video generation method described in any one of claims 1 to 12 or the training method of the video generation model described in any one of claims 13 to 15 is implemented.

Images