Video generation method, device, electronic device and storage medium

Abstract

The disclosed embodiments disclose a video generation method, device, electronic device and storage medium, which obtain a starting frame image and an ending frame image of a target video to be generated, extract a first potential feature of the starting frame image and a second potential feature of the ending frame image, extract a first optical flow between the starting frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the ending frame image and the intermediate frame image, distort the first potential feature based on the first optical flow, distort the second potential feature based on the second optical flow, fuse the distorted first potential feature and the distorted second potential feature to obtain a first distorted feature of the intermediate frame image, splice the first potential feature, the first distorted feature and the second potential feature with random noise respectively, and input them into a diffusion model for denoising to obtain a target video, thereby improving the quality of video generation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a video generation method, device, electronic device and storage medium. Background Art

[0002] At present, in order to improve the smoothness of videos, the operation of interpolating frames in videos has been widely used. In related technologies, the input frames (for example, the starting frame image and the ending frame image) are generally input into a generative model, and the intermediate frame images are predicted to generate the interpolated video to achieve the effect of video interpolation. In this way, the relationship between the input frames is crucial for the understanding and reasoning of the model. However, related technologies usually require a generative model to infer the relationship between the input frames. This ability is often not fully cultivated during the model training process, resulting in the generated video being easily affected by factors such as unstable motion or incoherent input frame content, thereby reducing the quality of video generation. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present disclosure provide a video generation method, device, electronic device and storage medium, which can improve the quality of video generation.

[0005] In one aspect, an embodiment of the present disclosure provides a video generation method, comprising:

[0006] Acquire a starting frame image and an ending frame image of a target video to be generated, and extract a first potential feature of the starting frame image and a second potential feature of the ending frame image;

[0007] Extracting a first optical flow between the start frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the end frame image and the intermediate frame image;

[0008] distorting the first latent feature based on the first optical flow, distorting the second latent feature based on the second optical flow, and fusing the distorted first latent feature and the distorted second latent feature to obtain a first distorted feature of the intermediate frame image;

[0009] The first latent feature, the first distorted feature, and the second latent feature are respectively spliced ​​with random noise and input into a diffusion model for denoising to obtain the target video.

[0010] On the other hand, an embodiment of the present disclosure provides a video generating device, including:

[0011] A feature extraction module, used to obtain a starting frame image and an ending frame image of a target video to be generated, and extract a first potential feature of the starting frame image and a second potential feature of the ending frame image;

[0012] An optical flow extraction module, used to extract a first optical flow between the start frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the end frame image and the intermediate frame image;

[0013] a feature change module, configured to distort the first potential feature based on the first optical flow, distort the second potential feature based on the second optical flow, and fuse the distorted first potential feature and the distorted second potential feature to obtain a first distorted feature of the intermediate frame image;

[0014] A generation module is used to splice the first potential feature, the first distortion feature, and the second potential feature with random noise respectively, and input them into a diffusion model for denoising to obtain the target video.

[0015] Furthermore, the optical flow extraction module is also used to:

[0016] Obtaining the frame sequence number of the intermediate frame image to be generated in the target video;

[0017] The starting frame image, the ending frame image and the frame sequence number are input into an optical flow prediction model for prediction to obtain a first optical flow between the starting frame image and the intermediate frame image, and a second optical flow between the ending frame image and the intermediate frame image.

[0018] Furthermore, the feature change module is also used to:

[0019] determining a first Hadamard product between the occlusion mask and the warped first latent feature;

[0020] determining a complementary mask of the occlusion mask, and determining a second Hadamard product between the complementary mask and the distorted second latent feature;

[0021] A first distortion feature of the intermediate frame image is obtained according to the sum of the first Hadamard product and the second Hadamard product.

[0022] Furthermore, the generation module is also used to:

[0023] The first latent feature, the first distorted feature, and the second latent feature are respectively concatenated with random noise and input into a diffusion model;

[0024] For each of the processing blocks, sequentially mapping the input of the processing block through the spatial layer and the temporal layer to obtain the output of the processing block;

[0025] The target video is obtained based on the output of the last processing block.

[0026] Furthermore, the generation module is also used to:

[0027] Mapping the input of the processing block through the spatial layer, and extracting the first spatial feature corresponding to the start frame image and the second spatial feature corresponding to the end frame image from the output of the spatial layer;

[0028] distorting the first spatial feature based on the first optical flow, distorting the second spatial feature based on the second optical flow, and fusing the distorted first spatial feature and the distorted second spatial feature to obtain a second distorted feature of the intermediate frame image;

[0029] The output of the spatial layer is mapped through the temporal layer, and the output of the temporal layer is fused with the second warping feature to obtain the output of the processing block.

[0030] Furthermore, the generation module is also used to:

[0031] Acquire an occlusion mask corresponding to the frame number between the start frame image and the end frame image;

[0032] determining a third Hadamard product between the occlusion mask and the warped first spatial feature;

[0033] Determining a complementary mask of the occlusion mask, and determining a fourth Hadamard product between the complementary mask and the distorted second spatial feature;

[0034] A second distortion feature of the intermediate frame image is obtained according to the sum of the third Hadamard product and the fourth Hadamard product.

[0035] Furthermore, the generation module is also used to:

[0036] mapping the second distortion feature through the smoothing layer;

[0037] The output of the temporal layer is fused with the mapped second warped feature to obtain the output of the processing block.

[0038] Furthermore, the generation module is also used to:

[0039] Obtaining a constraint text for constraining the video content of the target video, and extracting text features of the constraint text;

[0040] Performing self-attention processing on the output of the spatial layer through the temporal layer to obtain a first attention feature;

[0041] Performing cross-attention processing on the text feature and the first attention feature to obtain a second attention feature;

[0042] The second attention feature is mapped to obtain the output of the time layer.

[0043] Furthermore, the generation module is also used to:

[0044] Acquire a sample video, wherein the sample video includes a first sample image located at a start frame, a second sample image located at an end frame, and a third sample image located between the first sample image and the second sample image;

[0045] Extracting a first sample feature of the first sample image, a second sample feature of the second sample image, and a third sample feature of the third sample image;

[0046] Adding sample noise to the first sample feature, the second sample feature and the third sample feature respectively, to obtain a first noise feature, a second noise feature and a third noise feature respectively;

[0047] extracting a third optical flow between the first sample image and the third sample image, and a fourth optical flow between the second sample image and the third sample image;

[0048] Distorting the first sample feature based on the third optical flow, distorting the second sample feature based on the fourth optical flow, and fusing the distorted first sample feature and the distorted second sample feature to obtain a third distorted feature of the third sample image;

[0049] splicing the first noise feature with the first sample feature to obtain a first splicing feature, splicing the second noise feature with the second sample feature to obtain a second splicing feature, and splicing the third noise feature with the third distortion feature to obtain a third splicing feature;

[0050] Inputting the first splicing feature, the second splicing feature and the third splicing feature into the diffusion model for denoising;

[0051] The predicted noise output by the diffusion model during the denoising process is obtained, a model loss is determined according to a difference between the predicted noise and the sample noise, and the diffusion model is trained based on the model loss.

[0052] Furthermore, the generation module is also used to:

[0053] inputting the first stitching feature, the second stitching feature, and the third stitching feature into the diffusion model;

[0054] For each of the processing blocks, the input of the processing block is mapped through the spatial layer, the third spatial feature corresponding to the first sample image and the fourth spatial feature corresponding to the second sample image are extracted from the output of the spatial layer, the third spatial feature is distorted based on the third optical flow, the fourth spatial feature is distorted based on the fourth optical flow, the distorted third spatial feature and the distorted fourth spatial feature are fused to obtain the fourth distorted feature of the third sample image, the output of the spatial layer is mapped through the temporal layer, the output of the temporal layer is fused with the fourth distorted feature to obtain the output of the processing block.

[0055] Furthermore, the generation module is also used to:

[0056] Freezing the parameters of the time layer in each of the processing blocks;

[0057] The parameters of the spatial layers in each of the processing blocks are adjusted based on the model loss to train the diffusion model.

[0058] On the other hand, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned video generation method when executing the computer program.

[0059] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned video generation method.

[0060] On the other hand, the embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned video generation method.

[0061] The disclosed embodiment includes at least the following beneficial effects: by acquiring the start frame image and the end frame image of the target video to be generated, extracting the first potential feature of the start frame image and the second potential feature of the end frame image, extracting the first optical flow between the start frame image and the intermediate frame image to be generated in the target video, and extracting the second optical flow between the end frame image and the intermediate frame image; since the first optical flow and the second optical flow can be used to simulate the motion trajectory and the appearance of the object, the first potential feature is distorted based on the first optical flow, the second potential feature is distorted based on the second optical flow, and the distorted first potential feature and the distorted second potential feature are fused, so that the first distorted feature of the intermediate frame image can be approximately estimated as an effective guide for motion inference in the generation process, reflecting the correlation between the start frame image and the end frame image; subsequently, when the first potential feature, the first distorted feature, and the second potential feature are spliced ​​with random noise respectively and input into the diffusion model for denoising, the first distorted feature can be used as an explicit motion clue, so that the model can be aware of the correlation between the start frame image and the end frame image without inference, thereby improving the motion stability and content consistency of the generated target video, and improving the quality of video generation.

[0062] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0064] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure;

[0065] Figure 2 An optional flow chart of the video generation method provided in the embodiment of the present disclosure;

[0066] Figure 3 An optional schematic diagram of a noise generation module provided in an embodiment of the present disclosure;

[0067] Figure 4 An optional schematic diagram of a denoising module provided in an embodiment of the present disclosure;

[0068] Figure 5 An optional schematic diagram of a processing block structure provided in an embodiment of the present disclosure;

[0069] Figure 6 Another optional schematic diagram of a processing block structure provided by an embodiment of the present disclosure;

[0070] Figure 7 An optional schematic diagram of a time layer structure provided in an embodiment of the present disclosure;

[0071] Figure 8 Another optional schematic diagram of a time layer structure provided in an embodiment of the present disclosure;

[0072] Fig. 9 An optional overall flow chart of a video generation method provided in an embodiment of the present disclosure;

[0073] Fig.10 A schematic diagram of an optional structure of a video generating device provided in an embodiment of the present disclosure;

[0074] Fig.11 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0075] Fig.12 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0077] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present disclosure will be obtained.

[0078] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0079] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0080] Diffusion Model: A deep learning generative model based on random processes. It generates new data samples by simulating the diffusion process of data distribution. The idea is to model the data generation process as an inverse Markov chain. The diffusion process is divided into two stages: noise addition and data generation. The noise addition process simulates the process of data distribution gradually changing to Gaussian noise distribution, gradually adding noise to the original data until the original data becomes pure noise that does not contain any original data information. The data generation process is to reversely restore the original data from the noise state. It is mostly used in scenarios such as image generation and image restoration.

[0081] Optical Flow: refers to a two-dimensional vector field that describes the motion trajectory and speed of pixels in consecutive frames. It is used to capture the motion of objects in the image and the displacement information between adjacent frames. The optical flow algorithm estimates the speed and direction of each pixel by analyzing the changes in pixel brightness in adjacent frames.

[0082] Backward warping: also known as reverse deformation, is an image deformation technology that reversely calculates the position of pixels before deformation based on the coordinates of the pixels after deformation.

[0083] Hadamard product: also known as element-wise product, is a special multiplication operation between matrices or vectors. It obtains a new vector by multiplying the elements at corresponding positions of two vectors of the same size.

[0084] At present, in order to improve the smoothness of videos, the operation of interpolating videos is widely used. In the related technology, the stream-based video frame interpolation method occupies a dominant position. It generates intermediate frame images corresponding to time nodes by estimating the motion changes between input frames to achieve the purpose of video interpolation. However, this method is mainly applied to rigid motion scenes. When faced with complex scenes, the intermediate frame images generated by it will have obvious blur and artifacts. In order to overcome the problems of blur and artifacts, some methods use generative models for frame interpolation, and use the powerful generation ability of generative models in dynamic scenes to generate intermediate frame images. However, although the generation effect of intermediate frame images in complex scenes is improved, this method needs to rely on generative models to infer the correlation between input frames, and this ability is often not fully cultivated during the model training process, resulting in the generated video being easily affected by factors such as unstable motion or incoherent input frame content, thereby reducing the quality of video generation.

[0085] Based on this, the embodiments of the present disclosure provide a video generation method, device, electronic device and storage medium, which can improve the quality of video generation.

[0086] Reference Figure 1 , Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected via a communication network.

[0087] Exemplarily, a starting frame image and an ending frame image of a target video to be generated are obtained in terminal 101, and the starting frame image and the ending frame image are sent to server 102. In server 102, a first latent feature of the starting frame image and a second latent feature of the ending frame image are extracted, a first optical flow between the starting frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the ending frame image and the intermediate frame image are extracted, the first latent feature is distorted based on the first optical flow, the second latent feature is distorted based on the second optical flow, the distorted first latent feature and the distorted second latent feature are fused to obtain a first distorted feature of the intermediate frame image, the first latent feature, the first distorted feature, and the second latent feature are respectively spliced ​​with random noise, and the resultant are input into a diffusion model for denoising to obtain a target video, the target video is sent to terminal 101, and the target video is displayed on a display page of terminal 101.

[0088] It is understandable that the above target video generation process may also be performed only in the terminal 101 .

[0089] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, server 102 can also be a node server in a blockchain network.

[0090] The terminal 101 may be a mobile phone, a computer, an intelligent voice interaction device, an intelligent wearable device, an intelligent home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0091] Reference Figure 2 , Figure 2An optional flowchart of a video generation method provided in an embodiment of the present disclosure. The video generation method can be executed by a terminal, or by a server, or by a terminal and a server in cooperation. The video generation method includes but is not limited to the following steps S201 to S204.

[0092] Step S201: obtaining a starting frame image and an ending frame image of a target video to be generated, and extracting a first latent feature of the starting frame image and a second latent feature of the ending frame image.

[0093] The target video to be generated includes a start frame image, an end frame image, and an intermediate frame image to be generated. The target video to be generated can be expressed as N is the number of video frames in the target video, 3 indicates that the number of color channels is 3, H×W is the image size of each video frame, the starting frame image and the ending frame image both contain image information, the intermediate frame image to be generated is located between the starting frame image and the ending frame image, does not contain any information, and can be regarded as a blank frame, or a placeholder between the starting frame image and the ending frame image, which is used to subsequently insert the generated intermediate frame image or feature information related to the intermediate frame image at this position; the first latent feature and the second latent feature can be encoded by a first encoder, and the first encoder can be a variational autoencoder.

[0094] Specifically, the starting frame image and the ending frame image of the target video to be generated are obtained, the starting frame image and the ending frame image are input into the first encoder for encoding, the first image feature and the second image feature of the starting frame image are extracted, the first image feature and the second image feature are mapped from the pixel space to the latent space, the first probability distribution of the first image feature in the latent space and the second probability distribution of the second image feature in the latent space are obtained, the first latent feature of the starting frame image is obtained according to the first probability distribution, and the second latent feature corresponding to the ending frame image is obtained according to the second probability distribution. By converting the starting frame image and the ending frame image into low-dimensional first latent features and second latent features respectively, this conversion process reduces the dimension of the image features while retaining the original image information of the starting frame image and the ending frame image, effectively reducing the computational complexity and computational consumption of the subsequent generation process, accelerating the speed of generating intermediate frame images, and thus improving the generation efficiency of the target video.

[0095] Step S202: extracting a first optical flow between a start frame image and an intermediate frame image to be generated in a target video, and a second optical flow between an end frame image and an intermediate frame image.

[0096] Among them, after calculation, the intermediate frame image to be generated contains the motion information of the current time step, and the motion information of the intermediate frame image is obtained based on the motion information of the starting frame image and the motion information of the ending frame image. The motion information is used to describe the motion state of the moving object in the starting frame image, the ending frame image and the intermediate frame image, which can be the motion direction, motion speed, motion distance, etc.; the first optical flow is the optical flow vector from the starting frame image to the intermediate frame image, which is used to represent the motion trajectory of each pixel point in the starting frame image to the corresponding pixel point in the intermediate frame image; the second optical flow is the optical flow vector from the ending frame image to the intermediate frame image, which is used to represent the motion trajectory of each pixel point in the ending frame image to the corresponding pixel point in the intermediate frame image.

[0097] In a possible implementation, in the process of extracting the first optical flow between the starting frame image and the intermediate frame image to be generated in the target video, and the second optical flow between the ending frame image and the intermediate frame image, the frame sequence number of the intermediate frame image to be generated in the target video can be obtained, and the starting frame image, the ending frame image and the frame sequence number are input into the optical flow prediction model for prediction, so as to obtain the first optical flow between the starting frame image and the intermediate frame image, and the second optical flow between the ending frame image and the intermediate frame image. Among them, the frame sequence number is a unique sequence number assigned to the intermediate frame image to be generated starting from the starting frame image in chronological order, and the frame sequence number corresponds to the current time step of the intermediate frame image, and can be a continuous expression such as a number. For example, if the current time step of the intermediate frame image is i (i>1), the frame sequence number is also i; the optical flow prediction model is used to predict the optical flow information of the intermediate frame image.

[0098] Specifically, the frame number of the intermediate frame image to be generated in the target video is obtained, and the start frame image, the end frame image and the frame number are input into the optical flow prediction model for prediction. In the prediction process, the start frame image and the end frame image are firstly extracted for preliminary features to obtain multi-scale image features, and then the image features of each scale are fused to obtain the first fused feature corresponding to the start frame image and the second fused feature corresponding to the end frame image. The first fused feature and the second fused feature are further extracted based on the inter-frame attention mechanism to obtain the first motion feature corresponding to the start frame image and the second motion feature corresponding to the end frame image. The total optical flow vector between the start frame image and the end frame image is obtained based on the first motion feature and the second motion feature. The total optical flow vector can provide a priori information for predicting the intermediate frame image. Then, the third motion feature of the intermediate frame image corresponding to the frame number is predicted based on the total optical flow vector and the frame number. The first optical flow between the start frame image and the intermediate frame image is obtained based on the first motion feature of the start frame image and the third motion feature of the intermediate frame image. The second optical flow between the end frame image and the intermediate frame image is obtained based on the second motion feature of the end frame image and the third motion feature of the intermediate frame image. It should also be noted that in the prediction process, in addition to extracting the motion information of the frame image, the optical flow prediction model can also extract the visual information of the frame image, such as the appearance features of the moving object in the image, the background features, etc., and obtain the optical flow vector based on the motion information and visual information. Since the optical flow prediction model can capture the complex features in the frame image, the first optical flow and the second optical flow that provide accurate motion information can be obtained through the optical flow prediction model. Based on this, the intermediate frame image is generated, which effectively improves the accuracy of the intermediate frame image. In addition, accurate optical flow information can also effectively alleviate the artifacts that may be generated when generating intermediate frame images, thereby improving the quality and stability of the generated intermediate frame images.

[0099] In addition, when the data complexity of the intermediate frame image is low or the computing resources are limited, the motion features of the intermediate frame image can also be obtained by linear interpolation. Specifically, the first motion feature of the starting frame image and the second motion feature of the ending frame image are obtained by an optical flow algorithm, and the total optical flow vector between the starting frame image and the ending frame image is calculated according to the first motion feature and the second motion feature, wherein the optical flow algorithm can be a dense optical flow method or a sparse optical flow method. The interpolation point is determined according to the frame sequence number, and the linear relationship between the interpolation point and the total optical flow vector is calculated pixel by pixel to obtain the third motion feature corresponding to the intermediate frame image, and the first optical flow between the starting frame image and the intermediate frame image is obtained according to the first motion feature of the starting frame image and the third motion feature of the intermediate frame image, and the second optical flow between the ending frame image and the intermediate frame image is obtained according to the second motion feature of the ending frame image and the third motion feature of the intermediate frame image. Since the calculation process of the linear interpolation method is relatively simple, the linear interpolation method can quickly realize optical flow estimation with less resource consumption, effectively reducing the consumption of computing resources and improving the speed of generating intermediate frame images. In addition, due to the characteristics of easy debugging and optimization of the linear interpolation method, simple parameter adjustment or combination with other optimization strategies can help further improve the accuracy of optical flow estimation, thereby improving the accuracy of generating intermediate frame images.

[0100] Step S203: distorting the first latent feature based on the first optical flow, distorting the second latent feature based on the second optical flow, fusing the distorted first latent feature and the distorted second latent feature to obtain a first distorted feature of the intermediate frame image;

[0101] Among them, the warping operation can be a reverse warping operation, which is used to map each pixel point in the intermediate frame image to be generated to the corresponding position in the starting frame image or the ending frame image; the first warped feature is a feature obtained after warping based on the first latent feature and the second latent feature, and the warping operation can be performed in the latent space.

[0102] In one possible implementation, the optical flow prediction model can be a flow-based video frame interpolation model, and the optical flow prediction model also outputs an occlusion mask corresponding to the frame number between the start frame image and the end frame image. In the process of fusing the distorted first potential feature and the distorted second potential feature to obtain the first distorted feature of the intermediate frame image, it can be specifically determined that the first Hadamard product between the occlusion mask and the distorted first potential feature, the complementary mask of the occlusion mask, the second Hadamard product between the complementary mask and the distorted second potential feature, and the first distorted feature of the intermediate frame image is obtained according to the sum of the first Hadamard product and the second Hadamard product. Among them, the occlusion mask is an image or matrix with the same size as the intermediate frame image, which is used to identify and determine the pixels that need to be distorted, and its value range is between (0,1). For example, the mask value of the area with motion information is close to 1, and the mask value of the background area is close to 0. The size and value range of the complementary mask are the same as those of the occlusion mask, and the value of the complementary mask is complementary to the value of the occlusion mask. For example, if the occlusion mask is M, then the complementary mask is 1-M.

[0103] Specifically, in the process of using the optical flow prediction model to predict the first optical flow and the second optical flow, an occlusion mask of the intermediate frame image corresponding to the frame number between the start frame image and the end frame image is also generated. Therefore, the output of the optical flow prediction model is the first optical flow, the second optical flow and the occlusion mask of the intermediate frame image corresponding to the frame number. The above process of using the optical flow prediction model for prediction can be expressed by the following formula:

[0104] f 1→n ,f N→n ,M n = Ο(x 1 ,x N ,n),

[0105] Among them, Ο(·) represents the optical flow prediction model, x 1 Represents the starting frame image. The feature dimension of the starting frame image can be x N Represents the end frame image, and the feature dimension of the end frame image can be n represents the frame number, f 1→n Represents the first optical flow, and the feature dimension of the first optical flow can be f N→n Represents the second optical flow, and the feature dimension of the second optical flow can be M n Represents the occlusion mask of the nth intermediate frame image. The matrix dimension of the occlusion mask can be

[0106] Next, the first latent feature is distorted based on the first optical flow. For each pixel in the intermediate frame image, each pixel in the intermediate frame image is mapped back to the corresponding position in the starting frame image based on the first optical flow to obtain the distorted first latent feature. Similarly, the distorted second latent feature is obtained. The distorted first latent feature is Hadamard producted with the occlusion mask to obtain the first Hadamard product, and the distorted second latent feature is Hadamard producted with the complementary mask to obtain the second Hadamard product. The first Hadamard product is added to the second Hadamard product to obtain the first distorted feature of the intermediate frame image. According to the above description, the first distorted feature of the intermediate frame image can be obtained by the following formula:

[0107]

[0108] Where warp(·) represents the warping operation, M n Represents the occlusion mask of the nth intermediate frame image, 1-M n represents the complementary mask of the nth intermediate frame image, ⊙ represents the Hadamard product, represents the first potential feature, Represents the second potential feature. By generating an occlusion mask and performing a reverse warping operation based on the occlusion mask, the motion information in the start frame image and the end frame image can be more accurately aggregated into the corresponding motion information of the intermediate frame image, thereby improving the coherence between the motion information of the intermediate frame image and the motion information of the start frame image and the motion information of the end frame image. The transition between the intermediate frame image generated based on this and the start frame image and the end frame image is more natural and smooth, thereby improving the smoothness of the target video.

[0109] In addition, the optical flow prediction model can also be to gradually approximate the real optical flow by iteratively refining the initial optical flow estimation, such as the IRR (Iterative Residual Refinement) model, or to learn a rough occlusion mask through an asymmetric occlusion-aware matching module to filter out the occluded area to improve the accuracy of the optical flow estimation, such as MaskFlowNet. Different optical flow prediction models can be selected to predict the optical flow according to the actual application scenario, and this application does not specifically limit the optical flow prediction model.

[0110] Step S204: After splicing the first latent feature, the first distorted feature, and the second latent feature with random noise respectively, the splicing is input into the diffusion model for denoising to obtain the target video.

[0111] According to the above steps, the starting frame image and the ending frame image of the target video to be generated are obtained, the first potential feature of the starting frame image and the second potential feature of the ending frame image are extracted, and the starting frame image and the ending frame image are input into the optical flow prediction model to output the first optical flow between the starting frame image and the intermediate frame image, the second optical flow between the ending frame image and the intermediate frame image, and the occlusion mask corresponding to the intermediate frame image. The first potential feature is reversely distorted based on the first optical flow, and the second potential feature is reversely distorted based on the second optical flow. The distorted first potential feature and the distorted second potential feature are fused to obtain the first distorted feature of the intermediate frame image. After splicing random noises, the first latent feature, the first distorted feature, and the second latent feature are respectively input into the denoising module of the diffusion model for denoising. In the processing block of the denoising module, the input of the processing block is mapped through the time layer to obtain spatial features, and the first spatial features of the starting frame image and the second spatial features of the ending frame image are extracted from the spatial features. The first spatial features are reversely distorted based on the first optical flow, and the second spatial features are reversely distorted based on the second optical flow. The distorted first spatial features and the distorted second spatial features are fused to obtain the second distorted features of the intermediate frame image. The spatial features are input into the time layer for mapping to obtain the mapping features, and the second distorted features are fused with the mapping features to obtain the output of the processing layer. The target features output from the last processing layer of the denoising module are input into the first decoder for decoding to obtain the target video.

[0112] Among them, the diffusion model includes a first encoder, a noise generation module, a denoising module and a first decoder. The noise generation module is used to add noise to all frame images during the training process, and splice the added noise with the potential features corresponding to the frame images; the denoising module is used to denoise the features with noise and output the target features. The denoising module can be a UNet network target feature that is a noise-free feature output after being restored by the denoising module of the diffusion model; the target video is the video obtained after the first decoder decodes the target feature.

[0113] In the process of respectively splicing the first latent feature, the first distorted feature and the second latent feature with random noise, specifically, referring to Figure 3 , Figure 3 An optional schematic diagram of a noise generation module provided in an embodiment of the present disclosure, based on the potential features of the starting frame image and the potential features of the end frame image Perform reverse distortion to obtain the potential features corresponding to the three intermediate frame images Potential features Add noise step by step to get the potential features The corresponding noise characteristics Potential features The corresponding noise characteristics Potential features The corresponding noise characteristics Potential features The corresponding noise characteristics Potential features The corresponding noise characteristics Among them, t is the time step required in the process of adding noise, and the noise characteristic All obey pure Gaussian distribution. Finally, the potential features and noisy features To stitch together the potential features and noisy features To stitch together the potential features and noisy features To stitch together the potential features and noisy features To stitch together the potential features and noisy features Splice and get the noisy feature z corresponding to the target video t After adding noise step by step, we get the noisy feature z t The process can be expressed by the following formula:

[0114]

[0115] Among them, t is the time step required in the process of adding noise, z 0 represents the latent feature without adding noise, the latent feature z 0 The characteristic dimension of C is the number of channels, h×w is the size of the feature map of each video frame in the latent space, and z t represents the potential feature after adding noise at the current time step t, ∈ represents the noise added at the current time step t, It means that the added noise follows the normal distribution (0, N), where N is the total number of frame images in the target video.

[0116] In one possible implementation, the diffusion model includes a plurality of sequentially cascaded processing blocks for denoising, each processing block includes an interconnected spatial layer and a temporal layer. After the first latent feature, the first distorted feature, and the second latent feature are respectively spliced ​​with random noise, they are input into the diffusion model for denoising to obtain the target video. Specifically, the first latent feature, the first distorted feature, and the second latent feature are respectively spliced ​​with level noise and then input into the diffusion model. For each processing block, the input of the processing block is mapped in turn through the spatial layer and the temporal layer to obtain the output of the processing block, and the target video is obtained based on the output of the last processing block. Among them, the denoising module of the diffusion model includes a plurality of sequentially cascaded processing blocks for denoising, and the processing blocks include shrinkage blocks, intermediate blocks and expansion blocks. The shrinkage blocks are used to gradually reduce the spatial dimensions of the input features, and the shrinkage blocks can be regarded as processing blocks for downsampling operations; the intermediate blocks are located between the shrinkage blocks and the expansion blocks, and are used to connect the shrinkage blocks and the expansion blocks, and can be composed of one or more convolutional layers; the expansion blocks are used to gradually restore the low-dimensional features obtained from the shrinkage blocks to the same dimension as the input features, and the expansion blocks can be regarded as processing blocks for upsampling operations; the spatial layer is used to model the spatial information contained in the input features, and is composed of at least one sub-processing block, and the sub-processing blocks can be processing blocks of the same type or different types. For example, the sub-processing blocks can be composed of residual blocks with a spatial attention mechanism and Transformer blocks with a spatial attention mechanism; the temporal layer is used to map the output of the spatial layer, and can be composed of at least one Transformer block with a temporal attention mechanism.

[0117] Specifically, the corresponding noise is randomly generated according to the feature distribution of the first latent feature, the first distorted feature and the second latent feature, and the first latent feature, the first distorted feature and the second latent feature are respectively concatenated with the corresponding random noise to obtain a noisy feature, and the noisy feature, the first latent feature and the second latent feature are input into the denoising module of the diffusion model for denoising, wherein the first latent feature, the first distorted feature and the second latent feature are used as additional conditional information to guide the denoising module to perform noise prediction. Input of the denoising module It can be expressed as,

[0118]

[0119] Among them, z t represents the noisy feature, Represents the first potential feature of the starting frame image The first warped feature of the middle frame image and the second latent feature of the end frame image The set of denoising modules The characteristic dimension of

[0120] During the inference process, for the first processing block of the denoising module, the noisy features pass through the spatial layer and the temporal layer in sequence. The spatial information of the input features is modeled in the spatial layer to obtain the spatial features. The spatial features output by the spatial layer are mapped in the temporal layer to obtain the first intermediate output of the first processing block. The first output is used as the input of the next processing block. The first intermediate output is feature mapped in the next processing block to obtain the second intermediate output. In each cascaded processing block, feature mapping is performed based on the output of the previous processing block to obtain the first output of the last processing block, completing the noise prediction and denoising of the current time step. The noisy features are iteratively restored based on the first latent features, the second latent features, and the pure noise until the target features of the output of the last processing block at the last time step (i.e., the maximum number of iterations is reached) are obtained. The target features are input into the first decoder for decoding to generate the target video. Generate target video The process can be expressed as in, represents the target feature, and D(·) represents the first decoder.

[0121] Since the diffusion model can gradually smooth the noise by simulating the diffusion process of the noise, by using the diffusion model for denoising, a clearer intermediate frame image can be generated. Furthermore, in the denoising module of the diffusion model, the input features are mapped sequentially through the spatial layer and the temporal layer, and the captured spatial features can be smoothed and aligned, which further improves the denoising ability of the diffusion model, thereby improving the quality of the generated intermediate frame image.

[0122] Reference Figure 4 , Figure 4 This is an optional schematic diagram of a denoising module provided in an embodiment of the present disclosure. The denoising module is composed of a contraction block, an expansion block and an intermediate block. The contraction block is configured with a spatial layer, a temporal layer and a smoothing layer, and the expansion block is configured with a spatial layer and a temporal layer. t Input to the denoising module for denoising, with noisy feature z t First, it is input into the cascaded shrinkage blocks, and the downsampling operation is performed based on the first optical flow, the second optical flow and the occlusion mask to obtain the first intermediate feature f output by the last shrinkage block. 1 , the first intermediate feature f 1 After the intermediate block, the second intermediate feature f is obtained 2 , and then the second intermediate feature f 2 The input is sent to multiple sequentially cascaded expansion blocks for upsampling operations, and finally the output of the denoising module is obtained. During the denoising process, the denoising module performs iterative denoising according to the time step t.

[0123] It should also be noted that the smoothing layer can be configured only in the contraction block, or can also be configured in the contraction block and the expansion block at the same time, and the smoothing layer can be configured according to actual conditions. In addition, the intermediate block in the denoising module is not a must-use structure, and whether to use the intermediate block in the contraction block and the expansion block can be selected according to actual conditions.

[0124] In a possible implementation, in the process of mapping the input of the processing block through the spatial layer and the temporal layer to obtain the output of the processing block, the input of the processing block may be mapped through the spatial layer, a first spatial feature corresponding to the start frame image and a second spatial feature corresponding to the end frame image are extracted from the output of the spatial layer, the first spatial feature is distorted based on the first optical flow, the second spatial feature is distorted based on the second optical flow, the distorted first spatial feature and the distorted second spatial feature are fused to obtain the second distorted feature of the intermediate frame image. The output of the spatial layer is mapped through the temporal layer, the output of the temporal layer is fused with the second distorted feature to obtain the output of the processing block. The first spatial feature is the spatial feature corresponding to the start frame image, the second spatial feature is the spatial feature corresponding to the start frame image, the first spatial feature and the second spatial feature are used to perform an inverse distortion operation at the feature level; the second distorted feature is a predicted feature of the intermediate frame image under relevant prior knowledge, and the relevant prior knowledge is the first spatial feature and the second spatial feature.

[0125] Specifically, for the first processing block, the input feature of the first processing block is a noisy feature. The noisy feature is input into the spatial layer, and the spatial information in the noisy feature is modeled through multiple sub-processing blocks to obtain a spatial feature containing rich spatial information. The spatial feature includes features corresponding to the start frame image, the end frame image, and each intermediate frame image. The first spatial feature corresponding to the start frame image and the second spatial feature corresponding to the end frame image are extracted from the spatial feature. The first spatial feature is reversely warped based on the first optical flow, and the spatial feature of each pixel in the intermediate frame image is mapped to the corresponding position in the start frame image to obtain the warped first spatial feature. Similarly, the second spatial feature is reversely warped based on the second optical flow to obtain the warped second spatial feature. The warped first spatial feature and the warped second spatial feature are feature fused to obtain the second warped feature of the intermediate frame image. Then, the spatial feature is input into the time layer for mapping to obtain the mapping feature output by the time layer. The mapping feature output by the time layer and the second warped feature are feature fused by calculating the mean to obtain the output of the first processing block. For other processing blocks except the first processing block, their input is the output of the previous processing block. After the above process, the output of each processing block is obtained. It can be obtained by the following formula,

[0126]

[0127] Among them, F is the mapping feature obtained by mapping the spatial features output by the spatial layer by the time layer. is the second warped feature. By reversely warping the first spatial feature and the second spatial feature, the motion information of each video frame can be integrated based on different granularities, thereby improving the consistency of motion information between adjacent frames, making the generated intermediate frame image visually smoother, thereby improving the quality of the target video. Furthermore, by adopting a mean fusion strategy for the output of the temporal layer and the second warped feature, the guiding relationship between the first optical flow, the second optical flow and the occlusion mask is effectively utilized, and the denoising ability of the denoising module of the diffusion model can be improved without occupying too many computing resources. At the same time, the simple feature fusion method also enables the denoising module to adapt to complex and diverse input features, thereby improving the generalization and processing speed of the denoising module.

[0128] Reference Figure 5 , Figure 5 An optional schematic diagram of a processing block structure provided in an embodiment of the present disclosure, wherein the input of the processing block is a noisy feature The noisy features Input into the spatial layer for spatial information modeling, and obtain the spatial features corresponding to each frame image f = {f 1 ,...,f N}, extract the spatial features f of the starting frame image from it 1 and the spatial features f of the end frame image N ,, based on the first optical flow, the second optical flow and the occlusion mask, the spatial feature f 1 and spatial characteristics f N Perform reverse distortion respectively to obtain the distortion characteristics of the intermediate frame image Then input the spatial feature f into the time layer for mapping, and obtain the mapping feature F, and transform the distorted feature Perform feature fusion with the mapping feature F to obtain the output of the processing block

[0129] In a possible implementation, in the process of fusing the distorted first spatial feature and the distorted second spatial feature to obtain the second distorted feature of the intermediate frame image, specifically, the occlusion mask corresponding to the frame number between the start frame image and the end frame image is obtained, the third Hadamard product between the occlusion mask and the distorted first spatial feature is determined, the complementary mask of the occlusion mask is determined, the fourth Hadamard product between the complementary mask and the distorted second spatial feature is determined, and the second distorted feature of the intermediate frame image is obtained according to the sum of the third Hadamard product and the fourth Hadamard product. Specifically, the occlusion mask of the intermediate frame image corresponding to the frame number is obtained, and the corresponding complementary mask is obtained based on the occlusion mask. The distorted first spatial feature is Hadamard-producted with the occlusion mask to obtain the third Hadamard product, the distorted second spatial feature is Hadamard-producted with the complementary mask to obtain the fourth Hadamard product, and the third Hadamard product and the fourth Hadamard product are added to obtain the second distorted feature of the intermediate frame image corresponding to the frame number. According to the above description, the second distorted feature F of the intermediate frame image with frame number i i It can be obtained by the following formula:

[0130]

[0131] Among them, F 1 is the first spatial feature of the starting frame image, and the feature dimension of the first spatial feature can be D is the depth, F is the N To end the second spatial feature of the frame image, the feature dimension of the second spatial feature can be f 1-i represents the first optical flow, f N-i Represents the second optical flow. By reversely warping the first spatial feature and the second spatial feature at the feature level, and performing a Hadamard product between the reverse warping result and the occlusion mask, the second warped feature retains the key spatial information of the first spatial feature and the second spatial feature while maintaining motion consistency with the first spatial feature and the second spatial feature, making the intermediate frame image generated based on the second warped feature more coherent, thereby enhancing the overall fluency and authenticity of the target video.

[0132] In a possible implementation, the processing block is further provided with a smoothing layer, which is obtained by copying the parameters of the time layer, and then fusing the output of the time layer with the second warped feature to obtain the output of the processing block. Specifically, the second warped feature is mapped through the smoothing layer, and the output of the time layer is fused with the mapped second warped feature to obtain the output of the processing block. The parameters of the time layer are used to control and adjust the characteristic attributes of the output of the time layer, such as the characteristic distribution and characteristic dimension, and the parameters may be weight parameters.

[0133] Specifically, the second distorted feature is smoothed by the smoothing layer, and the feature distribution of the smoothed second distorted feature is adjusted so that the feature distribution of the adjusted second distorted feature is aligned with the feature distribution of the output of the time layer. The process of mapping the second distorted feature by the smoothing layer can be expressed by the following formula:

[0134]

[0135] in, represents the second distorted feature after the smoothing layer, and F represents the second distorted feature without the smoothing layer. Then, the output of the time layer is fused with the adjusted second distorted feature. The fusion method can be to calculate the mean of the output of the time layer and the adjusted second distorted feature to obtain the output of the processing block. By mapping the second distorted feature through the smoothing layer, the feature distribution of the second distorted feature and the output of the time layer can be smoothed and aligned without introducing additional parameters, thereby avoiding the risk of overfitting due to the increase in model complexity and ensuring the stability of the denoising module. In addition, the smooth mapping mainly depends on the inherent laws of the second distorted feature itself and the interdependence between features. Therefore, the generalization and denoising capabilities of the denoising module can be further improved by optimizing the feature distribution.

[0136] Further, the processing block includes a spatial layer, a temporal layer and a smoothing layer, referring to Figure 6 , Figure 6 Another optional schematic diagram of the processing block structure provided in the embodiment of the present disclosure, the input of the processing block is a noisy feature The noisy features Input into the spatial layer for spatial information modeling, and obtain the spatial features corresponding to each frame image f = {f 1 ,...,f N}, extract the spatial features f of the starting frame image from it 1 and the spatial features f of the end frame image N , based on the first optical flow, the second optical flow and the occlusion mask for the spatial feature f 1 and spatial characteristics f N Warp them separately to get the distortion features of the intermediate frame image Then input the spatial feature f into the time layer for mapping to obtain the mapping feature F. Copy the parameters of the time layer to obtain the smoothing layer, input the distortion feature F into the smoothing layer for mapping to obtain the smoothing feature Smooth features Perform feature fusion with the mapping feature F to obtain the output of the processing block

[0137] In a possible implementation, in the process of mapping the output of the spatial layer through the temporal layer, the constraint text for constraining the video content of the target video can be obtained, the text features of the constraint text can be extracted, the output of the spatial layer can be processed by self-attention through the temporal layer to obtain a first attention feature, a cross-attention process can be performed based on the text feature and the first attention feature to obtain a second attention feature, and the second attention feature can be mapped to obtain the output of the temporal layer. Among them, the constraint text is used as a guiding condition to describe the text of the specific content in the target video, which can be a text paragraph, a short sentence or a keyword; the text feature is a feature extracted from the constraint text by the text encoder, which can be a global text feature used to describe the nature or pattern of the constraint text, or a context feature used to describe the association between each word or phrase in the constraint text, or a mixed feature including global text features and context features; the first attention feature is a feature after self-attention processing, and the second attention feature is a feature after cross-attention processing.

[0138] Specifically, a constraint text for constraining the video content of the target video is obtained, and the constraint text is input into a text encoder for feature extraction to obtain text features of the constraint text, wherein the text encoder may be a CLIP encoder. Next, the spatial features output by the spatial layer are input into the temporal layer for self-attention processing to obtain a first attention feature. A query matrix is ​​constructed based on the first attention feature, a value matrix and a key matrix are constructed based on the text features, and a cross-attention operation is performed on the query matrix, the value matrix, and the key matrix to obtain a second attention feature. Next, the second attention feature is mapped to obtain the output of the temporal layer. Refer to Figure 7 , Figure 7 An optional schematic diagram of a temporal layer structure provided in an embodiment of the present disclosure, wherein the spatial layer outputs spatial features f s , based on the text encoder, the text feature f corresponding to the constrained text is obtained t , then the text feature f t is a hybrid feature, including global text features and context features. t and spatial characteristics f s Input to the time layer, and the spatial feature f is analyzed in the self-attention layer of the time layer s Perform feature extraction to obtain the attention feature f 1 Based on the attention feature f 1 Construct the query matrix Q based on the text feature f t Construct the value matrix V and the key matrix K, input the query matrix Q, the value matrix V and the key matrix K into the cross attention layer for feature extraction, and obtain the attention feature f 2 . The attention feature f 2The input is mapped to the feedforward network to obtain the output F of the time layer. By performing cross-attention processing on text features and spatial features, the cross-attention mechanism can capture the intrinsic connection between text features and image features, which helps to deepen the degree of information interaction between text features and image features, and enables the denoising module of the diffusion model to more comprehensively understand the complex relationship between the features of the two modalities, thereby improving the accuracy of the target features output by the denoising module, and then improving the quality of the generated target video.

[0139] Alternatively, a two-way cross-attention strategy can be used to perform cross-attention processing on global text features and context features respectively, referring to Figure 8 , Figure 8 Another optional schematic diagram of the temporal layer structure provided in the embodiment of the present disclosure, wherein the spatial layer outputs the spatial feature f s , based on the text encoder, the text feature f corresponding to the constrained text is obtained t , from the text feature f t The global text feature f is extracted from text and context feature f context . The spatial feature f s , global text feature f text and context feature f context Input to the time layer, and the spatial feature f is analyzed in the self-attention layer of the time layer s Perform feature extraction to obtain the first attention feature f 1 Based on the first attention feature f 1 Construct the query matrix Q based on the global text feature f text Construct the first value matrix V 1 and the first bond matrix K 1 , the query matrix Q, the first value matrix V 1 and the first bond matrix K 1 Input to the cross attention layer 1 for feature extraction to obtain the second attention feature f 2 Based on the context feature f context Construct the second value matrix V 2 and the first bond matrix K 2 , the query matrix Q, the second value matrix V 2 and the first bond matrix K 2 Input to the cross attention layer 2 for feature extraction to obtain the third attention feature f 3 . The third attention feature f 3 Input to the function gating module to obtain the fusion coefficient λ, which is used to promote the fusion between contextual features and spatial features, and obtain the adjusted third attention feature λf 3 , the adjusted third attention feature λf 3and the second attention feature f 2 Perform feature fusion to obtain text fusion feature f 4 , fusion feature f 4 The input is mapped into the feedforward network to obtain the output F of the time layer. By performing cross-attention processing on the global text features and context features respectively, the text features can be better aligned with the spatial features, so that the denoising module can capture the features of the frame image from multiple aspects, thereby improving the accuracy of the denoising module in predicting noise.

[0140] In addition to using constraint text as a constraint condition, frame frequency can also be used as a constraint condition. Different from the constraint text acting on the content of the target video, frame frequency is mainly used to constrain the magnitude of the motion amplitude between two adjacent frames, which can be understood as constraining the number of generated intermediate frame images. For example, for the same pair of start frame images and end frame images, the more the number of intermediate frame images, the smaller the difference in motion amplitude between two adjacent frames; when the number of intermediate frame images is less, the greater the difference in motion amplitude between two adjacent frames.

[0141] In a possible implementation, the diffusion model is trained by the following steps, specifically, obtaining a sample video, extracting a first sample feature of a first sample image, a second sample feature of a second sample image, and a third sample feature of a third sample image, respectively adding sample noise to the first sample feature, the second sample feature, and the third sample feature, to obtain a first noise feature, a second noise feature, and a third noise feature, respectively, extracting a third optical flow between the first sample image and the third sample image, and a fourth optical flow between the second sample image and the third sample image, distorting the first sample feature based on the third optical flow, distorting the second sample feature based on the fourth optical flow, and fusing the distorted first sample feature and the distorted second sample feature to obtain a third distorted feature of the third sample image. The sample video includes a first sample image located at a starting frame, a second sample image located at an ending frame, and a third sample image located between the first sample image and the second sample image, and the first sample image, the second sample image, and the third sample image are all sample images containing original image information; the sample noise is noise gradually added to the sample image, which may be noise randomly added according to the feature distribution of the sample image, or may be noise preset.

[0142] Specifically, a first sample image located at a starting frame image, a second sample image located at an ending frame image, and a third sample image located between the first sample image and the second sample image in a sample video are obtained. The first sample image, the second sample image, and the third sample image are input into a first encoder, and the first sample image, the second sample image, and the third sample image are mapped to a latent space to obtain a first sample feature of the first sample image, a second sample feature of the second sample image, and a third sample feature of the third sample image. Sample noise is gradually added to the first sample feature, the second sample feature, and the third sample feature through a noise generation module of a diffusion model until the added sample noise obeys a pure Gaussian distribution, thereby obtaining a first noise feature corresponding to the first sample feature, a second noise feature corresponding to the second sample feature, and a third noise feature corresponding to the third sample feature. Next, the first sample image, the second sample image and the third sample image are input into the optical flow prediction model to obtain a third optical flow between the first sample image and the third sample image, and a fourth optical flow between the second sample image and the third sample image. A reverse warping operation is performed on the first sample feature based on the third optical flow, and a reverse warping operation is performed on the second sample based on the fourth optical flow. The warped first sample feature and the warped second sample feature are feature fused to obtain a third warped feature of the third sample image.

[0143] Next, the first noise feature is concatenated with the first sample feature to obtain a first concatenated feature, the second noise feature is concatenated with the second sample feature to obtain a second concatenated feature, the third noise feature is concatenated with the third distortion feature to obtain a third concatenated feature, the first concatenated feature, the second concatenated feature and the third concatenated feature are input into the diffusion model for denoising, the predicted noise output by the diffusion model during the denoising process is obtained, the model loss is determined according to the difference between the predicted noise and the sample noise, and the diffusion model is trained based on the model loss. Specifically, the first noise feature is concatenated with the first sample feature to obtain a first concatenated feature, the second noise feature is concatenated with the second sample feature to obtain a second concatenated feature, and the third noise feature and the third distortion feature are concatenated to obtain a third concatenated feature. The first splicing feature, the second splicing feature, the third splicing feature, the first sample feature and the second sample feature are input into the denoising module of the diffusion model for denoising, and the predicted noise of each time step output by the denoising module during the denoising process is obtained. The model loss is determined according to the difference between the predicted noise of each time step and the sample noise of the corresponding time step, and the training direction of the diffusion model is guided based on the model loss. The model loss can be expressed by the following formula:

[0144]

[0145] Where E[·] represents the expected function, z 0represents the sample features without adding noise, t represents the time step, ∈ represents the sample noise, ∈ θ (·) represents the prediction noise, Represents the input of the denoising module, and c represents the constraint condition, which can be text information or frame frequency. By training the diffusion model, the denoising module of the diffusion model can learn the data distribution and potential rules of noisy features, thereby effectively removing the noise step by step. Furthermore, by determining the model loss by the difference between the predicted noise and the sample noise, the training effect of the diffusion model can be quantified according to the model loss, thereby guiding the optimization direction of the diffusion model, thereby improving the noise prediction ability of the denoising module of the diffusion model, and thus improving the quality of the generated target video.

[0146] In a possible implementation, the diffusion model includes a plurality of sequentially cascaded processing blocks for denoising, each processing block includes an interconnected spatial layer and a temporal layer, and in the process of inputting the first splicing feature, the second splicing feature and the third splicing feature into the diffusion model for denoising, specifically, the first splicing feature, the second splicing feature and the third splicing feature are input into the diffusion model, for each processing block, the input of the processing block is mapped through the spatial layer, the third spatial feature corresponding to the first sample image and the fourth spatial feature of the second sample image are extracted from the output of the spatial layer, the third spatial feature is distorted based on the third optical flow, the fourth spatial feature is distorted based on the fourth optical flow, the distorted third spatial feature and the distorted fourth spatial feature are fused to obtain a fourth distorted feature of the third sample image, the output of the spatial layer is mapped through the temporal layer, the output of the temporal layer is fused with the fourth distorted feature, and the output of the processing block is obtained.

[0147] Specifically, in the process of training the diffusion model, the first splicing feature, the second splicing feature, the third splicing feature, the first sample feature and the second sample feature are input into the denoising module of the diffusion model for noise prediction. For each processing block, the spatial information modeling is performed on the input of the processing block through the spatial layer to obtain the corresponding spatial feature, and the third spatial feature corresponding to the first sample image and the fourth spatial feature of the second sample image are extracted from the spatial feature. The third spatial feature is reversely distorted based on the third optical flow, and the fourth spatial feature is distorted based on the fourth optical flow. The distorted third spatial feature and the distorted fourth spatial feature are fused to obtain the fourth distorted feature of the third sample image. The spatial feature is input into the time layer for mapping to obtain the mapping feature, and the fourth distorted feature is mean-fused with the mapping feature to obtain the output of the processing block. By training the processing block of the denoising module in the diffusion model, the processing block can learn the noise distribution of the noisy feature, which helps to improve the ability of the diffusion model to predict noise.

[0148] In a possible implementation, after obtaining a pre-trained diffusion model, the diffusion model is fine-tuned. In the process of training the diffusion model based on the model loss, the parameters of the time layer in each processing block can be frozen, and the parameters of the space layer in each processing block can be adjusted based on the model loss to train the diffusion model. Specifically, the parameters of the time layer and the parameters of the space layer can be weight parameters, which are used to control and adjust the output features of the time layer and the space layer. After training the diffusion model, the pre-trained diffusion model is obtained, the parameters of the time layer in each processing block of the denoising module in the pre-trained diffusion model are frozen, the gradient of the time layer in each processing block is calculated based on the training loss, the parameters of the time layer are updated based on the gradient, and the pre-trained denoising module is fine-tuned based on the updated parameters of the space layer until a preset iteration round is reached. By freezing the parameters of the time layer and only fine-tuning the parameters of the space layer to compensate for the changes in the feature distribution of the spatial features, the spatial information modeling capability of the spatial layer can be optimized, so that the denoising module can better capture the spatial information in the noisy features, thereby improving the accuracy of the denoising module in predicting noise. In addition, only fine-tuning the parameters of the spatial layer can reduce the computational consumption of the diffusion model during the training process, so that the model training can be performed without occupying too many computing resources, thereby improving the efficiency of model training.

[0149] Reference Fig. 9 , Fig. 9 An optional overall flow chart of the video generation method provided in the embodiment of the present disclosure is provided below. The principle of the video generation method in the embodiment of the present disclosure is fully described in general as follows:

[0150] The video generation method provided by the embodiment of the present disclosure can be implemented by a combination of an optical flow prediction model and a diffusion model, wherein the diffusion model includes a first encoder, a noise generation module, a denoising module and a first decoder, and the noise generation module is mainly used to add noise to the video frames during the training process, and splice the added noise with the potential features corresponding to the video frames.

[0151] Before generating the target video, the diffusion model needs to be trained. First, the starting frame sample image and the ending frame sample image are input into the optical flow prediction model to obtain the third optical flow between the starting frame sample image and the intermediate frame sample image, and the fourth optical flow between the ending frame sample image and the intermediate frame sample image. During the diffusion model training process, the first sample feature of the starting frame sample image, the second sample feature of the ending frame sample image, and the third sample feature of the intermediate frame sample image are extracted by the first encoder. In the noise generation module, sample noise is gradually added to the first sample feature, the second sample feature, and the third sample feature, the first sample feature is distorted based on the third optical flow, the second sample feature is distorted based on the fourth optical flow, the distorted first sample feature and the distorted second sample feature are fused, and the third distorted feature of the intermediate frame sample image is obtained. The first sample feature, the second sample feature, and the third distorted feature are spliced ​​with the corresponding generated noise respectively to obtain the noisy feature, and the noisy feature is input into the denoising module for noise prediction, and the model loss is determined according to the difference between the predicted noise and the sample noise, and the training direction of the diffusion model is guided based on the model loss. Next, after obtaining the pre-trained diffusion model, the pre-trained diffusion model is fine-tuned by freezing the parameters of the temporal layer in the denoising module and adjusting the parameters of the spatial layer based on the model loss to fine-tune the diffusion model.

[0152] After completing the training, the starting frame image and the ending frame image of the target video to be generated are obtained, and the starting frame image and the ending frame image are input into the optical flow prediction model to obtain the first optical flow between the starting frame image and the intermediate frame image, the second optical flow between the ending frame image and the intermediate frame image, and the occlusion mask corresponding to the intermediate frame image. The first optical flow, the second optical flow and the occlusion mask are introduced into the diffusion model as motion guidance information.

[0153] Next, the start frame image and the end frame image are input into the first encoder, and the start frame image and the end frame image are converted from the pixel space to the latent space to obtain the first latent feature of the start frame image and the second latent feature of the end frame image. The first latent feature and the second latent feature are reversely warped based on the motion guidance information to obtain the third latent feature corresponding to the intermediate frame image. The first latent feature, the second latent feature and the third latent feature are input into the denoising module for noise prediction, and denoising is performed based on the predicted noise. In addition, constraint text or frame frequency can be introduced as a constraint condition for more accurate denoising.

[0154] The denoising module includes a plurality of processing blocks cascaded in sequence. In the process of noise prediction by the denoising module, for any processing block in the denoising module, the input of the processing block is mapped through the spatial layer to obtain spatial features, and the spatial features include spatial features corresponding to the starting frame image, the ending frame image and the intermediate frame image. The first spatial feature of the starting frame image and the second spatial feature of the ending frame image are extracted from the spatial features. The first spatial feature and the second spatial feature are reversely distorted based on the motion guidance information to obtain the prediction feature corresponding to the intermediate frame image. The spatial feature is input into the time layer for mapping to obtain the mapping feature, and the feature distribution of the prediction feature, the first spatial feature, the second spatial feature and the mapping feature is smoothed and aligned to obtain the smoothed feature, and the mapping feature and the smoothed feature are mean-fused to obtain the output of the processing block.

[0155] Then, after completing the predetermined iteration rounds, the denoising module restores the noisy features to obtain the target features, and inputs the target features into the first decoder for decoding to obtain the target video, thus completing one video generation.

[0156] The video generation method provided by the embodiment of the present disclosure extracts the optical flow between the start frame image and the intermediate frame image and the optical flow between the end frame image and the intermediate frame image through the optical flow prediction model, which can effectively utilize the dynamic information between the start frame image and the end frame image, so that the distortion features obtained by the reverse distortion operation based on this can effectively guide the motion reasoning in the generation process. In addition, by performing the reverse distortion operation at the potential level (latent features), it is beneficial to maintain the consistency of background information, and by performing the reverse distortion operation at the feature level (spatial features), the consistency of motion information of adjacent frames can be improved. Combining the fusion features of the potential level and the feature level, the action guidance information of different granularities can be used to promote the generation of intermediate frames, thereby improving the motion stability and content consistency of the generated target video, and improving the quality of video generation.

[0157] In one possible implementation, the video generation method provided by the embodiment of the present disclosure can be applied to game development scenarios. Specifically, the action content of the game character is determined, the starting state of the action is used as the starting frame, and the ending state of the action is used as the ending frame. The starting frame and the ending frame are input into the video generation model, and the action guidance information of the action content (the optical flow between the starting frame and the middle frame, the optical flow between the ending frame and the middle frame, and the occlusion mask corresponding to the middle frame) is obtained through the optical flow prediction model. In the generation model, a first reverse warping operation is performed based on the action guidance information to obtain the distortion feature corresponding to the middle frame. The distortion information corresponding to the starting frame, the ending frame, and the middle frame is input into the generation model for denoising. A second reverse warping operation is performed during the denoising process to finally obtain the target feature that does not contain noise, and the target feature is decoded to obtain complete and smooth action content.

[0158] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0159] Reference Fig.10 , Fig.10 This is an optional structural diagram of a video generating device provided in an embodiment of the present disclosure. The video generating device 1000 can execute the video generating method in any of the above embodiments. The device includes:

[0160] The feature extraction module 1001 is used to obtain the starting frame image and the ending frame image of the target video to be generated, and extract the first potential feature of the starting frame image and the second potential feature of the ending frame image;

[0161] The optical flow extraction module 1002 is used to extract a first optical flow between a start frame image and an intermediate frame image to be generated in a target video, and a second optical flow between an end frame image and an intermediate frame image;

[0162] A feature change module 1003 is used to distort the first potential feature based on the first optical flow, distort the second potential feature based on the second optical flow, and fuse the distorted first potential feature and the distorted second potential feature to obtain a first distorted feature of the intermediate frame image;

[0163] The generating module 1004 is used to concatenate the first latent feature, the first distorted feature, and the second latent feature with random noise respectively, and then input the concatenated features into the diffusion model for denoising to obtain a target video.

[0164] Furthermore, the optical flow extraction module 1002 is also used for:

[0165] Obtain the frame number of the intermediate frame image to be generated in the target video;

[0166] The starting frame image, the ending frame image and the frame sequence number are input into the optical flow prediction model for prediction, and the first optical flow between the starting frame image and the middle frame image and the second optical flow between the ending frame image and the middle frame image are obtained.

[0167] Furthermore, the feature change module 1003 is also used for:

[0168] determining a first Hadamard product between the occlusion mask and the warped first latent feature;

[0169] determining a complementary mask of the occlusion mask, and determining a second Hadamard product between the complementary mask and the distorted second latent feature;

[0170] A first distortion feature of the intermediate frame image is obtained according to the sum of the first Hadamard product and the second Hadamard product.

[0171] Further, the generating module 1004 is also used for:

[0172] The first latent feature, the first distorted feature, and the second latent feature are respectively concatenated with random noise and input into the diffusion model;

[0173] For each processing block, the input of the processing block is mapped in turn through the spatial layer and the temporal layer to obtain the output of the processing block;

[0174] The target video is obtained based on the output of the last processing block.

[0175] Further, the generating module 1004 is also used for:

[0176] Mapping the input of the processing block through the spatial layer, extracting the first spatial feature corresponding to the start frame image and the second spatial feature corresponding to the end frame image from the output of the spatial layer;

[0177] The first spatial feature is distorted based on the first optical flow, the second spatial feature is distorted based on the second optical flow, and the distorted first spatial feature and the distorted second spatial feature are fused to obtain a second distorted feature of the intermediate frame image;

[0178] The output of the spatial layer is mapped through the temporal layer, and the output of the temporal layer is fused with the second warped feature to obtain the output of the processing block.

[0179] Further, the generating module 1004 is also used for:

[0180] Obtaining the occlusion mask corresponding to the frame number between the start frame image and the end frame image;

[0181] determining a third Hadamard product between the occlusion mask and the warped first spatial feature;

[0182] determining a complementary mask of the occlusion mask, and determining a fourth Hadamard product between the complementary mask and the distorted second spatial feature;

[0183] The second distortion feature of the intermediate frame image is obtained according to the sum of the third Hadamard product and the fourth Hadamard product.

[0184] Further, the generating module 1004 is also used for:

[0185] Mapping the second warped feature through a smoothing layer;

[0186] The output of the temporal layer is fused with the mapped second warp features to obtain the output of the processing block.

[0187] Further, the generating module 1004 is also used for:

[0188] Obtaining constraint text for constraining video content of a target video, and extracting text features of the constraint text;

[0189] Perform self-attention processing on the output of the spatial layer through the temporal layer to obtain the first attention feature;

[0190] Perform cross-attention processing based on the text feature and the first attention feature to obtain the second attention feature;

[0191] The second attention feature is mapped to obtain the output of the temporal layer.

[0192] Further, the generating module 1004 is also used for:

[0193] Acquire a sample video, wherein the sample video includes a first sample image located at a start frame, a second sample image located at an end frame, and a third sample image located between the first sample image and the second sample image;

[0194] Extracting a first sample feature of the first sample image, a second sample feature of the second sample image, and a third sample feature of the third sample image;

[0195] Adding sample noise to the first sample feature, the second sample feature, and the third sample feature respectively, to obtain a first noise feature, a second noise feature, and a third noise feature respectively;

[0196] extracting a third optical flow between the first sample image and the third sample image, and a fourth optical flow between the second sample image and the third sample image;

[0197] Distorting the first sample feature based on the third optical flow, distorting the second sample feature based on the fourth optical flow, and fusing the distorted first sample feature and the distorted second sample feature to obtain a third distorted feature of the third sample image;

[0198] The first noise feature is spliced ​​with the first sample feature to obtain a first spliced ​​feature, the second noise feature is spliced ​​with the second sample feature to obtain a second spliced ​​feature, and the third noise feature is spliced ​​with the third distortion feature to obtain a third spliced ​​feature;

[0199] Inputting the first splicing feature, the second splicing feature and the third splicing feature into a diffusion model for denoising;

[0200] The predicted noise output by the diffusion model during the denoising process is obtained, the model loss is determined according to the difference between the predicted noise and the sample noise, and the diffusion model is trained based on the model loss.

[0201] Further, the generating module 1004 is also used for:

[0202] inputting the first stitching feature, the second stitching feature, and the third stitching feature into a diffusion model;

[0203] For each processing block, the input of the processing block is mapped through the spatial layer, the third spatial feature corresponding to the first sample image and the fourth spatial feature corresponding to the second sample image are extracted from the output of the spatial layer, the third spatial feature is distorted based on the third optical flow, the fourth spatial feature is distorted based on the fourth optical flow, the distorted third spatial feature and the distorted fourth spatial feature are fused to obtain the fourth distorted feature of the third sample image, the output of the spatial layer is mapped through the temporal layer, the output of the temporal layer is fused with the fourth distorted feature to obtain the output of the processing block.

[0204] Further, the generating module 1004 is also used for:

[0205] Freeze the parameters of the temporal layers in each processing block;

[0206] The parameters of the spatial layers in each processing block are adjusted based on the model loss to train the diffusion model.

[0207] In summary, the video generating device 1000 in the embodiment of the present disclosure obtains the starting frame image and the ending frame image of the target video to be generated, extracts the first potential feature of the starting frame image and the second potential feature of the ending frame image, extracts the first optical flow between the starting frame image and the intermediate frame image to be generated in the target video, and extracts the second optical flow between the ending frame image and the intermediate frame image. Since the first optical flow and the second optical flow can be used to simulate the motion trajectory and the appearance of the object, the first potential feature is distorted based on the first optical flow, and the second potential feature is distorted based on the second optical flow. The distorted first potential feature and the distorted second potential feature are fused, and the first distorted feature of the intermediate frame image can be approximately estimated as an effective guide for motion inference in the generation process, reflecting the correlation between the starting frame image and the ending frame image. Subsequently, when the first potential feature, the first distorted feature, and the second potential feature are spliced ​​with random noise respectively and input into the diffusion model for denoising, the first distorted feature can be used as an explicit motion clue, so that the model can be aware of the correlation between the starting frame image and the ending frame image without inference, thereby improving the motion stability and content consistency of the generated target video, and improving the quality of video generation.

[0208] The electronic device for executing the above-mentioned video generation method provided in the embodiment of the present disclosure may be a terminal. Fig.11 , Fig.11 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1110, a first memory 1120, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WiFi) module 1170, a first processor 1180, and a first power supply 1190. Those skilled in the art can understand that Fig.11 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0209] The camera assembly 1110 can be used to capture images or videos. Optionally, the camera assembly 1110 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0210] The first memory 1120 may be used to store software programs and modules. The first processor 1180 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1120 .

[0211] The input unit 1130 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1130 may include a touch panel 1131 and other input devices 1132 .

[0212] The display unit 1140 may be used to display input information or provided information and various menus of the terminal. The display unit 1140 may include a display panel 1141 .

[0213] The audio circuit 1160 , the speaker 1161 , and the microphone 1162 may provide an audio interface.

[0214] The first power source 1190 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0215] The number of sensors 1150 may be one or more, and the one or more sensors 1150 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0216] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1180 can control the display unit 1140 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting motion data of games or users.

[0217] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1180 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0218] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1140. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1180 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1140, the first processor 1180 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1140. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0219] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1180 can control the display brightness of the display unit 1140 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1140 is increased; when the ambient light intensity is low, the display brightness of the display unit 1140 is decreased. In another embodiment, the first processor 1180 can also dynamically adjust the shooting parameters of the camera assembly 1110 according to the ambient light intensity collected by the optical sensor.

[0220] In this embodiment, the first processor 1180 included in the terminal can execute the video generating method of the previous embodiment.

[0221] The electronic device for executing the above-mentioned video generation method provided in the embodiment of the present disclosure may also be a server. Fig.12 , Fig.12 This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1210 and a second memory 1230, and one or more storage media 1240 (e.g., one or more mass storage devices) storing application programs 1243 or data 1242. Among them, the second memory 1230 and the storage medium 1240 may be short-term storage or persistent storage. The program stored in the storage medium 1240 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1210 may be configured to communicate with the storage medium 1240 to execute a series of instruction operations in the storage medium 1240 on the server.

[0222] The server may also include one or more second power supplies 1220, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1260, and / or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0223] The second processor 1210 in the server may be configured to execute the video generating method.

[0224] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the video generation method of each of the aforementioned embodiments.

[0225] The present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device implements the above-mentioned video generation method.

[0226] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0227] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0228] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0229] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0233] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0234] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A video generation method, characterized in that: include: Acquire a starting frame image and an ending frame image of a target video to be generated, and extract a first potential feature of the starting frame image and a second potential feature of the ending frame image; Extracting a first optical flow between the start frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the end frame image and the intermediate frame image; distorting the first latent feature based on the first optical flow, distorting the second latent feature based on the second optical flow, and fusing the distorted first latent feature and the distorted second latent feature to obtain a first distorted feature of the intermediate frame image; The first latent feature, the first distorted feature, and the second latent feature are respectively spliced ​​with random noise and input into a diffusion model for denoising to obtain the target video.

2. The video generation method according to claim 1, characterized in that: The extracting of a first optical flow between the start frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the end frame image and the intermediate frame image, comprises: Obtaining the frame sequence number of the intermediate frame image to be generated in the target video; The starting frame image, the ending frame image and the frame sequence number are input into an optical flow prediction model for prediction to obtain a first optical flow between the starting frame image and the intermediate frame image, and a second optical flow between the ending frame image and the intermediate frame image.

3. The video generation method according to claim 2, characterized in that: The optical flow prediction model further outputs an occlusion mask corresponding to the frame number between the start frame image and the end frame image, and the first latent feature after distortion and the second latent feature after distortion are fused to obtain the first distortion feature of the intermediate frame image, including: determining a first Hadamard product between the occlusion mask and the warped first latent feature; determining a complementary mask of the occlusion mask, and determining a second Hadamard product between the complementary mask and the distorted second latent feature; A first distortion feature of the intermediate frame image is obtained according to the sum of the first Hadamard product and the second Hadamard product.

4. The video generation method according to claim 1, characterized in that: The diffusion model includes a plurality of sequentially cascaded processing blocks for denoising, each of the processing blocks includes an interconnected spatial layer and a temporal layer, and the first latent feature, the first distorted feature, and the second latent feature are spliced ​​with random noise respectively and input into the diffusion model for denoising to obtain the target video, including: The first latent feature, the first distorted feature, and the second latent feature are respectively concatenated with random noise and input into a diffusion model; For each of the processing blocks, sequentially mapping the input of the processing block through the spatial layer and the temporal layer to obtain the output of the processing block; The target video is obtained based on the output of the last processing block.

5. The video generation method according to claim 4, characterized in that: Mapping the input of the processing block in sequence through the spatial layer and the temporal layer to obtain the output of the processing block includes: Mapping the input of the processing block through the spatial layer, and extracting the first spatial feature corresponding to the start frame image and the second spatial feature corresponding to the end frame image from the output of the spatial layer; distorting the first spatial feature based on the first optical flow, distorting the second spatial feature based on the second optical flow, and fusing the distorted first spatial feature and the distorted second spatial feature to obtain a second distorted feature of the intermediate frame image; The output of the spatial layer is mapped through the temporal layer, and the output of the temporal layer is fused with the second warping feature to obtain the output of the processing block.

6. The video generation method according to claim 5, characterized in that: The step of fusing the first distorted spatial feature and the second distorted spatial feature to obtain a second distorted feature of the intermediate frame image includes: Obtaining an occlusion mask corresponding to the frame sequence number between the start frame image and the end frame image; determining a third Hadamard product between the occlusion mask and the warped first spatial feature; Determining a complementary mask of the occlusion mask, and determining a fourth Hadamard product between the complementary mask and the distorted second spatial feature; A second distortion feature of the intermediate frame image is obtained according to the sum of the third Hadamard product and the fourth Hadamard product.

7. The video generation method according to claim 5, characterized in that: The processing block is further provided with a smoothing layer, the smoothing layer is obtained by copying the parameters of the time layer, and the output of the time layer is fused with the second distortion feature to obtain the output of the processing block, including: mapping the second distortion feature through the smoothing layer; The output of the time layer is fused with the mapped second warped feature to obtain the output of the processing block.

8. The video generation method according to claim 5, characterized in that: Mapping the output of the spatial layer through the time layer includes: Obtaining a constraint text for constraining the video content of the target video, and extracting text features of the constraint text; Performing self-attention processing on the output of the spatial layer through the temporal layer to obtain a first attention feature; Performing cross attention processing on the text feature and the first attention feature to obtain a second attention feature; The second attention feature is mapped to obtain the output of the time layer.

9. The video generation method according to claim 1, characterized in that: The diffusion model is trained by the following steps: Acquire a sample video, wherein the sample video includes a first sample image located at a start frame, a second sample image located at an end frame, and a third sample image located between the first sample image and the second sample image; Extracting a first sample feature of the first sample image, a second sample feature of the second sample image, and a third sample feature of the third sample image; Adding sample noise to the first sample feature, the second sample feature and the third sample feature respectively, to obtain a first noise feature, a second noise feature and a third noise feature respectively; extracting a third optical flow between the first sample image and the third sample image, and a fourth optical flow between the second sample image and the third sample image; Distorting the first sample feature based on the third optical flow, distorting the second sample feature based on the fourth optical flow, and fusing the distorted first sample feature and the distorted second sample feature to obtain a third distorted feature of the third sample image; splicing the first noise feature with the first sample feature to obtain a first splicing feature, splicing the second noise feature with the second sample feature to obtain a second splicing feature, and splicing the third noise feature with the third distortion feature to obtain a third splicing feature; Inputting the first splicing feature, the second splicing feature and the third splicing feature into the diffusion model for denoising; The predicted noise output by the diffusion model during the denoising process is obtained, a model loss is determined according to a difference between the predicted noise and the sample noise, and the diffusion model is trained based on the model loss.

10. The video generation method according to claim 9, characterized in that: The diffusion model includes a plurality of sequentially cascaded processing blocks for denoising, each of the processing blocks includes a space layer and a time layer connected to each other, and the first splicing feature, the second splicing feature and the third splicing feature are input into the diffusion model for denoising, including: inputting the first stitching feature, the second stitching feature, and the third stitching feature into the diffusion model; For each of the processing blocks, the input of the processing block is mapped through the spatial layer, the third spatial feature corresponding to the first sample image and the fourth spatial feature corresponding to the second sample image are extracted from the output of the spatial layer, the third spatial feature is distorted based on the third optical flow, the fourth spatial feature is distorted based on the fourth optical flow, the distorted third spatial feature and the distorted fourth spatial feature are fused to obtain the fourth distorted feature of the third sample image, the output of the spatial layer is mapped through the temporal layer, the output of the temporal layer is fused with the fourth distorted feature to obtain the output of the processing block.

11. The video generation method according to claim 10, characterized in that: The step of training the diffusion model based on the model loss includes: Freezing the parameters of the time layer in each of the processing blocks; The parameters of the spatial layers in each of the processing blocks are adjusted based on the model loss to train the diffusion model.

12. A video generating device, characterized in that: include: A feature extraction module, used to obtain a starting frame image and an ending frame image of a target video to be generated, and extract a first potential feature of the starting frame image and a second potential feature of the ending frame image; An optical flow extraction module, used to extract a first optical flow between the start frame image and an intermediate frame image to be generated in the target video, and a second optical flow between the end frame image and the intermediate frame image; a feature change module, configured to distort the first potential feature based on the first optical flow, distort the second potential feature based on the second optical flow, and fuse the distorted first potential feature and the distorted second potential feature to obtain a first distorted feature of the intermediate frame image; A generation module is used to splice the first potential feature, the first distortion feature, and the second potential feature with random noise respectively, and input them into a diffusion model for denoising to obtain the target video.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the video generating method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video generation method according to any one of claims 1 to 11 is implemented.

Images