Video generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video generation method. The video generation method comprises the following steps: acquiring a video generation condition; converting the video generation condition into a condition feature; inputting the condition features into a trained video generation model to generate a video corresponding to the condition features; wherein the trained video generation model is obtained through optical flow information supervision training. The invention further provides a video generation device, electronic equipment, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generation method, device, electronic device, storage medium, and program product. Background Art

[0002] With the rapid development of diffusion models, image generation technology has made significant progress, which in turn has driven research in video generation technology. However, compared to image generation, video generation requires additional attention to temporal consistency and the rationality of object motion. Therefore, many problems still exist: First, current generation models lack the ability to explicitly model motion, resulting in abrupt content transitions or severe object deformation between video frames, and a lack of inter-frame consistency. Second, current generation models lack explicit motion supervision, resulting in motion blur or misdirection in the generated videos. In particular, in complex description or long-term time series generation tasks, the generated videos often lack the motion trajectory of real objects. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a video generation method, apparatus, electronic device, storage medium, and program product, which can solve or partially solve the above-mentioned problems to a certain extent.

[0004] In some embodiments of the present disclosure, the video generation method described in the embodiments of the present disclosure may include: obtaining video generation conditions; converting the video generation conditions into conditional features; and inputting the conditional features into a trained video generation model to generate a video corresponding to the conditional features; wherein the trained video generation model is obtained through supervised training of optical flow information.

[0005] In some embodiments of the present disclosure, the trained video generation model is obtained by training the video generation model to be trained based on the training loss, and the training loss is obtained based on the generation loss and optical flow loss of the video sample and the predicted video, and the predicted video is generated by the video generation model to be trained based on the conditional features of the video sample.

[0006] In some embodiments of the present disclosure, the generation loss includes the distance loss between the video sample and the predicted video; and / or, the optical flow loss is determined based on the first optical flow information corresponding to the video sample and the second optical flow information corresponding to the predicted video; and / or, the training loss is obtained by weighted summing the generation loss and the optical flow loss.

[0007] In some embodiments of the present disclosure, the first optical flow information includes the inter-frame similarity of the video samples, and / or the second optical flow information includes the inter-frame similarity of the predicted video; and / or the optical flow loss includes the distance loss and / or classification loss between the first optical flow information and the second optical flow information.

[0008] In some embodiments of the present disclosure, optical flow information supervised training includes: obtaining a video sample in a sample data set and conditional features corresponding to the video sample; inputting the conditional features corresponding to the video sample into a video generation model to be trained, and obtaining a predicted video generated by the video generation model to be trained; determining a generation loss based on the video sample and the predicted video; determining an optical flow loss based on the video sample and the predicted video; determining a training loss based on the generation loss and the optical flow loss; training the video generation model to be trained based on the training loss; and returning to the step of obtaining a video sample in a sample data set and the conditional features corresponding to the video sample until a training completion condition is met to obtain the trained video generation model.

[0009] In some embodiments of the present disclosure, determining the generation loss based on the video sample and the predicted video includes: determining a distance loss between the video sample and the predicted video, and using the distance loss between the video sample and the predicted video as the generation loss.

[0010] In some embodiments of the present disclosure, determining the optical flow loss based on the video sample and the predicted video includes: determining first optical flow information corresponding to the video sample; determining second optical flow information corresponding to the predicted video; and determining the optical flow loss based on the first optical flow information and the second optical flow information.

[0011] In some embodiments of the present disclosure, determining the first optical flow information corresponding to the video sample includes: inputting the video sample into an optical flow model, and using the inter-frame similarity of the video sample output by the optical flow model as the first optical flow information; and determining the second optical flow information corresponding to the predicted video includes: inputting the predicted video into the optical flow model, and using the inter-frame similarity of the predicted video output by the optical flow model as the second optical flow information.

[0012] In some embodiments of the present disclosure, determining the first optical flow information corresponding to the video sample includes: determining the inter-frame similarity of the video sample based on an optical flow determination method for energy minimization, and using the inter-frame similarity of the video sample as the first optical flow information; and determining the second optical flow information corresponding to the predicted video includes: determining the inter-frame similarity of the predicted video based on the optical flow determination method for energy minimization, and using the inter-frame similarity of the predicted video as the second optical flow information.

[0013] In some embodiments of the present disclosure, determining the optical flow loss based on the first optical flow information and the second optical flow information includes: determining the distance loss between the first optical flow information and the second optical flow information, and taking the distance loss between the first optical flow information and the second optical flow information as the optical flow loss.

[0014] In some embodiments of the present disclosure, determining the optical flow loss based on the first optical flow information and the second optical flow information includes: determining a classification loss between the first optical flow information and the second optical flow information, and using the classification loss as the optical flow loss.

[0015] In some embodiments of the present disclosure, determining the training loss based on the generation loss and the optical flow loss includes: performing a weighted summation on the generation loss and the optical flow loss to obtain the training loss.

[0016] Corresponding to the above-mentioned video generation method, an embodiment of the present disclosure further discloses a video generation device, including:

[0017] An acquisition module, used to obtain video generation conditions;

[0018] A conversion module, configured to convert the video generation condition into a conditional feature;

[0019] A generation module is used to input the conditional features into a trained video generation model to generate a video corresponding to the conditional features; wherein the trained video generation model is obtained through optical flow information supervision training.

[0020] In addition, an embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned video generation method when executing the computer program.

[0021] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned video generation method.

[0022] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the above-mentioned video generation method.

[0023] As can be seen from this, in the aforementioned video generation method, apparatus, electronic device, storage medium, and program product, a video generation model can first be trained using optical flow information for supervised training to obtain a trained video generation model. The trained video generation model can then generate a video based on the input conditional features of the video generation conditions. By introducing optical flow information between video frames as an auxiliary supervisory signal during the training phase, the temporal modeling capabilities of the video generation model can be enhanced, effectively addressing deficiencies in inter-frame consistency and natural or reasonable motion generated by existing video generation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram showing an exemplary system provided by an embodiment of the present disclosure is shown.

[0026] Figure 2 The implementation process of the video generation method described in some embodiments of the present disclosure is shown.

[0027] Figure 3 The implementation process of the optical flow information supervision training method described in some embodiments of the present disclosure is shown.

[0028] Figure 4 The internal structure of the video generating device described in some embodiments of the present disclosure is shown.

[0029] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0031] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0032] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0033] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0034] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0036] As previously mentioned, because video generation technology requires additional attention to temporal consistency and the naturalness or rationality of object motion, problems still exist, such as a lack of inter-frame consistency and unnatural, unreasonable, or blurred motion. In view of this, the embodiments of the present disclosure provide a video generation method, apparatus, electronic device, storage medium, and program product that can partially or partially address these issues.

[0037] For the sake of clarity in description, before describing the specific technical solutions of the embodiments of the present disclosure, the technical terms involved in the embodiments of the present disclosure are first explained.

[0038] Video generation has been a research hotspot in generative AI in recent years. It refers to the process of automatically synthesizing new video content using AI techniques, encompassing the ability to generate coherent video sequences from inputs such as text, images, action descriptions, and even videos. This capability involves multiple subproblems, including multimodal modeling, temporal modeling, and high-quality visual generation, and presents a high level of technical difficulty. Existing video generation approaches include methods based on generative adversarial networks, which use adversarial training to enable the generator to learn realistic video distributions, and methods based on diffusion models, which generate video frames through a stepwise inverse denoising process.

[0039] Optical flow is a core concept in computer vision, used to describe the motion of objects in an image between consecutive frames. Essentially, it captures motion patterns in dynamic scenes through pixel displacement. Simply put, optical flow represents the direction and speed of an object's motion in a video and can be used to estimate the motion vector of each pixel in the image over time. Classic optical flow algorithms include those based on energy minimization, such as the Lucas-Kanade method and the Horn-Schunck method, and those based on deep learning neural networks, such as FlowNet, PWC-Net, RAFT, and GM-Flow.

[0040] Text-to-Video can be an artificial intelligence technology that generates dynamic video sequences based on text, aiming to convert the text descriptions input by users into dynamic and coherent video content.

[0041] Image-to-Video is an artificial intelligence technology that generates dynamic video sequences based on static images. It aims to automatically generate coherent video content containing elements such as camera movement, special effects, and music by inputting single or multiple images and combining them with text descriptions or default parameters.

[0042] Distance loss can be a type of loss function in machine learning that optimizes model parameters by quantifying the spatial difference between predicted values and true values. The core goal of distance loss is to reduce the geometric distance between the predicted results and the true distribution. The distance loss function can be directly applied to numerical data and is suitable for scenarios such as regression tasks and similarity calculations. Distance loss can generally include: Manhattan distance loss, Euclidean distance loss, cosine distance loss, and Hamming distance loss, etc. Among them, the above-mentioned Manhattan distance can also be called L1 distance or Mean Absolute Error (MAE) distance; the above-mentioned Euclidean distance can also be called L2 distance or Mean Squared Error (MSE) distance.

[0043] Classification loss is a quantitative metric used in supervised learning to measure the difference between the model's predicted probability distribution and the true label distribution. It guides model parameter optimization through backpropagation, ultimately improving classification accuracy. Essentially, classification loss uses a mathematical function to convert the degree of prediction error into an optimizable numerical signal. Typical classification losses include cross-entropy loss, Hamming loss, and Hinge loss.

[0044] The U-Net model is a convolutional neural network designed specifically for image segmentation. It is widely used in fields such as image analysis due to its unique U-shaped structure and high efficiency.

[0045] DiT (Diffusion Transformer) can be a generative artificial intelligence model that combines the diffusion model and the transformer architecture, and is designed specifically for image and video generation tasks.

[0046] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0047] like Figure 1 As shown, system 100 may include terminal device 102, terminal device 104, and server 106. A medium (e.g., a network) providing a communication link may be included between terminal device 102, terminal device 104, and server 106. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0048] Exemplarily, an application program (APP) or software that can realize video generation can be installed on the terminal device 102 and the terminal device 104. The terminal device 102 and the terminal device 104 here can be hardware or software. When the terminal device 102 and the terminal device 104 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers (Laptops) and desktop computers (PCs), etc. When the terminal device 102 and the terminal device 104 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0049] The server 106 can be a server that provides video generation services, such as a background server that provides support for the application or software displayed by the terminal device 102 or the terminal device 104. The server 106 here can also be hardware or software. When the server 106 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 106 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0050] It should be understood that Figure 1 The numbers of terminal devices, users and servers in the embodiment are merely illustrative. Any number of terminal devices, users and servers may be used as required.

[0051] As an example scenario, server 106 can provide a video generation service. User 112 can use a video generation application on terminal device 102 to submit conditional features for video generation to server 106. The conditional features can include various types of data, such as text, images, or videos, for video generation. Server 106 generates a video corresponding to the conditional features submitted by user 112 via terminal device 102. Server 106 can then feed the generated video back to user 112 via terminal device 102.

[0052] In another exemplary scenario, the video generation application downloaded and installed by the terminal device 104 from the server 106 can support offline video generation. In this case, the user 114 can directly use the video generation application of the terminal device 104 to generate a video based on the conditional features used to generate the video. The conditional features used to generate the video can also include various types of data such as text, images, or videos used to generate the video. In other words, the specific process of the video generation can be completed independently by the terminal device 104 offline without the real-time participation of the server 106.

[0053] Based on the above system 100, in order to solve various problems existing in video generation in related technologies, the embodiment of the present disclosure provides a video generation method. The video generation method provided by the embodiment of the present disclosure is described below with reference to specific embodiments and drawings.

[0054] Figure 2 The implementation process of the video generation method described in the embodiment of the present disclosure is shown. Figure 2 As shown, the video generation method described in the embodiment of the present disclosure may specifically include the following steps.

[0055] In step 210, video generation conditions are obtained.

[0056] In step 220 , the video generation conditions are converted into conditional features.

[0057] In step 230, the conditional features are input into a trained video generation model to generate a video corresponding to the conditional features; wherein the trained video generation model is obtained through supervised training of optical flow information.

[0058] As can be seen from the above video generation method, the video generation model can first be trained based on optical flow information to obtain a trained video generation model. The trained video generation model can then generate a video based on the conditional features of the input video generation conditions. By introducing optical flow information between video frames as an auxiliary supervisory signal during the training phase, the temporal modeling capabilities of the video generation model can be improved, effectively addressing the shortcomings of existing video generation models in generating videos with inter-frame consistency and natural or reasonable motion.

[0059] The following describes in detail the various steps of the video generation method according to the embodiment of the present disclosure with reference to specific examples and accompanying drawings.

[0060] For the above step 210, illustratively, Figure 1The user 112 inputs the video generation conditions through the video generation application of the terminal device 102. In some embodiments, the video generation conditions may include: one or a combination of multiple types of data, such as text, images, and videos. Assuming that the video generation conditions include text, the trained video generation model can be applied to text-to-video applications, that is, generating videos based on text; assuming that the video generation conditions include images, the trained video generation model can be applied to image-to-video applications, that is, generating videos based on images; assuming that the video generation conditions include videos, the trained video generation model can be applied to applications such as video completion or video frame insertion.

[0061] In some embodiments, assuming that the video generation method is executed by the server 106, the terminal device 102 can upload the video generation conditions to the server 106, so that the server 106 can obtain the video generation conditions. Assuming that the video generation method is executed by the terminal device 102, the terminal device 102 can obtain the video generation conditions when receiving the video generation conditions input by the user 112.

[0062] With respect to the above step 220, after obtaining the video generation condition, the terminal device 102 or the server 106 can convert the video generation condition into a conditional feature to be used as an input of a trained video generation model to generate a corresponding video.

[0063] In some embodiments, semantic features of the video generation conditions can be extracted as the conditional features. This can preserve the semantic information in the video generation conditions to guide the model generation process, while also compressing the amount of input data and improving computational efficiency. It is understood that in addition to semantic features, other characteristic information of the video generation conditions can also be extracted as the conditional features.

[0064] In some scenarios, step 220 may also directly use the video generation condition as the conditional feature, thereby omitting specific conversion steps and improving processing efficiency.

[0065] With respect to the above step 230 , the conditional features used to generate the video may be input into a trained video generation model.

[0066] As mentioned above, the above-mentioned trained video generation model can be obtained through supervised training with optical flow information. By introducing optical flow information between video frames as an auxiliary supervisory signal during the training phase, the temporal modeling capability of the video generation model can be improved, thereby effectively solving the deficiencies in inter-frame consistency and naturalness or rationality of motion in the videos generated by the existing video generation model.

[0067] In some embodiments, the trained video generation model can be obtained by training the video generation model to be trained based on a training loss. The training loss can be obtained based on the generation loss and optical flow loss of the video sample and the predicted video. The predicted video can be generated by the video generation model to be trained based on the conditional features of the video sample. In this way, by introducing the optical flow loss into the training loss when training the model, smoother gradient changes can be obtained when backpropagating based on the training loss, thereby improving the generation effect of the model.

[0068] Figure 3 The implementation process of the optical flow information supervision training method described in some embodiments of the present disclosure is shown. Figure 3 As shown, the implementation process of the optical flow information supervision training method described in the embodiment of the present disclosure may include the following steps:

[0069] In step 310 , a video sample in a sample data set and conditional features corresponding to the video sample are obtained.

[0070] In the embodiments of the present disclosure, the sample data set may generally include a plurality of video samples and conditional features corresponding to the video samples respectively.

[0071] In step 320 , the conditional features corresponding to the video sample are input into the video generation model to be trained, and a predicted video generated by the video generation model to be trained is obtained.

[0072] In the embodiments of the present disclosure, the video generation model to be trained can be implemented based on a U-Net model or a DiT generation model. In addition, the video generation model to be trained can be a denoising model for the original pixel space or a denoising model for the compressed latent space. The embodiments of the present disclosure do not limit the specific type and structure of the video generation model to be trained.

[0073] At step 330 , a loss is generated based on the video samples and the predicted video determination.

[0074] In some embodiments, the generation loss may include the distance loss between the video sample and the predicted video, so that the distance loss can more smoothly guide the model to learn the real data distribution, reduce the homogeneity problem of the generated results, and improve the stability and convergence efficiency of the model.

[0075] In an embodiment of the present disclosure, the above-mentioned step 330 may include: determining the distance loss between the video sample and the predicted video, and using the distance loss between the video sample and the predicted video as the generation loss. Specifically, in some specific examples, the above-mentioned distance loss may include: one of the distance losses such as Manhattan distance loss, Euclidean distance loss, cosine distance loss and Hamming distance loss. It should be noted that during the training process, by quantifying the distribution difference between the predicted video and the video sample, the distance loss can more smoothly guide the model to learn the real data distribution, reduce the homogeneity problem of the generated results, and improve the stability and convergence efficiency of the model. In addition, in video generation, the use of distance loss can simultaneously constrain spatial details (such as object texture) and temporal coherence (such as motion trajectory), ensure the rationality of frame-by-frame changes, and optimize the spatiotemporal consistency of the model.

[0076] At step 340 , an optical flow loss is determined based on the video samples and the predicted video.

[0077] In some embodiments, the optical flow loss can be determined based on the first optical flow information corresponding to the video sample and the second optical flow information corresponding to the predicted video. The optical flow loss is determined by comparing the difference between the first optical flow information and the second optical flow information. When back propagation is performed based on the optical flow loss, a smoother gradient change is obtained, thereby improving the generation effect of the model.

[0078] In an embodiment of the present disclosure, step 340 may include: first, determining first optical flow information corresponding to the video sample and second optical flow information corresponding to the predicted video, and then determining an optical flow loss based on the determined first optical flow information and second optical flow information.

[0079] Specifically, in some embodiments of the present disclosure, the first optical flow information corresponding to the video sample and the second optical flow information corresponding to the predicted video can be determined respectively by a classical optical flow algorithm or an optical flow algorithm based on a deep learning neural network.

[0080] In some embodiments, the first optical flow information may include the inter-frame similarity of the video sample, and the second optical flow information may include the inter-frame similarity of the predicted video. Compared to directly constraining the pixel coordinate difference between the predicted optical flow and the true optical flow, constraining the inter-frame similarity of pixels within a local area can achieve smoother gradient changes and improve training efficiency.

[0081] Specifically, in some embodiments, a video sample can be input into an optical flow model, and the inter-frame similarity of the video sample output by the optical flow model is used as the first optical flow information; and a predicted video can be input into the optical flow model, and the inter-frame similarity of the predicted video output by the optical flow model is used as the second optical flow information. In some embodiments of the present disclosure, the optical flow model can include one of deep learning neural networks such as FlowNet, PWC-Net, RAFT, and GM-Flow.

[0082] Specifically, in other embodiments, the inter-frame similarity of video samples can be determined based on an energy-minimized optical flow determination method, and the inter-frame similarity of the video samples can be used as the first optical flow information; and the inter-frame similarity of the predicted video can be determined based on an energy-minimized optical flow determination method, and the inter-frame similarity of the predicted video can be used as the second optical flow information. In some embodiments of the present disclosure, the above-mentioned optical flow determination method can include: a Lucas-Kanade optical flow algorithm or a Horn-Schunck optical flow algorithm.

[0083] In some embodiments of the present disclosure, the inter-frame similarity can represent the confidence level of feature-level matching between adjacent video frames and can be generated by calculating the correlation between the feature maps of two frames. The inter-frame similarity can be used as a key output of the intermediate layer of the optical flow model or as an intermediate key parameter output of the optical flow determination method based on energy minimization to guide the generation of the optical flow field, especially to improve prediction accuracy when processing large displacement motion. The inter-frame similarity can be a similarity matrix (or cost volume) obtained by calculating the similarity of the feature maps of two frames; wherein each element of the similarity matrix represents the feature similarity of corresponding positions in the two frames. More specifically, in the embodiments of the present disclosure, the inter-frame similarity can be global inter-frame similarity or local inter-frame similarity. Global inter-frame similarity is mainly used to measure the overall similarity between video frames; while local inter-frame similarity is mainly used to measure the similarity of specific regions or local features (such as objects and edges) within a frame, focusing on changes in local details. The use of inter-frame similarity as the optical flow information for the sample videos and the predicted videos takes into account the discrete distribution of motion vectors, which can easily lead to excessive backpropagation gradients during training. Specifically, in the embodiments of the present disclosure, the use of inter-frame similarity as the optical flow information for the sample videos and the predicted videos allows for a continuous and smooth loss function during training, resulting in smoother gradient changes, reducing learning difficulty and effectively improving training efficiency.

[0084] In some embodiments, the optical flow loss may include the distance loss between the first optical flow information and the second optical flow information. During the training process, by quantifying the distribution difference between the first optical flow information and the second optical flow information, the distance loss can more smoothly guide the model to learn the real data distribution, thereby improving the stability and convergence efficiency of the model. In addition, the spatiotemporal consistency of the model can also be further optimized. In other embodiments, the optical flow loss may include the classification loss between the first optical flow information and the second optical flow information. Introducing a classification loss function in a video generation model can optimize the semantic consistency and category discrimination of the generated content, enhance the semantic accuracy of the generated content, and improve the stability of model training.

[0085] In some embodiments of the present disclosure, the above-mentioned determination of the optical flow loss based on the first optical flow information and the second optical flow information may include: determining the distance loss between the first optical flow information and the second optical flow information, and using the distance loss between the first optical flow information and the second optical flow information as the optical flow loss. Specifically, in some specific examples, the above-mentioned distance loss may include: one of the distance losses such as Manhattan distance, Euclidean distance, cosine distance loss and Hamming distance loss. Similarly, during the training process, by quantifying the distribution difference between the first optical flow information and the second optical flow information, the distance loss can more smoothly guide the model to learn the real data distribution, thereby improving the stability and convergence efficiency of the model. In addition, the spatiotemporal consistency of the model can also be further optimized.

[0086] In some other embodiments of the present disclosure, the above-mentioned determination of the optical flow loss based on the first optical flow information and the second optical flow information may include: determining the classification loss between the first optical flow information and the second optical flow information, and taking the classification loss between the first optical flow information and the second optical flow information as the optical flow loss. Specifically, in some specific examples, the above-mentioned classification loss may include one of the classification losses such as cross entropy loss, Hamming loss and Hinge loss. It can be understood that the introduction of the classification loss function in the video generation model can optimize the semantic consistency and category discrimination of the generated content, enhance the semantic accuracy of the generated content, and also improve the stability of model training.

[0087] At step 350 , a training loss is determined based on the generation loss and the optical flow loss.

[0088] In some embodiments, the training loss can be obtained by taking a weighted sum of the generation loss and the optical flow loss. This training loss includes both the generation loss and the optical flow loss of the model. This allows the optical flow information between video frames to be introduced as an auxiliary supervisory signal into the model training process, thereby improving the temporal modeling capabilities of the video generation model and effectively addressing deficiencies in inter-frame consistency and natural or reasonable motion generated by existing video generation models.

[0089] In some embodiments of the present disclosure, determining the training loss based on the generation loss and the optical flow loss may include performing a weighted summation of the generation loss and the optical flow loss to obtain the training loss. Specifically, in embodiments of the present disclosure, the training loss may be determined by the following expression:

[0090] Loss = α1·L G +α2·L O

[0091] Among them, Loss represents the above training loss; L G represents the above-mentioned generation loss; L O Represents the above optical flow loss; α1 represents the weighting coefficient of the corresponding generation loss; and α2 represents the weighting coefficient of the corresponding optical flow loss. Among them, the above α1 and α2 can be determined according to the actual situation, and usually α1+α2=1 can be satisfied. The embodiments of the present disclosure do not limit the specific values of the above weighting coefficients. It can be seen that the above training loss includes both the generation loss part of the model and the optical flow loss part, so that the optical flow information between video frames can be introduced into the training process of the model as an auxiliary supervision signal to improve the temporal modeling ability of the video generation model, and effectively solve the deficiencies of the video generated by the existing video generation model in terms of inter-frame consistency and naturalness or rationality of motion.

[0092] In step 360, the video generation model to be trained is trained based on the training loss, and then the process returns to step 310 to obtain a new video sample in the sample data set and conditional features corresponding to the video sample until the training completion condition is met, thereby obtaining a trained video generation model.

[0093] In some embodiments of the present disclosure, the training loss can be back-propagated to train the video generation model to be trained. The training process is iteratively repeated until a training completion condition is met, at which point the training is terminated, resulting in a trained video generation model. Specifically, the training completion condition may include: the model has converged or a predetermined number of training cycles has been reached.

[0094] As previously mentioned, by introducing inter-frame similarity between video samples and predicted videos as an auxiliary supervisory signal during the training phase of the video generation model, the temporal modeling capabilities of the video generation model can be improved, effectively addressing the shortcomings of existing video generation models in terms of inter-frame consistency and natural or reasonable motion. Furthermore, using inter-frame similarity as the optical flow information for the sample videos and predicted videos can produce a continuous and smooth loss function during training, resulting in smoother gradient changes, reducing the learning difficulty and effectively improving training efficiency.

[0095] Corresponding to the above-mentioned video generation method, some embodiments of the present disclosure further disclose a video generation device. Figure 4 The internal structure of the video generating device according to the embodiment of the present disclosure is shown. Figure 4 As shown, the video generation device may include the following modules:

[0096] An acquisition module 410 is used to acquire video generation conditions;

[0097] A conversion module 420, configured to convert the video generation condition into a conditional feature;

[0098] A generation module 430 is configured to input the conditional features into a trained video generation model to generate a video corresponding to the conditional features;

[0099] The trained video generation model is obtained through supervised training using optical flow information.

[0100] As can be seen from this, in the above-described video generation method and apparatus, a video generation model can be first supervised and trained based on optical flow information to obtain a trained video generation model. The trained video generation model can then generate a video based on the input conditional features of the video generation conditions. By introducing optical flow information between video frames as an auxiliary supervisory signal during the training phase, the temporal modeling capabilities of the video generation model can be improved, effectively addressing the shortcomings of existing video generation models in generating videos with regard to inter-frame consistency and natural motion.

[0101] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video generation method described in any of the above embodiments is implemented.

[0102] Figure 5 1 is a schematic diagram showing the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other within the device via the bus 2050.

[0103] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0104] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0105] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0106] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0107] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0108] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0109] The electronic device of the above embodiment is used to implement the corresponding video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0110] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video generation method described in any of the above embodiments.

[0111] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0112] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the video generation method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0113] Based on the same inventive concept, corresponding to the video generation method in any of the above embodiments, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, when the computer program instructions are executed on a computer, the computer executes each step in each embodiment of the video generation method. For each step in each embodiment of the video generation method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0114] The computer program product of the above embodiment is used to enable a processor to execute the video generation method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0115] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0116] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0117] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0118] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A video generation method, comprising: Get video generation conditions; Converting the video generation condition into a conditional feature; as well as The conditional features are input into a trained video generation model to generate a video corresponding to the conditional features; wherein the trained video generation model is obtained through supervised training of optical flow information.

2. The method according to claim 1, wherein The trained video generation model is obtained by training the video generation model to be trained based on the training loss; The training loss is obtained based on the generation loss and optical flow loss of the video samples and the predicted video; and The predicted video is generated by the video generation model to be trained based on the conditional features of the video sample.

3. The method according to claim 2, wherein: The generation loss includes a distance loss between the video sample and the predicted video; and / or, The optical flow loss is determined based on first optical flow information corresponding to the video sample and second optical flow information corresponding to the predicted video; and / or, The training loss is obtained by weighted summing the generation loss and the optical flow loss.

4. The method according to claim 3, wherein: The first optical flow information includes inter-frame similarity of the video samples; and / or, The second optical flow information includes inter-frame similarity of the predicted video; and / or, The optical flow loss includes a distance loss and / or a classification loss between the first optical flow information and the second optical flow information.

5. The method according to claim 1, wherein The optical flow information supervision training includes: Obtain a video sample from a sample data set and conditional features corresponding to the video sample; Inputting the conditional features corresponding to the video sample into the video generation model to be trained, and obtaining a predicted video generated by the video generation model to be trained; Determining a generation loss based on the video sample and the predicted video; determining an optical flow loss based on the video sample and the predicted video; Determine a training loss based on the generation loss and the optical flow loss; Training the video generation model to be trained based on the training loss; and Return to the step of obtaining a video sample in the sample data set and the conditional features corresponding to the video sample until the training completion condition is met, thereby obtaining the trained video generation model.

6. The method according to claim 5, wherein: The determining of the generation loss based on the video sample and the predicted video includes: determining a distance loss between the video sample and the predicted video, and using the distance loss between the video sample and the predicted video as the generation loss.

7. The method according to claim 5, wherein: The determining of the optical flow loss based on the video sample and the predicted video includes: Determining first optical flow information corresponding to the video sample; Determining second optical flow information corresponding to the predicted video; and The optical flow loss is determined based on the first optical flow information and the second optical flow information.

8. The method according to claim 7, wherein: Determining the first optical flow information corresponding to the video sample includes: inputting the video sample into an optical flow model, and using the inter-frame similarity of the video sample output by the optical flow model as the first optical flow information; and Determining the second optical flow information corresponding to the predicted video includes: inputting the predicted video into the optical flow model, and using the inter-frame similarity of the predicted video output by the optical flow model as the second optical flow information.

9. The method according to claim 7, wherein: The determining the first optical flow information corresponding to the video sample includes: determining the inter-frame similarity of the video sample based on an optical flow determination method of energy minimization, and using the inter-frame similarity of the video sample as the first optical flow information; and The determining the second optical flow information corresponding to the predicted video includes: determining the inter-frame similarity of the predicted video based on the energy minimization optical flow determination method, and using the inter-frame similarity of the predicted video as the second optical flow information.

10. The method according to claim 7, wherein: The determining the optical flow loss based on the first optical flow information and the second optical flow information includes: determining a distance loss between the first optical flow information and the second optical flow information, and using the distance loss between the first optical flow information and the second optical flow information as the optical flow loss.

11. The method according to claim 7, wherein: The determining the optical flow loss based on the first optical flow information and the second optical flow information includes: determining a classification loss between the first optical flow information and the second optical flow information, and using the classification loss as the optical flow loss.

12. The method according to claim 5, wherein: The determining of the training loss based on the generation loss and the optical flow loss includes: performing weighted summation on the generation loss and the optical flow loss to obtain the training loss.

13. A video generating device, comprising: An acquisition module is used to obtain video generation conditions; A conversion module, configured to convert the video generation condition into a conditional feature; A generation module, configured to input the conditional features into a trained video generation model to generate a video corresponding to the conditional features; The trained video generation model is obtained through supervised training using optical flow information.

14. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video generation method according to any one of claims 1 to 12 when executing the computer program. 15 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the video generation method according to claim 1 . 16 . A computer program product comprising computer program instructions, which, when executed on a computer, enable the computer to execute the video generation method according to claim 1 .

Images