Video generation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of image processing, and relates to a video generation method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining standard video data; extracting a first video frame, a second video frame and a third video frame from the standard video data according to a preset framing rule; performing video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame; generating a second extended video frame according to the second video frame and the first extended video frame; generating a third extended video frame according to the first extended video frame, the second extended video frame and the third video frame; and combining the first extended video frame, the second extended video frame and the third extended video frame to obtain an output video. According to the invention, continuous and content-consistent output video data can be obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, specifically to the field of digital medicine, and in particular to a video generation method, apparatus, computer equipment and storage medium. Background Art

[0002] In the wave of digital transformation in the insurance industry, video marketing, as an intuitive and efficient communication method, has shown great potential, especially in the field of medical insurance. Medical insurance video can significantly improve consumers' awareness and willingness to buy by vividly displaying medical service scenarios, analyzing insurance terms, and popularizing health knowledge. However, the current insurance video diffusion technology faces many challenges, the most notable of which is the lack of control over the diffusion process.

[0003] Existing technologies often rely on simple video editing software or platform automation. These methods can generate a large amount of video content quickly but are difficult to ensure the consistency and accuracy of the video content. Due to the lack of a sophisticated control mechanism, the diffused medical insurance video is likely to deviate from the original video content in terms of subject matter, information transmission or visual effects, and even mislead consumers. For example, exaggerated descriptions of medical procedures, misinterpretations of insurance terms or distortions of visual effects may damage the authenticity and credibility of the video, thereby weakening consumers' trust and willingness to buy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video generation method, apparatus, computer device and storage medium to solve the problem of being unable to generate continuous output video data with consistent content.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a video generation method, which adopts the following technical solution:

[0006] Get standard video data;

[0007] Extracting a first video frame, a second video frame, and a third video frame from the standard video data according to a preset frame division rule;

[0008] Performing video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame;

[0009] generating a second extended video frame according to the second video frame and the first extended video frame;

[0010] Generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame;

[0011] The first extended video frame, the second extended video frame, and the third extended video frame are combined to obtain an output video.

[0012] Furthermore, the step of obtaining standard video data specifically includes:

[0013] Get the video extraction identifier;

[0014] Extracting initial video data from a database according to the video extraction identifier;

[0015] The initial video data is preprocessed to obtain the standard video data.

[0016] Furthermore, the step of extracting the first video frame, the second video frame, and the third video frame from the standard video data according to the preset frame division rule specifically includes:

[0017] Obtaining a preset framing rule, parsing the preset framing rule, and obtaining a first video frame extraction rule, a second video frame extraction rule, and a third video frame extraction rule;

[0018] Extracting the first frame of the standard video data according to the first video frame extraction rule, and using the first frame of the standard video data as the first video frame;

[0019] dividing the standard video data after extracting the first video frame into a plurality of video picture groups according to the second video frame extraction rule, and extracting the last frame of the video picture group as the second video frame;

[0020] According to the third video frame extraction rule, all frames in the video picture group after the second video frame is extracted are used as the third video frame.

[0021] Furthermore, the step of performing video diffusion on the first video frame based on the pre-trained diffusion model to generate a first extended video frame specifically includes:

[0022] Get the pre-trained diffusion model;

[0023] Extracting condition information from the first video frame to obtain video frame condition information;

[0024] The first video frame is input into the pre-trained diffusion model, and conditional control is performed according to a preset neural network structure and the video frame condition information to generate the first extended video frame.

[0025] Further, the step of generating a second extended video frame according to the second video frame and the first extended video frame specifically includes:

[0026] using the first extended video frame as a first reference frame;

[0027] Extracting optical flow information from the standard video data to obtain video optical flow information;

[0028] The second video frame is motion guided according to the first reference frame and the video optical flow information to generate the second extended video frame.

[0029] Further, the step of generating a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame specifically includes:

[0030] using the first extended video frame and the second extended video frame as second reference frames;

[0031] Calculate optical flow motion information between the second reference frame and the third video frame according to an optical flow estimation algorithm;

[0032] Reversely warping and forward warping the second reference frame according to the optical flow motion information to obtain a first warped reference frame and a second warped reference frame;

[0033] Calculating residual values ​​between the third video frame and the first warped reference frame and the second warped reference frame, and constructing a first residual map and a second residual map according to the residual values;

[0034] Perform forward propagation and backward propagation on a preset reference image according to the optical flow motion information to obtain a first occlusion image and a second occlusion image;

[0035] The third extended video frame is obtained by performing calculation according to the first warped reference frame, the second warped reference frame, the first residual map, the second residual map, the first occlusion map, and the second occlusion map.

[0036] Further, the step of calculating according to the first warped reference frame, the second warped reference frame, the first residual map, the second residual map, the first occlusion map, and the second occlusion map to obtain the third extended video frame specifically includes:

[0037] Calculate a first intermediate score and a second intermediate score according to the first residual map, the second residual map, the first occlusion map, and the second occlusion map;

[0038] Calculating a first matching score and a second matching score according to the first intermediate score and the second intermediate score;

[0039] A weighted calculation is performed according to the first matching score, the second matching score, and the warped reference frame to obtain the third extended video frame.

[0040] In order to solve the above technical problems, the embodiment of the present application further provides a video generation device, which adopts the following technical solution:

[0041] A data acquisition module, used for acquiring standard video data;

[0042] A data extraction module, used to extract a first video frame, a second video frame, and a third video frame from the standard video data according to a preset frame division rule;

[0043] A first video acquisition module, configured to perform video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame;

[0044] A second video acquisition module, configured to generate a second extended video frame according to the second video frame and the first extended video frame;

[0045] A third video acquisition module, configured to generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame;

[0046] The video output module is used to combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0048] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above video generation methods when executing the computer-readable instructions.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0050] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method described in any one of the above items are implemented.

[0051] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment obtains standard video data; extracts the first video frame, the second video frame, and the third video frame from the standard video data according to a preset frame division rule; performs video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame; generates a second extended video frame according to the second video frame and the first extended video frame; generates a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; and combines the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video. This effectively achieves the acquisition of continuous output video data with consistent content. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0054] Figure 2 A flowchart of an embodiment of a video generation method according to the present application;

[0055] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S10;

[0056] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S20;

[0057] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S30;

[0058] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S40;

[0059] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S50;

[0060] Figure 8 yes Figure 7 A flowchart of a specific implementation of step S506;

[0061] Fig. 9is a structural schematic diagram of an embodiment of a video generating device according to the present application;

[0062] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0064] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0066] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0067] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0068] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0069] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0070] It should be noted that the video generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the video generation device is generally arranged in the server / terminal device.

[0071] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0072] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for video generation according to the present application. The video generation method comprises the following steps:

[0073] Step S10, obtaining standard video data;

[0074] In this embodiment, standard video data is data information that meets processing requirements after preprocessing of initial video data. Initial video data refers to original video data collected by the device. The initial video data may be medical video data. Medical video data includes surgical demonstration videos, disease diagnosis videos, medical technology display videos, patient education videos, academic research videos, teaching case videos, telemedicine videos, etc.

[0075] Step S20, extracting a first video frame, a second video frame, and a third video frame from the standard video data according to a preset frame division rule;

[0076] In this embodiment, the first video frame, the second video frame, and the third video frame are respectively video frame data effectively extracted from the standard video data. The preset framing rule is rule information indicating how to extract the first video frame, the second video frame, and the third video frame respectively from the standard video data. The preset framing rule can be saved in a configuration file or a database and obtained by querying.

[0077] Step S30, performing video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame;

[0078] In this embodiment, the pre-trained diffusion model can adopt the Transformer model, and the training steps of the pre-trained diffusion model include: obtaining sample video data; selecting the Transformer architecture, including the design of the encoder (Encoder) and the decoder (Decoder); integrating the diffusion process into the Transformer model, the diffusion process includes adding noise to the input or output part of the model, and gradually reducing the noise during the training process to simulate the data generation process; defining the loss function according to the task requirements and adding conditional information loss; inputting the sample video data into the Transformer model for training to obtain the initial diffusion model; performing model evaluation on the initial diffusion model, and adjusting and optimizing the model according to the evaluation results to obtain the pre-trained diffusion model.

[0079] Step S40, generating a second extended video frame according to the second video frame and the first extended video frame;

[0080] In this embodiment, motion guidance is performed on the second video frame based on the optical flow information of the first extended video frame and the video frame data to generate a second extended video frame. The optical flow information refers to a two-dimensional vector field that describes the motion trajectory and speed of pixels over time in continuous frame images, and is used to capture the motion of objects in the image and the displacement information between adjacent frames. Motion guidance refers to using the optical flow information to guide the movement and arrangement of pixels in the second video frame to generate a continuous video frame that matches the first extended video frame. The continuous video frame is the second extended video frame.

[0081] Step S50, generating a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame;

[0082] In this embodiment, the generation of the third extended video frame is realized based on reverse warping, residual map construction, and occlusion map generation, wherein reverse warping is a processing method for reversely adjusting the video frame according to known motion information, the residual map refers to the difference map between the original video frame and the processed (such as prediction, encoding, etc.) video frame, and the occlusion map is an image used to represent the occluded area in the video frame. Through reverse warping, residual map construction, occlusion map generation and other operations, the third extended video frame is effectively generated.

[0083] Step S60: Combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video.

[0084] In this embodiment, the combination of the first extended video frame, the second extended video frame, and the third extended video frame includes: arranging the first extended video frame, the second extended video frame, and the third extended video frame in a correct order; checking whether the frame rates of the first extended video frame, the second extended video frame, and the third extended video frame match, and if not, adjusting the rate matching of each extended video frame; performing transition processing on adjacent frames of the checked first extended video frame, the second extended video frame, and the third extended video frame to smoothly connect each extended video frame; encoding and compressing the connected extended video frame sequence to generate an output video.

[0085] In this embodiment, the above method can be applied to a medical service system, in which the original medical video is diffused in the medical service system to generate a corresponding medical diffusion video. Specifically, in this embodiment, the medical service system can be one or more of a medical insurance system and a disease insurance system, the standard video data is medical video data collected by a device, the standard video data, the preset frame division rules, and the pre-trained diffusion model are all stored in the medical insurance system and the disease insurance system and obtained from the database of the above system, and the output video is generated by the above system through the method of this embodiment and stored in the system database for preservation.

[0086] This embodiment obtains standard video data; extracts the first video frame, the second video frame, and the third video frame from the standard video data according to a preset frame division rule; performs video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame; generates a second extended video frame according to the second video frame and the first extended video frame; generates a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; and combines the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video. This effectively achieves the acquisition of continuous output video data with consistent content.

[0087] This embodiment can be applied to crop planting risk warning. By generating risk warning information for real-time warning, the risks of crop planting can be effectively compared to reduce the economic losses of crop planting.

[0088] refer to Figure 3 In some optional implementations of this embodiment, step S10 includes the following steps:

[0089] Step S101, obtaining a video extraction identifier;

[0090] In this embodiment, the video extraction identifier is unique identification information corresponding to the initial video data, and the video extraction identifier may be a data ID or a UUID.

[0091] Step S102, extracting initial video data from a database according to the video extraction identifier;

[0092] In this embodiment, a matching query is performed in the database through the video extraction identifier, and the matched initial video data is extracted, thereby obtaining the initial video data.

[0093] Step S103: pre-process the initial video data to obtain the standard video data.

[0094] In this embodiment, preprocessing includes decoding the initial video data from its original encoding format to obtain decoded video data; converting the decoded video data into a unified format and encoding standard to facilitate subsequent processing and analysis; splitting the decoded video data into separate frames; removing noise from the video frames, including image noise and noise caused by camera motion, light changes, etc.; performing data enhancement and standardization on the video frames after noise removal to obtain labeled video data.

[0095] This embodiment obtains a video extraction identifier; extracts initial video data from a database according to the video extraction identifier; and preprocesses the initial video data to obtain standard and effective standard video data to provide reliable data support for subsequent processing.

[0096] refer to Figure 4 In some optional implementations of this embodiment, step S20 includes the following steps:

[0097] Step S201, obtaining a preset framing rule, parsing the preset framing rule, and obtaining a first video frame extraction rule, a second video frame extraction rule, and a third video frame extraction rule;

[0098] In this embodiment, the parsing of the preset framing rules includes reading the preset framing rules, performing grammatical parsing on the preset framing rules to obtain framing rule items, identifying corresponding information of the framing rule items to obtain the first video frame extraction rules, the second video frame extraction rules, and the third video frame extraction rules.

[0099] Step S202, extracting the first frame of the standard video data according to the first video frame extraction rule, and using the first frame of the standard video data as the first video frame;

[0100] In this embodiment, the time sequence information of each frame in the standard video frame data can be obtained, and then the first frame can be determined according to the time sequence information, so as to perform accurate extraction.

[0101] Step S203, dividing the standard video data after extracting the first video frame into a plurality of video picture groups according to the second video frame extraction rule, and extracting the last frame of the video picture group as the second video frame;

[0102] In this embodiment, according to the segmentation basis (such as time interval, frame number interval, etc.) defined in the second video frame extraction rule, the standard video data after extracting the first video frame is divided into several video picture groups, and the actual segmentation operation can be performed using video processing software or libraries (such as OpenCV, FFmpeg, etc.), wherein each video picture group includes several video frame pictures. Then, each video picture group is traversed, and the single video frame at the end of the time sequence is extracted from the frame list of each video picture group, so as to obtain the last frame of each video picture group, and a video frame set is generated according to the last frame, and the video frame set is the second video frame.

[0103] Step S204: extract all frames in the video picture group after the second video frame is extracted as the third video frame according to the third video frame extraction rule.

[0104] In this embodiment, the remaining video frames in the video picture group after the second video frame is extracted are extracted, and the extracted video frames are saved as the third video frames, thereby effectively acquiring the third video frame.

[0105] This embodiment obtains a preset framing rule, parses the preset framing rule, and obtains a first video frame extraction rule, a second video frame extraction rule, and a third video frame extraction rule; extracts the first frame of the standard video data according to the first video frame extraction rule, and uses the first frame of the standard video data as the first video frame; divides the standard video data after the first video frame is extracted into a plurality of video picture groups according to the second video frame extraction rule, and extracts the last frame of the video picture group as the second video frame; and uses all frames in the video picture group after the second video frame is extracted as the third video frame according to the third video frame extraction rule. Thus, the first video frame, the second video frame, and the third video frame are effectively extracted from the standard video data according to different video frame extraction rules, so as to facilitate the subsequent generation of the first extended video frame, the second extended video frame, and the third extended video frame.

[0106] refer to Figure 5 In some optional implementations of this embodiment, step S30 includes the following steps:

[0107] Step S301, obtaining a pre-trained diffusion model;

[0108] In this embodiment, the pre-trained diffusion model can be obtained by matching and querying the model database through model extraction identification information. The model extraction identification information is unique identification information corresponding to the pre-trained diffusion model. The model extraction identification can be generated by an event, and the trigger condition of the event can be obtaining the first video frame.

[0109] Step S302, extracting condition information from the first video frame to obtain video frame condition information;

[0110] In this embodiment, the video frame condition information may be a depth map extracted from the first video frame. The depth map is an image in which the value of each pixel represents the depth (or distance) information of the corresponding point in the first video frame scene in three-dimensional space. In this embodiment, the extraction of the depth map can be achieved by processing the first video frame using a depth estimation method (such as predicting depth through a machine learning model) using a visual processing tool. The processing of the first video frame includes the steps of image processing, feature extraction, matching, and depth calculation. Finally, based on the output of the depth estimation algorithm, the depth map of the first video frame is effectively generated.

[0111] Step S303: input the first video frame into the pre-trained diffusion model, and perform conditional control according to a preset neural network structure and the video frame condition information to generate the first extended video frame.

[0112] In this embodiment, the preset neural network structure adopts ControlNet, and ControlNet is configured as a neural network structure to add a conditional control function to the pre-trained diffusion model, and the conditional control is achieved by locking the parameters of the pre-trained diffusion model and creating a trainable encoding layer copy. Conditional control is performed according to the preset neural network structure and the video frame conditional information, and the step of generating the first extended video frame includes: matching the extracted conditional information (such as a depth map) with the requirements of ControlNet through appropriate preprocessing steps (such as scaling, normalization, etc.), and inputting the preprocessed conditional information and the first video frame (or its potential representation) into ControlNet together; ControlNet will use this information to guide the generation process of the diffusion model. Under the guidance of ControlNet, the pre-trained diffusion model generates an image according to the input video frame conditional information, and generates a first extended video frame related to the first video frame but containing more details and meeting the depth information condition through multiple iterations or continuous application of the diffusion process.

[0113] This embodiment obtains a pre-trained diffusion model; extracts conditional information from the first video frame to obtain video frame conditional information; inputs the first video frame into the pre-trained diffusion model, and performs conditional control according to a preset neural network structure and the video frame conditional information to generate the first extended video frame. Thus, the first video frame is effectively diffused to generate a first extended video frame whose content is consistent with the information of the first video frame, so as to facilitate subsequent video frame combination operations.

[0114] Continue to refer Figure 6 In some optional implementations of this embodiment, step S40 includes the following steps:

[0115] Step S401, using the first extended video frame as a first reference frame;

[0116] In this embodiment, the first reference frame is a base frame referenced by the generation process of the second extended video frame. The first extended video frame is set as the first reference frame in the initial stage of the second extended video frame processing flow, thereby achieving the acquisition of the first reference frame.

[0117] Step S402, extracting optical flow information from the standard video data to obtain video optical flow information;

[0118] In this embodiment, the step of extracting optical flow information includes: converting the frame image of the video frame data into a format suitable for optical flow calculation (converting the image into a grayscale image); selecting a suitable optical flow algorithm for calculation, and commonly used optical flow algorithms include the Lucas-Kanade method, the Horn-Schunck method (HS optical flow method), the Farneback optical flow method, and an optical flow estimation algorithm based on deep learning (such as FlowNet, RAFT, etc.). In this embodiment, the HS optical flow method is adopted; using the optical flow algorithm to estimate the pixel motion between consecutive frames of the frame image to obtain an optical flow field, wherein the optical flow field describes the motion vector of each pixel in the image between consecutive frames; filtering, interpolation, etc. are performed on the calculated optical flow field to remove noise, fill missing values, etc.; converting the optical flow field into a visualized optical flow image, wherein the optical flow image represents the optical flow vector in a color-coded manner, and the color and brightness can represent the direction and size of the vector. The optical flow image finally obtained is the video optical flow information.

[0119] Step S403: performing motion guidance on the second video frame according to the first reference frame and the video optical flow information to generate the second extended video frame.

[0120] In this embodiment, the step of motion guidance includes: mapping the motion vector in the video optical flow information to the second video frame, and matching the pixel or feature point position in the optical flow information with the corresponding position in the second video frame to determine the motion of the corresponding part in the second video frame; based on the time interval between the first reference frame and the second video frame and the optical flow information, performing motion prediction on the pixels or feature points in the second video frame, which can be achieved through linear interpolation, polynomial fitting or other prediction models, so as to effectively estimate the positions of these pixels or feature points in the second extended video frame; according to the prediction results, synthesizing the pixels or feature points in the second video frame according to the predicted motion trajectory, and generating a second extended video frame that conforms to the law of motion by rearranging and interpolating the pixel values.

[0121] In this embodiment, the first extended video frame is used as the first reference frame; optical flow information is extracted from the standard video data to obtain video optical flow information; motion guidance is performed on the second video frame according to the first reference frame and the video optical flow information to generate the second extended video frame. Thus, the second extended video frame that is consistent with the optical flow motion of the video frame data is effectively obtained to facilitate the subsequent video frame combination operation.

[0122] Continue to refer Figure 7 In some optional implementations of this embodiment, step S50 includes the following steps:

[0123] Step S501, using the first extended video frame and the second extended video frame as second reference frames;

[0124] In this embodiment, the second reference frame is a base frame used as a reference in the generation process of the third extended video frame. The first extended video frame and the second extended video frame are set as second reference frames in the initial stage of the third extended video frame processing flow, thereby acquiring the second reference frame.

[0125] Step S502, calculating optical flow motion information between the second reference frame and the third video frame according to an optical flow estimation algorithm;

[0126] In this embodiment, the optical flow estimation algorithm adopts the Horn-Schunck method (HS optical flow method). Before calculating the optical flow motion information between the second reference frame and the third reference frame, the second reference frame and the third reference frame may be pre-processed by grayscale processing and noise processing to obtain standard second reference frames and third reference frames, and then feature point detection and feature description are performed on the processed second reference frame and the third reference frame to generate corresponding feature descriptors. Calculating the optical flow motion information between the second reference frame and the third reference frame includes: using the spatiotemporal gradient of the image grayscale to calculate the motion speed of the pixel, which can be achieved by performing temporal and spatial differential operations on the grayscale image and solving the optical flow equation. After the motion speed of the pixel is calculated, the corresponding optical flow field is generated according to the motion speed of the pixel, thereby obtaining the optical flow motion information.

[0127] Step S503, reversely warping and forward warping the second reference frame according to the optical flow motion information to obtain a first warped reference frame and a second warped reference frame;

[0128] In this embodiment, the reverse warping step includes calculating the reverse mapping: calculating the corresponding position of each pixel in the second reference frame in the adjacent frame (usually the previous frame) according to the optical flow motion information, and the corresponding position is usually a non-integer value, which represents the "floating" position of the pixel in the adjacent frame; interpolation calculation: using an interpolation method (such as bilinear interpolation, trilinear interpolation, etc.) to calculate the pixel value of the corresponding position of each pixel in the second reference frame in the adjacent frame; generating a first warped reference frame: assigning the pixel value obtained by the interpolation calculation to the corresponding pixel position in the second reference frame, thereby generating a first warped reference frame. The forward warping step includes calculating the forward mapping: mapping the pixel points in the adjacent frame to the second reference frame, which can be achieved by first inverting or adjusting the optical flow information and then performing the pixel mapping step; processing the mapping relationship: calculating the corresponding position of each pixel in the adjacent frame in the second reference frame according to the adjusted optical flow information; interpolation or resampling: using an interpolation method or a resampling technique to calculate the pixel value of the corresponding position in the second reference frame, thereby generating a second warped reference frame.

[0129] Step S504, calculating residual values ​​between the third video frame and the first warped reference frame and the second warped reference frame, and constructing a first residual map and a second residual map according to the residual values;

[0130] In this embodiment, by traversing each pixel of the third video frame and the first warped reference frame, the third video frame and the second warped reference frame, and comparing the pixel value difference at the same position, the pixel value difference is the residual value of the pixel, wherein the residual value can be the direct difference between the pixel values, or the difference after some transformation (such as normalization, taking the absolute value, etc.). The construction of the residual map includes the following steps: determining the representation method of the residual map, the residual map can be presented in the form of a grayscale map, wherein the grayscale value of each pixel represents the residual value of the point, the higher (or lower) the grayscale value, the larger (or smaller) the residual value, and a color map can also be selected to represent the residual map, and different colors are used to distinguish the size and positive and negative of the residual value; according to the selected representation method, the calculated residual value is mapped to the corresponding color or grayscale value, and the mapping can be performed through a mapping function or a lookup table, so as to convert the residual value into a color or grayscale value; using the mapped color or grayscale value to generate the first residual map and the second residual map, which can be achieved through an image processing library.

[0131] Step S505, forward propagating and backward propagating the preset reference image according to the optical flow motion information to obtain a first occlusion image and a second occlusion image;

[0132] In this embodiment, the preset reference image is a preset image that is used as a reference for generating an occlusion map. The forward propagation step includes: creating an occlusion map of the same size as the reference image, and initializing all pixel values ​​to a non-occluded state (for example, represented by a specific color or value); for each pixel (or selected feature point) in the reference image, forward propagation is performed using the optical flow velocity vector corresponding to the optical flow motion information; during the forward propagation process, it is checked whether the expected position of each pixel is within the image boundary or overlaps with other pixels. If the expected position of the pixel exceeds the image boundary or the position is occupied by other pixels (i.e., occlusion occurs), the pixel is marked as being in an occluded state. The occlusion detection method includes a boundary check: checking whether the expected position coordinates of the pixel exceed the boundary range of the image, and a pixel coverage check: checking in the occlusion map whether the expected position has been marked as occluded by other pixels or has a valid pixel value; based on the occlusion detection result, the state of the corresponding pixel in the occlusion map is updated, wherein the pixel marked as occluded can be represented by a specific color or value to generate a first occlusion map. The steps of back propagation include: converting or inverting the optical flow motion information so as to map it from the next frame or target frame back to the current frame or reference frame; using the inverted optical flow information to map the pixels in the next frame or target frame back to the position of the reference map; comparing the reference map and the "reconstructed" reference map after reverse mapping, indirectly inferring the occlusion information through the difference, and generating a second occlusion map based on the occlusion information.

[0133] Step S506: performing calculation according to the first warped reference frame, the second warped reference frame, the first residual map, the second residual map, the first occlusion map, and the second occlusion map to obtain the third extended video frame.

[0134] In this embodiment, the calculations performed based on the residual map, the occlusion map, and the distorted reference frame correspond to the first calculation formula, the second calculation formula, and the third calculation formula. The first calculation formula, the second calculation formula, and the third calculation formula are stored in the generation processing information of the third extended video frame, and the generation processing information can be extracted from the database.

[0135] In this embodiment, the first extended video frame and the second extended video frame are used as the second reference frame; the optical flow motion information between the second reference frame and the third video frame is calculated according to the optical flow estimation algorithm; the second reference frame is reversely warped and forwardly warped according to the optical flow motion information to obtain the first warped reference frame and the second warped reference frame; the residual value of the third video frame and the first warped reference frame and the second warped reference frame is calculated, and the first residual map and the second residual map are constructed according to the residual value; the preset reference map is forward propagated and reversely propagated according to the optical flow motion information to obtain the first occlusion map and the second occlusion map; the third extended video frame is calculated according to the first warped reference frame, the second warped reference frame, the first residual map, the second residual map, the first occlusion map, and the second occlusion map. In this way, the third extended video frame for motion prediction according to the optical flow motion information and the residual value is effectively obtained, so as to facilitate the subsequent video frame combination operation.

[0136] Continue to refer Figure 8 In some optional implementations of this embodiment, step S506 includes the following steps:

[0137] Step S5061, calculating a first intermediate score and a second intermediate score according to the first residual map, the second residual map, the first occlusion map, and the second occlusion map;

[0138] In this embodiment, the first intermediate score and the second intermediate score are calculated according to a first calculation formula, and the first calculation formula is:

[0139]

[0140] in, is the first intermediate score, is the second intermediate score, is the occlusion value of the first occlusion map, is the occlusion value of the second occlusion map, is the residual value of the first residual map, is the residual value of the second residual graph, β is the preset weight, the initial value is set to 1, and can be adjusted accordingly according to actual conditions.

[0141] Step S5062, calculating a first matching score and a second matching score according to the first intermediate score and the second intermediate score;

[0142] In this embodiment, the first matching score and the second matching score are calculated according to a second calculation formula, and the second calculation formula is:

[0143]

[0144] in, is the first matching score, is the second matching score, is the first intermediate score, is the second intermediate fraction, and τ is the time constant.

[0145] Step S5063: Perform weighted calculation according to the first matching score, the second matching score, and the warped reference frame to obtain the third extended video frame.

[0146] In this embodiment, weighted calculation is performed according to a third calculation formula to obtain a third extended video frame. The third calculation formula is:

[0147]

[0148] in, is the third extended video frame, is the first matching score, is the second matching score, is the first matching weight corresponding to the first matching score, is the second matching weight corresponding to the second matching score, the first matching weight and the second matching weight are preset values, wherein the first matching weight The initial setting is 0.5, the second matching weight The initial setting is 0.5, which can be adjusted accordingly according to actual conditions.

[0149] This embodiment calculates a first intermediate score and a second intermediate score according to the first residual map, the second residual map, the first occlusion map, and the second occlusion map; calculates a first matching score and a second matching score according to the first intermediate score and the second intermediate score; and performs weighted calculations according to the first matching score, the second matching score, and the warped reference frame, thereby effectively performing calculations based on the residual map, the occlusion map, and the warped reference frame to obtain a third extended video frame generated based on the residual map, the occlusion map, and the warped reference frame, thereby facilitating subsequent video frame combination operations.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0151] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0152] Further references Fig. 9 , as a response to the above Figure 1 The present application provides an embodiment of a video generation device, and the device embodiment is Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0153] like Fig. 9 As shown, the video generation device 700 described in this embodiment includes: a data acquisition module 701, a data extraction module 702, a first video acquisition module 703, a second video acquisition module 704, a third video acquisition module 705, and a video output module 706. Among them:

[0154] The data acquisition module 701 is used to acquire standard video data;

[0155] A data extraction module 702 is used to extract a first video frame, a second video frame, and a third video frame from the standard video data according to a preset frame division rule;

[0156] A first video acquisition module 703, configured to perform video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame;

[0157] A second video acquisition module 704, configured to generate a second extended video frame according to the second video frame and the first extended video frame;

[0158] A third video acquisition module 705, configured to generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame;

[0159] The video output module 706 is configured to combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video.

[0160] By adopting the above-mentioned video generation device, this embodiment can obtain standard video data; extract the first video frame, the second video frame, and the third video frame from the standard video data according to the preset frame division rule; perform video diffusion on the first video frame based on the pre-trained diffusion model to generate a first extended video frame; generate a second extended video frame according to the second video frame and the first extended video frame; generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video. This effectively achieves the acquisition of continuous output video data with consistent content.

[0161] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0162] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0163] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0164] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions of the video generation method, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0165] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer-readable instructions stored in the memory 81 or process data, such as computer-readable instructions for running the video generation method.

[0166] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0167] By adopting the above-mentioned computer device, this embodiment can obtain standard video data; extract the first video frame, the second video frame, and the third video frame from the standard video data according to the preset frame division rule; perform video diffusion on the first video frame based on the pre-trained diffusion model to generate a first extended video frame; generate a second extended video frame according to the second video frame and the first extended video frame; generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video. This effectively achieves the acquisition of continuous output video data with consistent content.

[0168] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the video generation method as described above.

[0169] By adopting the above-mentioned computer-readable storage medium, this embodiment can obtain standard video data; extract the first video frame, the second video frame, and the third video frame from the standard video data according to the preset frame division rule; perform video diffusion on the first video frame based on the pre-trained diffusion model to generate a first extended video frame; generate a second extended video frame according to the second video frame and the first extended video frame; generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video. This effectively achieves the acquisition of continuous output video data with consistent content.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0171] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0172] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A video generation method, characterized in that: The steps include: Get standard video data; Extracting a first video frame, a second video frame, and a third video frame from the standard video data according to a preset frame division rule; Performing video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame; generating a second extended video frame according to the second video frame and the first extended video frame; Generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; The first extended video frame, the second extended video frame, and the third extended video frame are combined to obtain an output video.

2. The video generation method according to claim 1, characterized in that: The step of obtaining standard video data specifically includes: Get the video extraction identifier; Extracting initial video data from a database according to the video extraction identifier; The initial video data is preprocessed to obtain the standard video data.

3. The video generation method according to claim 1, characterized in that: The step of extracting the first video frame, the second video frame, and the third video frame from the standard video data according to the preset frame division rule specifically includes: Obtaining a preset framing rule, parsing the preset framing rule, and obtaining a first video frame extraction rule, a second video frame extraction rule, and a third video frame extraction rule; Extracting the first frame of the standard video data according to the first video frame extraction rule, and using the first frame of the standard video data as the first video frame; dividing the standard video data after extracting the first video frame into a plurality of video picture groups according to the second video frame extraction rule, and extracting the last frame of the video picture group as the second video frame; According to the third video frame extraction rule, all frames in the video picture group after the second video frame is extracted are used as the third video frame.

4. The video generation method according to claim 1, characterized in that: The step of performing video diffusion on the first video frame based on the pre-trained diffusion model to generate a first extended video frame specifically includes: Get the pre-trained diffusion model; Extracting condition information from the first video frame to obtain video frame condition information; The first video frame is input into the pre-trained diffusion model, and conditional control is performed according to a preset neural network structure and the video frame condition information to generate the first extended video frame.

5. The video generation method according to claim 1, characterized in that: The step of generating a second extended video frame according to the second video frame and the first extended video frame specifically includes: using the first extended video frame as a first reference frame; Extracting optical flow information from the standard video data to obtain video optical flow information; The second video frame is motion guided according to the first reference frame and the video optical flow information to generate the second extended video frame.

6. The video generation method according to claim 1, characterized in that: The step of generating a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame specifically includes: using the first extended video frame and the second extended video frame as second reference frames; Calculate optical flow motion information between the second reference frame and the third video frame according to an optical flow estimation algorithm; Reversely warping and forward warping the second reference frame according to the optical flow motion information to obtain a first warped reference frame and a second warped reference frame; Calculating residual values ​​between the third video frame and the first warped reference frame and the second warped reference frame, and constructing a first residual map and a second residual map according to the residual values; Perform forward propagation and backward propagation on a preset reference image according to the optical flow motion information to obtain a first occlusion image and a second occlusion image; The third extended video frame is obtained by performing calculation according to the first warped reference frame, the second warped reference frame, the first residual map, the second residual map, the first occlusion map, and the second occlusion map.

7. The video generation method according to claim 6, characterized in that: The step of performing calculation according to the first warped reference frame, the second warped reference frame, the first residual map, the second residual map, the first occlusion map, and the second occlusion map to obtain the third extended video frame specifically includes: Calculate a first intermediate score and a second intermediate score according to the first residual map, the second residual map, the first occlusion map, and the second occlusion map; Calculating a first matching score and a second matching score according to the first intermediate score and the second intermediate score; A weighted calculation is performed according to the first matching score, the second matching score, and the warped reference frame to obtain the third extended video frame.

8. A video generating device, characterized in that: include: A data acquisition module, used for acquiring standard video data; A data extraction module, used to extract a first video frame, a second video frame, and a third video frame from the standard video data according to a preset frame division rule; A first video acquisition module, configured to perform video diffusion on the first video frame based on a pre-trained diffusion model to generate a first extended video frame; A second video acquisition module, configured to generate a second extended video frame according to the second video frame and the first extended video frame; A third video acquisition module, configured to generate a third extended video frame according to the first extended video frame, the second extended video frame, and the third video frame; The video output module is used to combine the first extended video frame, the second extended video frame, and the third extended video frame to obtain an output video.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are implemented.

Images