Video generation method and device, computer program product and electronic equipment

Abstract

The invention relates to the technical field of computers, and relates to a video generation method and device, a computer program product and electronic equipment. The video generation method comprises the following steps: obtaining potential spatial features of a current video frame, wherein the current video frame is obtained by denoising an initial noise image through a diffusion model; decision parameters of a preset decision model are adjusted based on the potential spatial features of the current video frame, the preset decision model is used for generating diffusion parameters, and the diffusion parameters are used for controlling the denoising process of the diffusion model; and based on diffusion parameters generated by the preset decision model after the decision parameters are adjusted, performing denoising processing on the potential spatial features of the current video frame through the diffusion model, and generating a next video frame. The quality of the generated video can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a video generating method, a video generating device, a computer program product, and an electronic device. Background Art

[0002] With the development of computer technology, deep learning and generative models have gradually developed, and video generation technology based on diffusion models has made great progress, especially in generating high-quality static images. Diffusion models have demonstrated excellent performance. Unlike image generation, video generation requires that each frame has high quality, and also needs to ensure high quality and coherence between video frames. However, the current diffusion model still has the disadvantage of low quality in generating videos.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a video generation method and device, a computer program product and an electronic device, thereby improving the quality of the generated video at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a video generation method is provided, including: obtaining latent spatial features of a current video frame, the current video frame being obtained by denoising an initial noise image through a diffusion model; adjusting decision parameters of a preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, denoising the latent spatial features of the current video frame through the diffusion model to generate the next video frame.

[0007] In an exemplary embodiment of the present disclosure, the diffusion parameters generated by the preset decision model include noise level parameters and / or conditional input information; the noise level parameters are used to control the amplitude of denoising, and the conditional input information is used to control the content of the generated video frame.

[0008] In an exemplary embodiment of the present disclosure, decision parameters of a preset decision model are adjusted based on the latent space features of a current video frame, including: determining current video reward information based on the latent space features of the current video frame; wherein the current video reward information is used to evaluate the performance of current diffusion parameters used to generate the current video frame; and adjusting the decision parameters of the preset decision model based on the current video reward information.

[0009] In an exemplary embodiment of the present disclosure, current video reward information is determined based on latent space features of the current video frame, including: determining a first reward function value based on feature differences between the latent space features of the current video frame and the latent space features of an adjacent previous video frame; evaluating the picture quality of the current video frame using the latent space features of the current video frame, and determining a second reward function value based on the evaluation result; performing a physical rationality evaluation on the current video frame using the latent space features of the current video frame, and determining a third reward function value based on the evaluation result; and determining the current video reward information based on the first reward function value, the second reward function value, and the third reward function value.

[0010] In an exemplary embodiment of the present disclosure, decision parameters of a preset decision model are adjusted based on current video reward information, including: obtaining historical video reward information of historical video frames; determining an expected cumulative reward based on the current video reward information and the historical video reward information; and adjusting the decision parameters of the preset decision model to maximize the expected cumulative reward.

[0011] In an exemplary embodiment of the present disclosure, the decision parameters of a preset decision model are adjusted to maximize the expected cumulative reward, including: performing advantage estimation based on current video reward information and historical video reward information, and determining an advantage estimation result, wherein the advantage estimation result is used to reflect the advantage of current diffusion parameters over historical diffusion parameters; obtaining current decision parameters corresponding to a current video frame, and obtaining historical diffusion parameters and historical decision parameters corresponding to an adjacent previous video frame; determining a proximal strategy optimization loss function based on the advantage estimation result, current diffusion parameters, current decision parameters, historical diffusion parameters and historical decision parameters corresponding to a previous video frame; and adjusting the decision parameters of the preset decision model according to the proximal strategy optimization loss function.

[0012] In an exemplary embodiment of the present disclosure, if the diffusion parameters generated by the preset decision model include noise level parameters and conditional input information, based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, the latent spatial features of the current video frame are denoised through the diffusion model to generate the next video frame, including: adding random noise to the latent spatial features of the current video frame based on the noise level parameters in the diffusion parameters; and denoising the latent spatial features to which the random noise is added according to the conditional input information to generate the next video frame.

[0013] In an exemplary embodiment of the present disclosure, before obtaining the latent spatial features of the current video frame, the method further includes: obtaining initial random noise, and determining an initial noise image according to the initial random noise.

[0014] According to one aspect of the present disclosure, a video generation device is provided, including: a feature acquisition module, used to acquire latent spatial features of a current video frame, the current video frame being obtained by denoising an initial noise image through a diffusion model; a parameter adjustment module, used to adjust decision parameters of a preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; an image generation module, used to denoise the latent spatial features of the current video frame through a diffusion model based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, to generate a next video frame.

[0015] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0016] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.

[0017] The video generation method in the exemplary embodiment of the present disclosure obtains the latent spatial features of the current video frame, and adjusts the decision parameters of the preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; based on the diffusion parameters generated by the preset decision model after the decision parameters are updated, the latent spatial features of the current video frame are denoised by the diffusion model to generate the next video frame. On the one hand, in the process of gradually denoising the initial noise image based on the diffusion model, the preset decision model of reinforcement learning is introduced, and when the video frames are generated frame by frame, the diffusion parameters of the diffusion model are adjusted by the preset decision model to achieve frame-by-frame optimization, thereby improving the quality of the generated video frames frame by frame. On the other hand, the decision parameters of the preset decision model are updated by using the latent spatial features of the current video frame to gradually improve the accuracy of the generated diffusion parameters, that is, to improve the decision accuracy of the preset decision model, thereby further improving the accuracy of the denoising process of the latent spatial features of the current video frame, ensuring the quality and coherence between video frames, and improving the overall quality of the final generated video. On the other hand, processing is performed in the latent spatial feature dimension, without the need to process high-dimensional pixel data, thereby improving processing efficiency.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown in an exemplary and non-limiting manner.

[0020] Figure 1 An application environment according to an exemplary embodiment of the present disclosure is shown.

[0021] Figure 2 A flowchart of a video generating method according to an exemplary embodiment of the present disclosure is shown.

[0022] Figure 3 A flow chart of an implementation method for adjusting decision parameters of a preset decision model according to an exemplary embodiment of the present disclosure is shown.

[0023] Figure 4 A flowchart of an implementation method for determining current video reward information according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 5 A flow chart of adjusting a decision parameter of a preset decision model based on current video reward information according to an exemplary embodiment of the present disclosure is shown.

[0025] Figure 6 A flowchart of adjusting a decision parameter of a preset decision model based on an expected cumulative reward according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 7 A schematic diagram showing the composition of a video generating device according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 8 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0028] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0029] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their detailed description will be omitted.

[0030] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0033] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0034] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0035] The technical solution provided by the exemplary embodiments of the present disclosure involves artificial intelligence machine learning technology, which uses machine learning technology to denoise the initial noisy image frame by frame, and optimizes the denoising parameters frame by frame through a preset decision model. At the same time, the decision parameters of the preset decision model are updated, thereby improving the overall quality of the final generated video.

[0036] It should be noted that the video generation method of the exemplary embodiment of the present disclosure can be applied to various video generation tasks, such as generating videos of coherent dynamic scenes, such as scenes of human movement, object interaction, changes in the natural environment, etc. The exemplary embodiment of the present disclosure does not specifically limit the application field of the video generation method.

[0037] Video generation is different from image generation. In addition to requiring realistic visual effects for each frame, it is also necessary to ensure the coherence, physical rationality and content consistency between video frames. However, when generating videos through video diffusion models, there is a disadvantage of low quality. Based on this, an exemplary embodiment of the present disclosure provides a video generation method, which improves the quality and coherence of the video by optimizing the frame-by-frame denoising process of the diffusion model.

[0038] The video generation method provided by the exemplary embodiment of the present disclosure can be applied to Figure 1 In the application environment shown, the terminal 101 communicates with the server 102 via a network. The data storage system can store data that the server 102 needs to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0039] In an exemplary embodiment, the video generation method provided by the exemplary embodiment of the present disclosure may be executed by the server 102, and the corresponding video generation device is disposed in the server 102. Correspondingly, in this mode of execution by the server 102, the server 102 may start to execute the steps in the technical solution of the exemplary embodiment of the present disclosure in response to a trigger command, wherein the trigger command may be sent by a terminal used by a user, or may be triggered locally by the server in response to some automation events.

[0040] The server 102 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server 102 may perform background tasks.

[0041] Furthermore, in another exemplary embodiment, the terminal 101 may also have similar functions to the server 102 , so as to execute the video generating method provided by the exemplary embodiment of the present disclosure.

[0042] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 101 may also be referred to as a mobile terminal, a terminal device, a mobile device, etc. The exemplary embodiments of the present disclosure do not limit the type of the terminal 101.

[0043] In addition, the technical solution of the exemplary embodiment of the present disclosure can also be executed by the terminal 101 and the server 102 in collaboration. In this way of collaborative execution by the terminal 101 and the server 102, some steps in the technical solution provided by the exemplary embodiment of the present disclosure are executed by the terminal 101, and other steps are executed by the server 102.

[0044] It should be noted that in this mode of collaborative execution by the terminal 101 and the server 102, the steps respectively executed by the terminal 101 and the server 102 can be dynamically adjusted according to actual conditions, and there is no special restriction on this.

[0045] The terminal 101 and the server 102 may be directly or indirectly connected via wireless communication, and the exemplary embodiments of the present disclosure do not impose any special limitation thereto.

[0046] In an exemplary embodiment of the present disclosure, a video generation method is provided, referring to Figure 2 FIG. 1 is a flowchart of a video generation method according to an exemplary embodiment of the present disclosure. The video generation method includes steps S210 to S230, which are specifically as follows:

[0047] In step S210, the latent spatial features of the current video frame are obtained. The current video frame is obtained by denoising an initial noisy image using a diffusion model.

[0048] In an exemplary embodiment of the present disclosure, the current video frame refers to the latest video frame that has been generated. The latent space feature is an abstract multi-dimensional space used to encode a meaningful internal representation of an external observation event. In deep learning, high-dimensional input data is encoded into a low-dimensional internal representation through latent space features, so that the model can more effectively perform tasks such as classification, regression or reconstruction. An exemplary embodiment of the present disclosure can extract the latent space feature of the current video frame through the encoder part (encoder) of the diffusion model. The latent space feature contains abstract information of the scene, such as high-level features such as the position, shape, and action of the object, and can accurately represent the real content of the current video frame. The latent space feature extracted from the current video frame (for example, the tth frame) can be expressed by the following formula (1):

[0049] z t =Encoder(x t ) (1)

[0050] Among them, z t is the potential spatial feature of the current video frame, x t For the current video frame, Encoder() represents the encoding operation of the encoder of the diffusion model.

[0051] The working principle of the diffusion model of the video is to gradually add noise to the data set and then learn how to reverse this process to generate high-quality data, that is, to generate high-quality video frames. The exemplary embodiment of the present disclosure uses the diffusion model to gradually denoise the initial noisy image to obtain a clear image, that is, a video frame. The latent space features of the video frame are then extracted and used to repeat the denoising process to generate the next video frame.

[0052] Among them, the denoising process of the diffusion model involves parameters such as the number of time steps, noise level, time-dependent variance, and noise scale. The denoising process usually involves multiple time steps, and each time step will gradually reduce the noise. The number of time steps represents the total number of steps in the denoising process, which needs to be determined according to the specific task and model design, and there is no restriction on this. The noise level determines the intensity of the noise added to the current video frame, which can determine the detail accuracy of the generated video frame. The exemplary embodiment of the present disclosure can adjust the noise level frame by frame through a preset decision model, which will be introduced later. In the diffusion process, the noise variance of each time step determines the variance of the added noise. This parameter varies with the time step to ensure that the noise is gradually reduced. In the denoising process, the noise scale is used to calculate the conditional Gaussian probability to ensure that the original image is gradually restored from the noise. The noise scale is a parameter that varies with the time step and is used to control the intensity of denoising.

[0053] It can be seen that the diffusion parameters of the exemplary embodiments of the present disclosure may include the above-mentioned noise level, which is used to determine the noise intensity added to the current video frame when generating the next video frame. It should also be understood that other parameters can be adjusted according to specific tasks, and no specific limitation is made to this.

[0054] In some optional embodiments, before obtaining the latent space features of the current video frame, initial random noise can be obtained, and an initial noise image can be determined based on the initial random noise. Specifically, starting with random noise, denoising is performed step by step through a diffusion process to generate a first video frame. The latent space features of the first video frame are extracted, and the latent space features of the first video frame are subjected to a denoising and denoising process using a diffusion model to obtain a second video frame, and so on, to obtain a combination of various video frames into a complete video. Based on this, the current video frame may refer to any one of the first video frame, the second video frame, ..., the N-1th video frame, where N is the number of frames of the complete video finally generated. The exemplary embodiment of the present disclosure takes the process of generating the next video frame based on the current video frame as an example to illustrate the process of generating a video.

[0055] In step S220, decision parameters of a preset decision model are adjusted based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model.

[0056] In an exemplary embodiment of the present disclosure, the preset decision model is a reinforcement learning model. Reinforcement learning is a method of machine learning and can be used to train a decision model for controlling the action decision of a target object in a target environment. The training process of the preset decision model includes: using a neural network model based on reinforcement learning to learn the action decision of the target object, and learning to control the target object to adapt to the target environment to obtain a preset decision model. The exemplary embodiment of the present disclosure can use the trained preset decision model to control the action strategy of the target object in the target environment, that is, in the process of gradually generating each video frame, the diffusion parameters are generated according to the latent space features of the current video frame to control the generation process of the next video frame. At the same time, the exemplary embodiment of the present disclosure also reversely updates the decision parameters of the preset decision model in the process of gradually generating video frames to gradually improve the decision accuracy of the preset decision model.

[0057] The decision parameter refers to the weight parameter of the preset decision model. The preset decision model can be represented by a neural network, and the decision parameter is the weight parameter in the neural network. By continuously adjusting the decision parameter, the preset decision model can gradually optimize its decision-making ability. The exemplary embodiments of the present disclosure do not specifically limit the network structure of the preset decision model.

[0058] In some optional embodiments, the diffusion parameters generated by the preset decision model include noise level parameters and / or conditional input information. The noise level parameters are used to control the amplitude of denoising, and the conditional input information is used to control the content of the generated video frame.

[0059] Among them, when generating each video frame, the diffusion model needs to gradually denoise the image containing noise to generate a clear image, add noise to the video frame based on the noise level, and perform the denoising process through the diffusion model to generate the next video frame. The adjustment of the noise level will directly affect the details of the denoising process and affect the quality and diversity of the generated video frame. The conditional input information is the information required in the scenario of generating conditional videos. That is, if the generation task itself has a clear control requirement for the generated content (for example, generating an image or video of a specific action or a specific posture), it is necessary to provide conditional input so that the generation process can be adjusted according to the specified conditions. For example, in the generated video, a certain person needs to walk in a certain gait. At this time, conditional input is needed to guide the model to generate content that meets the target. On the contrary, if the generation task is unconditional generation, that is, there is no specific requirement for the generated content, then this part of the parameters does not need to be adjusted, and the model will generate according to its internal generation rules. The exemplary embodiment of the present disclosure can set the diffusion parameters including noise level parameters and / or conditional input information according to the actual task requirements.

[0060] In step S230, based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, the latent spatial features of the current video frame are denoised by the diffusion model to generate the next video frame.

[0061] In an exemplary embodiment of the present disclosure, diffusion parameters can continue to be generated based on the prediction decision model after adjusting the decision parameters, that is, noise level parameters and / or conditional input information can be determined, and the diffusion parameters can be used as guidance information for generating the next video frame. The diffusion model is used to denoise the latent spatial features of the current video frame based on the guidance information to generate the next video frame.

[0062] The video generation method in the exemplary embodiment of the present disclosure, on the one hand, introduces a preset decision model of reinforcement learning in the process of gradually denoising the initial noise image based on the diffusion model, and uses the preset decision model to adjust the diffusion parameters of the diffusion model when generating video frames frame by frame, so as to achieve frame-by-frame optimization, thereby improving the quality of the generated video frames frame by frame. On the other hand, the decision parameters of the preset decision model are updated using the latent space features of the current video frame to gradually improve the accuracy of the generated diffusion parameters, thereby improving the decision accuracy of the preset decision model, thereby further improving the accuracy of denoising the latent space features of the current video frame, ensuring the quality and coherence between video frames, and improving the overall quality of the final generated video.

[0063] In an exemplary embodiment, a method for adjusting the decision parameters of a preset decision model is provided. Figure 3 As shown, adjusting the decision parameters of the preset decision model based on the potential spatial features of the current video frame may include:

[0064] Step S310: Determine current video reward information according to the latent spatial features of the current video frame.

[0065] The video reward information is used to evaluate the diffusion parameter generation effect of the preset decision model, where the current video reward information is used to evaluate the performance of the current diffusion parameters used to generate the current video frame. The current reward video information can guide the preset decision model to generate more realistic video frames.

[0066] The current video reward information may include one or more of an inter-frame coherence reward, a video quality reward, and a physical rationality reward. Of course, the current video reward information may also be adjusted to include other types of rewards, such as a style reward for reflecting whether the picture style is unified, etc., according to the actual needs of generating a video. Exemplary embodiments of the present disclosure include but are not limited to the above rewards.

[0067] In some optional embodiments, a method for determining the current video reward information is provided. Figure 4As shown, determining the current video reward information according to the potential spatial features of the current video frame may include:

[0068] Step S410: Determine a first reward function value according to a feature difference between a latent spatial feature of a current video frame and a latent spatial feature of an adjacent previous video frame.

[0069] The first reward function, also known as the inter-frame continuity reward function, can evaluate the similarity between the latent space features of adjacent frames. To ensure that the changes between adjacent frames are smooth, the difference between the latent space features between adjacent frames can be calculated, and the first reward function can be constructed based on the difference. The difference between the latent space features between adjacent frames can be determined by the Euclidean distance of the features. Of course, the feature differences between adjacent frames can also be calculated by other methods, which are not specifically limited.

[0070] For example, the difference between the latent space features between adjacent frames is expressed by the Euclidean distance of the features:

[0071] D t =||z t -z t-1 || (2)

[0072] Among them, D t represents the Euclidean distance between the latent spatial features of the current video frame (frame t) and the latent spatial features of the adjacent previous video frame, z t is the potential spatial feature of the current video frame, z t-1 is the potential spatial feature of the previous adjacent video frame.

[0073] Furthermore, to ensure inter-frame coherence, the goal of the preset decision model is to minimize the feature difference between two adjacent video frames, that is, to make the latent spatial features of two adjacent video frames as similar as possible to ensure that the generated content changes smoothly on the time axis. Based on this, the first reward function can be expressed as:

[0074] r t consistency =-||z t -z t-1 || (3)

[0075] Among them, r t consistency is the first reward function. When the difference between the latent spatial features of adjacent video frames is large, the value of the first reward function decreases, and the preset decision model is penalized. When the difference between the latent spatial features of adjacent video frames is small, the value of the first reward function increases, thereby encouraging the preset decision model to generate coherent video frames.

[0076] By minimizing the differences in latent space features between adjacent frames, the preset decision model can optimize the parameters in the generation process frame by frame to ensure that the changes in video content between frames are natural. For example, if in a video of a person walking, the person's legs are raised in the previous frame, the next frame should be a smooth leg step forward, rather than suddenly returning to the starting point or making other illogical movements. In addition, the process uses high-level features to measure the difference between frames, which is more efficient and accurate than directly comparing the differences between two frames in pixel space. This is because the latent space features contain a more abstract understanding of the scene, rather than simply comparing pixel values.

[0077] Step S420: Evaluate the picture quality of the current video frame using the latent space features of the current video frame, and determine the second reward function value according to the evaluation result.

[0078] The evaluation of picture quality is an evaluation of the visual quality of the current video frame. A pre-trained visual quality evaluation model can be used to evaluate the picture quality of the current video frame to determine the value of the second reward function according to the evaluation result. Exemplarily, the evaluation of the picture quality of the current video frame using the latent space feature of the current video frame can be expressed as:

[0079] r t quality =VQA(z t ) (4)

[0080] Among them, r t quality is the second reward function, VQA() is the visual quality assessment model, z t is the latent spatial feature of the current video frame.

[0081] The visual quality evaluation model may be a VGG (Visual Geometry Group Network) model or other network structures, and the exemplary embodiments of the present disclosure do not specifically limit this.

[0082] By evaluating the picture quality of the current video frame and determining the value of the second reward function, it can be ensured that the generated frame has a higher visual effect, such as visual clarity, texture retention, and detail richness, thereby ensuring that the entire video has a higher quality in detail expression.

[0083] Step S430: Using the latent space features of the current video frame, the physical rationality of the current video frame is evaluated, and the third reward function value is determined according to the evaluation result.

[0084] Physical rationality refers to whether the movement of objects in the generated video frames conforms to the laws of physics. For example, in a walking video, whether the generated objects conform to the physical laws of gravity, speed change, inertia, etc. In an exemplary embodiment of the present disclosure, the physical rationality of the current video frame can be evaluated by a preset physics engine (a physics engine is a computer software that calculates motion, rotation, collision reaction, etc. by assigning real physical properties to rigid objects) or a pre-trained differentiable physics network. Among them, the actual motion trajectory can be determined based on the latent spatial features of the current video frame, and the actual motion trajectory can be compared with the predicted motion trajectory to perform a physical rationality evaluation based on the comparison result, thereby determining the value of the third reward function. Exemplarily, the third reward function can be expressed by the following formula:

[0085] r t physics =-||actual trajectory-predicted trajectory|| (5)

[0086] Among them, r t physics is the third reward function.

[0087] By determining the value of the third reward function, the preset decision model can adjust the action trajectory to ensure that the objects in the generated video frames conform to the laws of physics and avoid unreasonable phenomena such as floating and sudden acceleration.

[0088] Step S440: Determine current video reward information based on the first reward function value, the second reward function value, and the third reward function value.

[0089] After determining the first reward function value, the second reward function value and the third reward function value, the reward function values ​​can be fused, such as weighted summation or direct summation, to determine the current video reward information of the current video frame. The exemplary embodiment of the present disclosure can select a method for fusing the reward function values ​​according to actual task requirements, and there is no specific limitation on this.

[0090] Step S320: adjusting the decision parameters of the preset decision model based on the current video reward information.

[0091] After the current video reward information is determined based on the first reward function value, the second reward function value, and the third reward function value, the decision parameters of the preset decision model may be adjusted based on the current video reward information.

[0092] Since the generated video needs to ensure the high quality of a single frame, it is also necessary to consider the filtering and smoothing between frames to avoid jumping and unnatural phenomena between frames. The exemplary embodiment of the present disclosure ensures the consistency of features between video frames by introducing a first reward function value. And the picture quality of the video frame is evaluated by introducing a second reward function value to ensure the high quality of each video frame. Furthermore, considering that the motion trajectory of the dynamic object in the current video generation method does not conform to the laws of physics, for example, the gait of the character may suddenly float, pause or incoherently accelerate, etc., which does not conform to gravity and inertia, the generated video frame is made to conform to the laws of physics by introducing a third reward function value, thereby ensuring that the generated video has physical rationality, inter-frame coherence and high visual effects as a whole, and improving the quality of the final generated video.

[0093] In an exemplary embodiment, if Figure 5 As shown, adjusting the decision parameters of the preset decision model based on the current video reward information may include:

[0094] Step S510: Obtain historical video reward information of historical video frames.

[0095] Step S520: Determine the expected cumulative reward based on the current video reward information and the historical video reward information.

[0096] The historical video frames refer to the video frames before the current video frame. For example, if the current video frame is the fifth frame, the historical video frames include the first to fourth video frames. The method of obtaining the historical video reward information of the historical video frames is the same as the method of obtaining the current video reward information of the current video frame, which will not be described in detail here. It should be understood that in the process of generating the first four video frames, the current video reward information of each video frame has been calculated, so this step can directly obtain the historical video reward information without repeated calculation.

[0097] The expected cumulative reward refers to the total amount of rewards that can be expected to be obtained after a preset decision model takes a certain action in each state during the reinforcement learning process. Specifically, the expected cumulative reward is obtained by calculating the expected value in each state, and these expected values ​​reflect the average value of the rewards that may be obtained after taking different actions in different states. In an exemplary embodiment of the present disclosure, the expected cumulative reward can be determined based on the current video reward information and the historical video reward information. It can be expressed by the following formula:

[0098]

[0099] Among them, J(θ) is the objective function of the expected cumulative reward, and E is the expected symbol, which means that the expected return of the current strategy (the current decision parameter corresponding to the current video frame) is calculated by taking the average of the rewards of all possible trajectories after multiple samplings (i.e., multiple runs in the environment) under the strategy πθ (i.e., the diffusion parameter in the process of generating each frame). The expected goal is to optimize the decision parameters of the preset decision model so that the expected cumulative reward is as large as possible, that is, the preset decision model can obtain as many rewards as possible in the long-term action process. It refers to the weighted sum of all video reward information from time t = 0 to t = T, r t is the video reward information obtained at frame t, including the first video reward value to the third video reward value. γ is the discount factor, ranging from 0 to 1, which represents the discount on future rewards and is used to balance the impact of immediate rewards and future rewards. The closer the discount factor γ is to 1, the higher the importance attached to future rewards. The closer the discount factor γ is to 0, the more attention is paid to current rewards. θ is the decision parameter of the preset decision model, π θ Represents the policy function, which is a probability distribution that defines the t (that is, the latent space feature z t ) Select action a t (diffusion parameter) probability.

[0100] Exemplarily, if the current video frame is the 5th video frame, then That is, the historical video reward information of each frame before the 5th video frame (including the 5th video frame) is multiplied by the corresponding discount factor to reflect the cumulative benefits of these video frames.

[0101] Step S530: adjusting the decision parameters of the preset decision model to maximize the expected cumulative reward.

[0102] The preset decision model optimizes the decision parameters with the goal of maximizing the expected cumulative reward, so that the cumulative reward obtained in the entire video generation process is as high as possible.

[0103] In the process of generating video frames frame by frame, the exemplary embodiment of the present disclosure updates the decision parameters of the preset decision model based on the expected cumulative reward corresponding to the current video frame, which can improve the accuracy of the decision-making of the preset decision model, and then improve the accuracy of the diffusion parameters used to generate the next video frame, realize adaptive adjustment of the model, and improve the overall video generation quality.

[0104] In an exemplary embodiment, if Figure 6 As shown, the decision parameters of the preset decision model are adjusted to maximize the expected cumulative reward, including:

[0105] Step S610: performing advantage estimation according to the current video reward information and the historical video reward information, and determining an advantage estimation result, where the advantage estimation result is used to reflect the advantage of the current diffusion parameter over the historical diffusion parameter.

[0106] Advantage estimation in reinforcement learning refers to optimizing strategies by estimating the advantage of an action over other actions. The exemplary embodiment of the present disclosure estimates the advantage of the current diffusion parameter (i.e., an action) over the historical diffusion parameter based on the current video reward information and the historical video reward information. For example, the advantage of the current diffusion parameter over the average of the historical diffusion parameters can be estimated. The core of advantage estimation is to calculate the advantage value of an action, that is, the value of an action minus the value of the state, which can be expressed by the following formula:

[0107] A(s,a) = Q(s,a)-V(s) (7)

[0108] Wherein, Q(s,a) represents the expected return of taking action (a) in state (s), and V(s) represents the expected return in state (s). The advantage function represents the additional benefit of the action relative to the state value. The method of advantage estimation in the exemplary embodiment of the present disclosure is no different from the traditional calculation method, which will not be described in detail.

[0109] Step S620: obtaining the current decision parameter corresponding to the current video frame, and obtaining the historical diffusion parameter and the historical decision parameter corresponding to the adjacent previous video frame.

[0110] The current decision parameters corresponding to the current video frame are also the decision parameters to be optimized. The historical decision parameters corresponding to the adjacent previous video frame refer to the strategy before the most recent strategy update. The historical diffusion parameters corresponding to the adjacent previous video frame refer to the diffusion parameters used when generating the previous video frame.

[0111] Step S630: Determine the proximal strategy optimization loss function based on the advantage estimation result, the current diffusion parameter, the current decision parameter, the historical diffusion parameter and the historical decision parameter corresponding to the previous video frame.

[0112] The proximal policy optimization algorithm improves the stability of reinforcement learning by keeping the magnitude of policy updates within a controllable range, that is, by comparing the probability of executing actions under the new and old policies and using these ratios to adjust the magnitude of policy updates. Specifically, the proximal policy optimization loss function is determined based on the advantage estimation results, the current diffusion parameters, the current decision parameters, the historical diffusion parameters and the historical decision parameters corresponding to the previous video frame, and can be expressed by the following formula:

[0113]

[0114] Among them, π θold is the strategy at the time of the last update (old strategy), that is, the strategy to be updated when the strategy was last updated (involving the historical decision parameters corresponding to the previous video frame), πθ represents the probability ratio of the new strategy (the current decision parameters to be updated) and the old strategy in selecting actions. If the new strategy is more inclined to select the same action than the old strategy, the ratio is greater than 1, otherwise it is less than 1. Represents advantage estimation information, if If it is greater than 0, it means that the current action takes precedence over the average level and the strategy is more inclined to choose the current action. If it is less than 0, the opposite is true. clip() is a clipping function, which is used to limit the amplitude of the policy update to prevent the policy from updating too quickly and affecting the stability of the model. The clipping function limits the above ratio to range, ensuring that the gap between the new strategy and the old strategy is not too large. min(·,·) means taking the minimum of the two values. The purpose is to actively give up some possible large gains when the gain exceeds a certain range to ensure the stability of the update.

[0115] The exemplary embodiments of the present disclosure introduce considerations of progressiveness and stability, and use a clipping function to enable the preset decision model to learn smoothly, thereby avoiding the impact of a certain update on the existing strategy.

[0116] Step S640: adjusting the decision parameters of the preset decision model according to the proximal strategy optimization loss function.

[0117] An exemplary embodiment of the present disclosure can adjust the decision parameters of a preset decision model according to a proximal strategy optimization loss function to maximize the expected cumulative reward, thereby obtaining a preset decision model after adjusting the decision parameters, so that the preset decision model gradually learns to optimize the decisions in the diffusion model process, and improves the accuracy of the diffusion parameters used in the process of generating video frames frame by frame.

[0118] In an exemplary embodiment, if the diffusion parameters generated by the preset decision model include noise level parameters and conditional input information, based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, denoising the latent spatial features of the current video frame through the diffusion model, generating the next video frame may include:

[0119] First, random noise is added to the latent spatial features of the current video frame based on the noise level parameter in the diffusion parameter, and then the latent spatial features to which the random noise is added are denoised according to the conditional input information to generate the next video frame.

[0120] The task of the preset decision model in the generation process of each video frame is to dynamically adjust the noise level parameters and conditional input information in the denoising process of the diffusion model based on the latent spatial features of the current video frame to optimize the generation results frame by frame. Among them, the preset decision model controls the noise level information in each diffusion step can be expressed by the following formula:

[0121]

[0122] Among them, α t is the learning rate, which is a preset parameter. It indicates the speed of change from a noisy image to a clearer image in each step and determines the step size of denoising in each step. The noise level information is dynamically adjusted by the preset decision model according to the potential spatial characteristics of the current video frame. If the preset decision model determines that the image details of the current video frame are insufficient or lack diversity, the noise level can be increased. On the contrary, if the preset decision model determines that the image of the current video frame is blurred or jumpy, the level can be reduced to make the image clearer and smoother. is a differential operator used to represent the gradient. t is the potential spatial feature of the current video frame (i.e. z t ), x t is the current video frame, x t+1 The next video frame.

[0123] If the model requires conditional generation, the preset decision model can adjust the conditional input information to affect the context of the generated video through the following formula:

[0124] x t+1 =Decoder(z t+1 ,c t ) (10)

[0125] Among them, x t+1 is the next video frame, Decoder() is the encoder of the diffusion model, z t+1 is the potential spatial feature of the current video frame, c t Enter information for the condition.

[0126] The exemplary embodiment of the present disclosure controls the diffusion model to denoise the latent spatial features of the current video frame by using the diffusion parameters generated by the preset decision model after updating the decision parameters, so as to gradually optimize the quality of generating each video frame and improve the overall quality of the final generated video.

[0127] In an exemplary embodiment, the diffusion model and the preset decision model are obtained by pre-training. The video generation model can be trained by combining a hybrid training strategy of supervised learning and reinforcement learning, and the video generation model can include the diffusion model and the preset decision model.

[0128] Among them, the diffusion model can be pre-trained using labeled data (such as videos containing people walking) to enable the diffusion model to have basic generation capabilities. Then, based on the pre-trained model, reinforcement learning is used to fine-tune the preset decision model. By adjusting the preset decision model, the realism and coherence of the video generation can be further optimized.

[0129] Specifically, video frame samples are collected from sample videos, each of which represents a person in different positions and gaits. Then, a diffusion model is used to gradually denoise the video frame starting from Gaussian noise to obtain clear video frames. By maximizing the similarity between the generated frames and the real video frames, the diffusion model learns to generate continuous and realistic walking videos of people from random noise. Furthermore, a preset decision model is introduced. The diffusion parameters in the frame-by-frame denoising process of the diffusion model are adjusted by training the decision model to control the generation of the next video frame. The preset decision model updates its decision parameters according to the latent spatial features and video reward information of the next video frame, and the above process is executed repeatedly. Through multiple training and strategy updates, the preset decision model gradually learns how to adjust the diffusion parameters in the generation process to generate more coherent and physically compliant video frames.

[0130] In addition, you can also use multiple GPUs (Graphics Processing Units) and distributed computing frameworks to accelerate the training process, and use model pruning, quantization and other technologies to reduce computing overhead and improve real-time generation efficiency. You can flexibly choose according to actual task requirements, and there are no specific restrictions on this.

[0131] The video generation method of the exemplary embodiment of the present disclosure is described below with specific examples.

[0132] The process of generating the first video frame is as follows:

[0133] From random noise x T At the beginning, the diffusion model is used to perform step-by-step denoising to obtain the first video frame x 1 , using the diffusion model encoder to extract the first video frame x 1 The latent space features z 1 , the latent space feature z 1 Represents high-level information of the first video frame.

[0134] The process of generating the second video frame is as follows:

[0135] The preset decision model is based on the latent spatial feature z of the first video frame 1 , through the policy function, select action a 2 (Noise level parameter 2, conditional input information 2). Thus, the diffusion model is based on action a 2The latent spatial feature z of the first video frame 1 Denoising is performed to obtain a second video frame, and latent spatial features of the second video frame are extracted through an encoder of a diffusion model.

[0136] The process of this round of strategy update is as follows:

[0137] After generating the second video frame, a first reward function value is determined according to the feature difference between the latent space features of the second video frame and the latent space features of the first video frame; the picture quality of the current video frame is evaluated using the latent space features of the second video frame, and a second reward function value is determined according to the evaluation result; the physical rationality of the current video frame is evaluated using the latent space features of the second video frame, and a third reward function value is determined according to the evaluation result; finally, the second video reward information is determined based on the first reward function value, the second reward function value and the third reward function value, and the advantage estimation result corresponding to the second video frame is calculated, so as to obtain the proximal strategy optimization loss function, so as to update the decision parameters of the preset decision model based on the loss function, and the updated decision parameters will affect the decision to generate the third video frame.

[0138] The updated decision parameters are used to generate the third video frame as follows:

[0139] Select action a based on the preset decision model that updates the decision parameters 3 =(noise level parameter 3, conditional input information 3), and then the diffusion model is based on action a 3 The latent spatial features of the second video frame are denoised to obtain a third video frame.

[0140] The above process is repeated until the complete video is generated. It should be noted that the specific details involved have been recorded in the above exemplary embodiments and will not be repeated here.

[0141] The video generation method in the exemplary embodiment of the present disclosure obtains the latent spatial features of the current video frame, and adjusts the decision parameters of the preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; based on the diffusion parameters generated by the preset decision model after the decision parameters are updated, the latent spatial features of the current video frame are denoised by the diffusion model to generate the next video frame. On the one hand, in the process of gradually denoising the initial noise image based on the diffusion model, the preset decision model of reinforcement learning is introduced, and when the video frames are generated frame by frame, the diffusion parameters of the diffusion model are adjusted by the preset decision model to achieve frame-by-frame optimization, thereby improving the quality of the generated video frames frame by frame. On the other hand, the decision parameters of the preset decision model are updated by using the latent spatial features of the current video frame to gradually improve the accuracy of the generated diffusion parameters, thereby improving the decision accuracy of the preset decision model, thereby further improving the accuracy of the denoising process of the latent spatial features of the current video frame, ensuring the quality and coherence between video frames, and improving the overall quality of the final generated video. In addition, through hybrid training strategies and distributed computing, the model training efficiency and model performance can be further improved.

[0142] In an exemplary embodiment of the present disclosure, a video generating device is also provided. Figure 7 As shown, the video generation device 700 may include a feature acquisition module 710, a parameter adjustment module 720 and an image generation module 730. Specifically:

[0143] The feature acquisition module 710 is used to obtain the latent spatial features of the current video frame, and the current video frame is obtained by denoising the initial noise image through a diffusion model; the parameter adjustment module 720 is used to adjust the decision parameters of the preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; the image generation module 730 is used to denoise the latent spatial features of the current video frame through a diffusion model based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, so as to generate the next video frame.

[0144] In an exemplary embodiment of the present disclosure, the diffusion parameters generated by the preset decision model include noise level parameters and / or conditional input information; the noise level parameters are used to control the amplitude of denoising, and the conditional input information is used to control the content of the generated video frame.

[0145] In an exemplary embodiment of the present disclosure, the parameter adjustment module 720 is configured to perform: determining current video reward information based on the latent space features of the current video frame; wherein the current video reward information is used to evaluate the performance of the current diffusion parameters used to generate the current video frame; and adjusting the decision parameters of the preset decision model based on the current video reward information.

[0146] In an exemplary embodiment of the present disclosure, the parameter adjustment module 720 is configured to perform: determining a first reward function value based on a feature difference between a latent space feature of a current video frame and a latent space feature of an adjacent previous video frame; evaluating the picture quality of the current video frame using the latent space feature of the current video frame, and determining a second reward function value based on the evaluation result; performing a physical rationality evaluation on the current video frame using the latent space feature of the current video frame, and determining a third reward function value based on the evaluation result; and determining current video reward information based on the first reward function value, the second reward function value, and the third reward function value.

[0147] In an exemplary embodiment of the present disclosure, the parameter adjustment module 720 is configured to perform: obtaining historical video reward information of historical video frames; determining the expected cumulative reward based on the current video reward information and the historical video reward information; and adjusting the decision parameters of the preset decision model to maximize the expected cumulative reward.

[0148] In an exemplary embodiment of the present disclosure, the parameter adjustment module 720 is configured to perform: performing advantage estimation based on current video reward information and historical video reward information, and determining the advantage estimation result, wherein the advantage estimation result is used to reflect the advantage of the current diffusion parameter over the historical diffusion parameter; obtaining the current decision parameter corresponding to the current video frame, and obtaining the historical diffusion parameter and the historical decision parameter corresponding to the adjacent previous video frame; determining the proximal strategy optimization loss function based on the advantage estimation result, the current diffusion parameter, the current decision parameter, the historical diffusion parameter and the historical decision parameter corresponding to the previous video frame; and adjusting the decision parameters of the preset decision model according to the proximal strategy optimization loss function.

[0149] In an exemplary embodiment of the present disclosure, if the diffusion parameters generated by the preset decision model include noise level parameters and conditional input information, the image generation module 730 is configured to execute: adding random noise to the latent spatial features of the current video frame based on the noise level parameters in the diffusion parameters; and denoising the latent spatial features to which the random noise is added according to the conditional input information to generate the next video frame.

[0150] In an exemplary embodiment of the present disclosure, the feature acquisition module 710 is further configured to execute: acquiring initial random noise, and determining an initial noise image according to the initial random noise.

[0151] Since the details of the functional modules of the video generating device of the exemplary embodiment of the present disclosure have been described in the exemplary embodiment of the video generating method described above, they will not be repeated here.

[0152] It should be noted that although several modules or units of the video generation device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0153] The exemplary embodiments of the present disclosure further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0154] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0155] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0156] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0157] The computer program can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. The electronic device can convert the signal carrying the computer program into a digital signal, and then run the computer program. When the computer program runs on the electronic device, its code is used to make the electronic device execute (more specifically, the processor of the electronic device can execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned video generation method, which includes the following steps: obtaining the latent spatial features of the current video frame, the current video frame is obtained by denoising the initial noise image through a diffusion model; adjusting the decision parameters of the preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, the latent spatial features of the current video frame are denoised through the diffusion model to generate the next video frame. 。

[0158] The above method steps are executed by a computer program. On the one hand, in the process of gradually denoising the initial noise image based on the diffusion model, a preset decision model of reinforcement learning is introduced. When generating video frames frame by frame, the diffusion parameters of the diffusion model are adjusted using the preset decision model to achieve frame-by-frame optimization, thereby improving the quality of the generated video frames frame by frame. On the other hand, the decision parameters of the preset decision model are updated using the latent space features of the current video frame to gradually improve the accuracy of the generated diffusion parameters, thereby improving the decision accuracy of the preset decision model, thereby further improving the accuracy of denoising the latent space features of the current video frame, ensuring the quality and coherence between video frames, and improving the overall quality of the final generated video.

[0159] In an exemplary embodiment of the present disclosure, the diffusion parameters generated by the preset decision model include noise level parameters and / or conditional input information; the noise level parameters are used to control the amplitude of denoising, and the conditional input information is used to control the content of the generated video frame.

[0160] In an exemplary embodiment of the present disclosure, decision parameters of a preset decision model are adjusted based on the latent space features of a current video frame, including: determining current video reward information based on the latent space features of the current video frame; wherein the current video reward information is used to evaluate the performance of current diffusion parameters used to generate the current video frame; and adjusting the decision parameters of the preset decision model based on the current video reward information.

[0161] In an exemplary embodiment of the present disclosure, current video reward information is determined based on latent space features of the current video frame, including: determining a first reward function value based on feature differences between the latent space features of the current video frame and the latent space features of an adjacent previous video frame; evaluating the picture quality of the current video frame using the latent space features of the current video frame, and determining a second reward function value based on the evaluation result; performing a physical rationality evaluation on the current video frame using the latent space features of the current video frame, and determining a third reward function value based on the evaluation result; and determining the current video reward information based on the first reward function value, the second reward function value, and the third reward function value.

[0162] In an exemplary embodiment of the present disclosure, decision parameters of a preset decision model are adjusted based on current video reward information, including: obtaining historical video reward information of historical video frames; determining an expected cumulative reward based on the current video reward information and the historical video reward information; and adjusting the decision parameters of the preset decision model to maximize the expected cumulative reward.

[0163] In an exemplary embodiment of the present disclosure, the decision parameters of a preset decision model are adjusted to maximize the expected cumulative reward, including: performing advantage estimation based on current video reward information and historical video reward information, and determining an advantage estimation result, wherein the advantage estimation result is used to reflect the advantage of current diffusion parameters over historical diffusion parameters; obtaining current decision parameters corresponding to a current video frame, and obtaining historical diffusion parameters and historical decision parameters corresponding to an adjacent previous video frame; determining a proximal strategy optimization loss function based on the advantage estimation result, current diffusion parameters, current decision parameters, historical diffusion parameters and historical decision parameters corresponding to a previous video frame; and adjusting the decision parameters of the preset decision model according to the proximal strategy optimization loss function.

[0164] In an exemplary embodiment of the present disclosure, if the diffusion parameters generated by the preset decision model include noise level parameters and conditional input information, based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, the latent spatial features of the current video frame are denoised through the diffusion model to generate the next video frame, including: adding random noise to the latent spatial features of the current video frame based on the noise level parameters in the diffusion parameters; and denoising the latent spatial features to which the random noise is added according to the conditional input information to generate the next video frame.

[0165] In an exemplary embodiment of the present disclosure, before obtaining the latent spatial features of the current video frame, the method further includes: obtaining initial random noise, and determining an initial noise image according to the initial random noise.

[0166] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module", or "system".

[0167] Refer to the following Figure 8 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0168] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0169] The storage unit stores a program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can execute the following steps: obtaining the latent spatial features of the current video frame, the current video frame is obtained by denoising the initial noise image through a diffusion model; adjusting the decision parameters of the preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control the denoising process of the diffusion model; based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, denoising the latent spatial features of the current video frame through the diffusion model to generate the next video frame.

[0170] In an exemplary embodiment of the present disclosure, the diffusion parameters generated by the preset decision model include noise level parameters and / or conditional input information; the noise level parameters are used to control the amplitude of denoising, and the conditional input information is used to control the content of the generated video frame.

[0171] In an exemplary embodiment of the present disclosure, decision parameters of a preset decision model are adjusted based on the latent space features of a current video frame, including: determining current video reward information based on the latent space features of the current video frame; wherein the current video reward information is used to evaluate the performance of current diffusion parameters used to generate the current video frame; and adjusting the decision parameters of the preset decision model based on the current video reward information.

[0172] In an exemplary embodiment of the present disclosure, current video reward information is determined based on latent space features of the current video frame, including: determining a first reward function value based on feature differences between the latent space features of the current video frame and the latent space features of an adjacent previous video frame; evaluating the picture quality of the current video frame using the latent space features of the current video frame, and determining a second reward function value based on the evaluation result; performing a physical rationality evaluation on the current video frame using the latent space features of the current video frame, and determining a third reward function value based on the evaluation result; and determining the current video reward information based on the first reward function value, the second reward function value, and the third reward function value.

[0173] In an exemplary embodiment of the present disclosure, decision parameters of a preset decision model are adjusted based on current video reward information, including: obtaining historical video reward information of historical video frames; determining an expected cumulative reward based on the current video reward information and the historical video reward information; and adjusting the decision parameters of the preset decision model to maximize the expected cumulative reward.

[0174] In an exemplary embodiment of the present disclosure, the decision parameters of a preset decision model are adjusted to maximize the expected cumulative reward, including: performing advantage estimation based on current video reward information and historical video reward information, and determining an advantage estimation result, wherein the advantage estimation result is used to reflect the advantage of current diffusion parameters over historical diffusion parameters; obtaining current decision parameters corresponding to a current video frame, and obtaining historical diffusion parameters and historical decision parameters corresponding to an adjacent previous video frame; determining a proximal strategy optimization loss function based on the advantage estimation result, current diffusion parameters, current decision parameters, historical diffusion parameters and historical decision parameters corresponding to a previous video frame; and adjusting the decision parameters of the preset decision model according to the proximal strategy optimization loss function.

[0175] In an exemplary embodiment of the present disclosure, if the diffusion parameters generated by the preset decision model include noise level parameters and conditional input information, based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, the latent spatial features of the current video frame are denoised through the diffusion model to generate the next video frame, including: adding random noise to the latent spatial features of the current video frame based on the noise level parameters in the diffusion parameters; and denoising the latent spatial features to which the random noise is added according to the conditional input information to generate the next video frame.

[0176] In an exemplary embodiment of the present disclosure, before obtaining the latent spatial features of the current video frame, the method further includes: obtaining initial random noise, and determining an initial noise image according to the initial random noise.

[0177] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 821 and / or a cache memory unit 822 , and may further include a read-only memory unit (ROM) 823 .

[0178] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, such program modules 825 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0179] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0180] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0181] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0182] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0183] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A video generation method, characterized in that: include: Acquire latent spatial features of a current video frame, wherein the current video frame is obtained by denoising an initial noisy image using a diffusion model; Adjusting decision parameters of a preset decision model based on the latent spatial features of the current video frame, wherein the preset decision model is used to generate diffusion parameters, and the diffusion parameters are used to control a denoising process of the diffusion model; Based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters, the potential spatial features of the current video frame are denoised through the diffusion model to generate the next video frame.

2. The method according to claim 1, characterized in that The diffusion parameters generated by the preset decision model include noise level parameters and / or conditional input information; The noise level parameter is used to control the amplitude of denoising, and the conditional input information is used to control the content of the generated video frame.

3. The method according to claim 1 or 2, characterized in that: The adjusting the decision parameters of the preset decision model based on the potential spatial features of the current video frame includes: Determining current video reward information according to the latent spatial features of the current video frame; wherein the video reward information is used to evaluate the diffusion parameter generation effect of the preset decision model, and the current video reward information is used to evaluate the performance of the current diffusion parameters used to generate the current video frame; The decision parameters of the preset decision model are adjusted based on the current video reward information.

4. The method according to claim 3, characterized in that The determining the current video reward information according to the potential spatial features of the current video frame includes: Determining a first reward function value according to a feature difference between a latent spatial feature of the current video frame and a latent spatial feature of an adjacent previous video frame; Using the latent spatial features of the current video frame to evaluate the picture quality of the current video frame, and determining a second reward function value according to the evaluation result; Performing a physical rationality evaluation on the current video frame using the latent spatial features of the current video frame, and determining a third reward function value according to the evaluation result; The current video reward information is determined based on the first reward function value, the second reward function value, and the third reward function value.

5. The method according to claim 3, characterized in that: The adjusting the decision parameters of the preset decision model based on the current video reward information includes: Get historical video reward information of historical video frames; Determine an expected cumulative reward based on the current video reward information and the historical video reward information; The decision parameters of the preset decision model are adjusted to maximize the expected cumulative reward.

6. The method according to claim 5, characterized in that The step of adjusting the decision parameters of the preset decision model so as to maximize the expected cumulative reward includes: Performing advantage estimation according to the current video reward information and the historical video reward information, and determining an advantage estimation result, wherein the advantage estimation result is used to reflect the advantage of the current diffusion parameter over the historical diffusion parameter; Obtaining a current decision parameter corresponding to the current video frame, and obtaining a historical diffusion parameter and a historical decision parameter corresponding to an adjacent previous video frame; Determine a proximal strategy optimization loss function based on the advantage estimation result, the current diffusion parameter, the current decision parameter, the historical diffusion parameter and the historical decision parameter corresponding to the previous video frame; The decision parameters of the preset decision model are adjusted according to the proximal strategy optimization loss function.

7. The method according to claim 2, characterized in that If the diffusion parameters generated by the preset decision model include noise level parameters and conditional input information; the diffusion parameters generated by the preset decision model after adjusting the decision parameters, denoising the latent spatial features of the current video frame through the diffusion model to generate the next video frame, including: adding random noise to the latent spatial features of the current video frame based on a noise level parameter in the diffusion parameter; According to the conditional input information, denoising is performed on the latent spatial features to which random noise is added to generate the next video frame.

8. The method according to claim 1, characterized in that Before obtaining the potential spatial features of the current video frame, the method further includes: Initial random noise is acquired, and the initial noise image is determined according to the initial random noise.

9. A video generating device, characterized in that: The device comprises: A feature acquisition module, used to acquire latent spatial features of a current video frame, wherein the current video frame is obtained by denoising an initial noisy image using a diffusion model; A parameter adjustment module, configured to adjust a decision parameter of a preset decision model based on the potential spatial features of the current video frame, wherein the preset decision model is used to generate a diffusion parameter, and the diffusion parameter is used to control a denoising process of the diffusion model; The image generation module is used to generate a next video frame by denoising the potential spatial features of the current video frame based on the diffusion parameters generated by the preset decision model after adjusting the decision parameters.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

Images