Adapting video generation models using generative model feedback

Abstract

Systems and methods for improving video generation models are provided. In some implementations, a video evaluation input prompt is assembled, including data indicative of a synthetic digital video and a request to evaluate the video based on one or more criteria. The synthetic digital video comprises a temporal sequence of frames generated by a video generation model and portrays simulated events. The video evaluation input prompt is processed using a generative model, such as a vision-language model, to generate evaluation output indicative of an outcome of the evaluation. Based on the evaluation output, the video generation model is adapted. The adaptation can be performed using reinforcement learning with the evaluation output as reward data or using direct preference optimization with the evaluation output as preference data. This process can improve the physical plausibility and logical consistency of videos subsequently generated by the adapted model.

Description

ADAPTING VIDEO GENERATION MODELS USING GENERATIVE MODEL FEEDBACKATTORNEY REFERENCE: GOOG-0790-WO-01 Background

[0001] This specification relates to processing data using machine learning models.

[0002] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0003] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

[0004] Large text-to-video models trained on internet-scale data offer tremendous potential, from automating movie creation and producing personalized educational content to enhancing immersive experiences in virtual environments. However, these models, and synthetic content generation models more generally, still face significant challenges in delivering high-quality synthetic video. Common issues include out-of-place objects, violations of real -world physics, and implausible movements. These shortcomings have thus far limited the applicability of video generation models for various downstream tasks, such as robot control, interacting with human users, etc. While increasing dataset and model size can improve text-to-video models, creating supervised text-video pairs by manually captioning vast amounts of video data is labor- intensive.

[0005] For example, generated video samples can contain movements that violate physical laws and include out-of-place objects, which may be due to insufficient alignment between the text prompt and the generated video content. Certain categories of object movements have been observed to be particularly challenging for these models. Such categories can include object removal, where an object is moved out of a container or the camera frame, which can introduce out-of-place objects during the scene transition (e.g., taking a pen out of a box or uncovering a key). Another challenging category involves interactions between multiple objects, where maintaining content consistency in a dynamic scene is difficult (e.g., taking one body spray from a group of similar sprays, or moving an interlocking building brick away from a computer mouse). Manipulating deformable objects, such as cloth or paper, also presents a challenge, asgenerating realistic movement for non-rigid items tests the model's text-content alignment (e.g., folding a rag or twisting a wet shirt until water comes out). A further challenging category includes directional movement, where an object is moved in a specified direction. While models may follow high-level directions, they can fail to generate consistent objects throughout the scene (e.g., pulling a water bottle from left to right, or moving a banana away from the camera). Yet another challenging category relates to falling down. In addition to the dynamic movement, this falling down often requires an expression towards the depth dimension. For example, putting marker pen onto plastic bottle so it falls off the table, or poking an apple so that it falls over.Summary

[0006] To improve text-to-video models, external feedback may be used to align generated outputs with desired outcomes. However, existing approaches for incorporating feedback have several limitations. One approach involves using metric-based feedback, where quantitative scores related to semantics, human preference, or dynamics are used as rewards. For example, metrics like CLIP score can evaluate text-video alignment, while other metrics can estimate human preference or quantify motion using optical flow. While these metrics can be calculated automatically and used for fine-tuning, improvements in these metrics do not always correlate with human perceptions of video quality or physical plausibility. Optimizing for these metrics can lead to undesirable artifacts or visual degradation, even as the scores themselves improve. This indicates that such metric-based rewards may not serve as reliable proxies for realistic object dynamics and overall video quality.

[0007] Another approach involves leveraging human feedback. While direct human evaluation is a reliable way to assess video quality, collecting this type of feedback is costly and not scalable for large-scale model training. The process of gathering, labeling, and integrating human preference data is resource-intensive and can be a significant bottleneck in the model development cycle.

[0008] Furthermore, existing fine-tuning algorithms present their own challenges. Some offline optimization algorithms can struggle with generalization to new, unseen prompts, exhibiting overfitting to the training data. Other preference-based optimization algorithms can suffer from over-optimization when used with metric-based rewards, leading to a decline in perceived video quality. These issues highlight the unmet need for more effective and scalable methods toevaluate and integrate feedback for improving the physical realism and prompt-adherence of generated videos.

[0009] Disclosed herein are systems and methods for improving video generation models using feedback. The described approaches leverage a generative model, such as a vision-language model, to evaluate synthetic digital videos and provide feedback that is then used to adapt the video generation model. This process enhances the quality of subsequently generated videos, particularly with respect to physical plausibility, logical consistency of events, and adherence to initial generation requests. The disclosed techniques provide a scalable and effective mechanism for refining video generation models to produce more realistic and coherent video content.

[0010] In one aspect, a method involves assembling a video evaluation input prompt. This prompt includes data representing a synthetic digital video, which was generated by a video generation model and portrays one or more simulated events, along with a request to evaluate the video based on one or more criteria. The method continues by processing this input prompt with a generative model to produce evaluation output. This output indicates the outcome of the evaluation. Based on this evaluation output, the video generation model is adapted. For example, the adaptation can be performed using reinforcement learning techniques where the evaluation output serves as reward data, or through direct preference optimization where the output serves as preference data.

[0011] In another aspect, a method is provided for utilizing a previously adapted video generation model. The method includes assembling an input prompt with data indicative of conditioning digital images and a request to generate a synthetic digital video portraying certain events. This prompt is processed using the video generation model to produce the requested synthetic video. The video generation model used in this process has been previously adapted through a feedback loop, wherein a training video was evaluated by a generative model against one or more criteria, and the resulting evaluation output was used to adapt the video generation model.

[0012] In a further aspect, a system is disclosed, comprising one or more processors and memory. The memory stores instructions that, when executed, cause the system to perform the methods described. This includes generating a first synthetic video, obtaining an evaluation of this video from a generative model based on criteria such as physical plausibility, adapting the video generation model based on the evaluation (for instance, via reinforcement learning usingthe evaluation as reward data), and then generating a second synthetic video with the adapted model that exhibits improved adherence to the evaluation criteria.

[0013] Implementations disclosed herein can mitigate (e.g., eliminate) various drawbacks with current techniques. For example, by leveraging a generative model, such as a vision-language model, to provide feedback on synthetic videos, the described approaches can address the limitations of metric-based feedback, where improvements in quantitative scores may not correlate with human perceptions of video quality or physical plausibility. As another example, using a generative model for evaluation offers a scalable and automated alternative to collecting direct human feedback, which can be costly and resource-intensive, thereby overcoming bottlenecks in the model development cycle. As another example, the use of adaptation techniques such as reinforcement learning and direct preference optimization based on this generative model feedback can address challenges with existing fine-tuning algorithms, such as overfitting to training data or over-optimization on metric-based rewards that can lead to a decline in perceived video quality.

[0014] As one non-limiting example, the techniques described herein can be used to improve synthetic video generation for robotic planning and control. Consider a scenario where a robot is tasked with assembling a complex mechanical part, such as a gearbox, in a manufacturing setting. This task involves a sequence of precise actions: identifying and picking up specific gears and bearings from a cluttered bin, orienting them correctly, and inserting them into a housing in a specific order. Generating physically plausible synthetic videos of these actions can be used to train a robot control policy or to plan a sequence of operations.

[0015] Initially, a video generation model, such as a denoising diffusion probabilistic model (DDPM), might be prompted with a request like "a robot arm picks a spur gear from a bin and inserts it onto a shaft inside a gearbox housing." A video generated by an unadapted model might portray physically implausible events. For example, the robot's gripper could appear to pass through the solid housing of the gearbox instead of moving around it, or the gear might suddenly change its orientation in an impossible way just before insertion. Such inaccuracies make the synthetic video unsuitable for generating reliable robot control data.

[0016] To improve the video generation model, the generated synthetic digital video is provided as input to a generative model, for instance, a vision-language model (VLM). A video evaluation input prompt is assembled, containing the synthetic video and a request to evaluate it based on one or more criteria. These criteria could include physical plausibility (e.g., "Do all movementsadhere to the laws of physics, such as collision avoidance and gravity?"), task completion (e.g., "Was the gear successfully inserted onto the shaft?"), and logical sequencing of actions (e.g., "Did the robot pick the gear before attempting to insert it?").

[0017] The VLM processes the video evaluation input prompt and generates an evaluation output. In one implementation, this output could be a binary "accept" or "reject" token. A video depicting the robot gripper passing through the gearbox housing would receive a "reject" output. This evaluation output is then used as feedback to adapt the video generation model. For example, using a reinforcement learning approach, the "reject" output can function as a negative reward signal (or lack of a positive reward) to update the parameters of the video generation model. Through an adaptation process such as direct preference optimization (DPO), pairs of videos (e.g., a more plausible one and a less plausible one) can be used with the VLM's evaluations as preference data to refine the model.

[0018] After one or more iterations of this adaptation process, the video generation model becomes more proficient at generating physically realistic videos. When prompted again with "a robot arm picks a spur gear from a bin and inserts it onto a shaft inside a gearbox housing," the adapted model generates a new synthetic video. This new video is more likely to portray the robot arm correctly maneuvering around the housing and the gear maintaining a consistent, physically correct orientation during the insertion action.

[0019] This higher-fidelity synthetic video can then be used for downstream robotics tasks. For example, the temporal sequence of frames in the new video can be processed to extract a trajectory for the robot's end-effector. This trajectory data can be converted into robot control data, such as a series of joint angle commands, which are then transmitted to a real or simulated robot. By operating the robot based on this control data, the robot can execute the gearbox assembly task more effectively, as the underlying plan is based on a physically plausible and coherent simulation of the required events. This process illustrates how adapting video generation models with feedback from a generative model can lead to the creation of more useful synthetic data for complex real-world applications like robotic planning and control.

[0020] In various implementations, a method may be performed using one or more processors. Such a method may include assembling, as a video evaluation input prompt, data indicative of a synthetic digital video and a request to evaluate the synthetic digital video based on one or more criteria. The synthetic digital video may include a temporal sequence of synthetic digital video frames, may be generated using a video generation model, and may portray one or more eventsbeing simulated. The method may also include processing the video evaluation input prompt using a generative model to generate evaluation output indicative of an outcome of evaluating the synthetic digital video based on the one or more criteria. Further, the method may include adapting the video generation model based on the evaluation output.

[0021] In some implementations, the request to evaluate the synthetic digital video may include a request to generate binary output, and the evaluation output may include one or more tokens expressing binary output. The one or more criteria may include adherence of the one or more events portrayed in the synthetic digital video with one or more laws of physics. The criteria may also include adherence of the one or more events portrayed in the synthetic digital video with a request to generate the synthetic digital video to portray the one or more events. This request may be obtained from a video generation prompt used to condition the video generation model to generate the synthetic digital video.

[0022] In various examples, the one or more events may include a plurality of events, and the one or more criteria may include logical transition between the plurality of events. The one or more events may include a plurality of actions. The one or more criteria may include logical transition between the plurality of actions and / or continuity between the plurality of actions. The criteria may also include consistency with one or more conditioning digital images used to condition the video generation model to generate the synthetic digital video, which can include preservation of one or more visual features depicted in the one or more conditioning digital images. In some cases, the one or more events may include an embodied agent performing one or more actions.

[0023] The adaptation of the video generation model may be performed in several ways. For instance, the adapting may include reinforcement learning using the evaluation output as reward data. Alternatively, the adapting may include direct preference optimization (DPO) using the evaluation output as preference data. The adapting may be based on reward-weighted regression (RWR), a unified objective with different Kullback-Leibler (KL) regularization, expectation maximization, or reward maximization with Bradley-Terry assumptions.

[0024] In various implementations, the generative model used for evaluation may be a visionlanguage model (VLM), and the video generation model may be a denoising diffusion probabilistic model (DDPM).

[0025] Following the adaptation, the method may further include assembling a new input prompt containing a request to generate a new synthetic digital video, and processing the newinput prompt using the adapted video generation model to generate the new synthetic video. Based on this new synthetic video, robot control data may be generated, which may then be used to cause a real or simulated robot to be operated.

[0026] In another aspect, a method may be described that includes assembling, as an input prompt, data indicative of one or more conditioning digital images and a request to generate a synthetic digital video that portrays one or more requested events. This input prompt may be processed using a video generation model to generate the requested synthetic digital video, which includes a temporal sequence of synthetic digital video frames portraying the one or more requested events. The video generation model may have been previously adapted by evaluating a training synthetic digital video with a generative model based on one or more criteria, and using the resulting training output to adapt the video generation model.

[0027] In yet another aspect, a method may be described which includes generating a first synthetic digital video using a video generation model, where the first video portrays one or more simulated events. An evaluation of this first video may be obtained from a generative model, with the evaluation based on criteria related to the physical plausibility of the simulated events. The video generation model may then be adapted based on this evaluation, for example, by performing reinforcement learning using the evaluation as reward data, to produce an adapted video generation model. Subsequently, a second synthetic digital video may be generated using the adapted model, where this second video portrays another set of simulated events that exhibit increased adherence to the evaluation criteria relative to the first video.

[0028] In a further aspect, a method may include generating a plurality of synthetic digital videos using a video generation model. Each video may portray one or more simulated events and include a temporal sequence of frames. A plurality of evaluations for these videos may be obtained from a vision-language model, with each evaluation indicating an assessment based on criteria such as physical plausibility and logical consistency. From these videos, a plurality of video pairs may be created, where each pair includes a first video and a second video, and where the evaluation for the first video indicates a higher adherence to the criteria than the evaluation for the second video. The video generation model may then be adapted by performing direct preference optimization using the plurality of video pairs and their evaluations as preference data, producing an adapted model configured to generate subsequent videos with increased adherence to the criteria.

[0029] Additionally, transitory and non-transitory computer-readable media and systems are also described herein that are adapted to perform, or cause the performance of, the methods described previously.Brief Description of the Drawings

[0030] Fig. l is a block diagram of an example system for adapting a video generation model using generative model feedback.

[0031] Fig. 2 is a data flow diagram illustrating an example process for adapting a text-to-video model using vision-language model feedback.

[0032] Fig. 3 is a flowchart of an example method for adapting a video generation model based on evaluation output from a generative model.

[0033] Fig. 4 is a flowchart of an example method for adapting a video generation model using direct preference optimization.

[0034] Fig. 5 is a block diagram of an example computing device.Detailed Description

[0035] Implementations are described herein for improving multimodal generative models such as text-to-video models (also referred to herein as “video generation” models”), text-to-image models, text-to-audio models, audio-to- video models, etc., using external feedback. More particularly, but not exclusively, implementations are described herein for leveraging reinforcement learning (RL) and artificial intelligence (Al) generated feedback, such as natural language predicted using a vision-language model (VLM), to train and / or fine-tune multimodal generative models. With regard to RL-based finetuning, various techniques may be implemented, including but not limited to reward-weighted regression (RWR) and direct preference optimization (DPO) from the unified objective with different Kullback-Leibler (KL) regularization.

[0036] In various implementations, a denoising diffusion probabilistic model (DDPM) may be adopted for generating a video comprising a temporal sequence of frames. A DDPM may be configured to approximate a reverse process for data generation with a parameterized model. A forward process may also be part of the DDPM, where Gaussian noise may be iteratively added to data with different noise levelsAfter training a parameterized denoising model, a video may begenerated from an initial noise through an iterative denoising process. Conditioning on text and a first frame may also be adopted, for instance, with classifier-free guidance, allowing for the following conditional formulation of to be considered:where

[0037] The denoising process in DDPMs may be formulated as a / '-horizon Markov decision process (MDP). This MDP may include a state s an action , an initialstate distribution a transition function Pa reward function , and a policywhere 8^ is the Dirac distribution and ![•] is an indicator function. Further, a binary event variable O E {0,1} may be introduced to this MDP, which may represent whether a generatedvideo x0is optimal or not. To derive a unified reinforcement learning (RL) finetuning objective for a DDPM, a log-likelihood logp ( 0 = 1 I c ) may be considered and decomposed with a variational distribution p e.g., as follows:

[0038] This may result in an evidence lower bound, such as the following:

[0039] An assumption may be made regarding a dependence on a reward such as p( 0 = 1 | x0, c ) oc exp(p-1r(x0, c)), for example, because 0 stands for the optimality of the generated video. This allows for the evidence lower bound <7R(p, p') to be rewritten in an explicit form of a unified objective for RL-finetuning: such as the following:Existing algorithms may be derived from this unified objective through policy projection with expectation-maximization methods, or with Bradley-Terry assumptions.

[0040] In some implementations, practical algorithms may be derived to optimize the unified objective for reinforcement learning finetuning. A brief overview of metric-based feedback for text-to-video models is provided, followed by a description of a pipeline for utilizing Al feedback from vision-language models (VLMs).

[0041] In some examples, the general form of the RL-finetuning objective for a DDPM set forth above may maximize a reward under KL-regularization. Practical implementations may be derived based on expectation-maximization and on Bradley-Terry assumptions.

[0042] One type of algorithm may be a Forward-EM-Proj ection. This type of algorithm may parameterize a variational distribution p' = pe( x0I c ) and p = pref ( x0| c ) as a function of the model output given a condition and a prior distribution as a function of a reference model output given the condition (z.e., forward KL-regularization). A coordinate ascent, similar to ageneric EM algorithm, may be performed. The unified objective may be solved with respect to the variational distribution while freezing the parametric posterior (E-step), and the new variational distribution may be projected into the model parameter (M-Step). The E-step may convert the unified objective<7R(p, p') into a constraint optimization problem and consider a corresponding Lagrangian, such as the following:

[0043] An analytical solution for this Lagrangian may correspond to a new variational distribution which is a function of a partition function, the parametric posterior, and an exponential function of a reward value, such as the following:where Z(c) is the partition function.

[0044] The M-step may project the non-parametric optimal modelto the parametric model by maximizing the unified objective described above with respect to the modelparameters, e.g., as follows:

[0045] To stabilize training, reward-weighted regression (RWR) for diffusion models may simplify the objective < by, for example, removing the intractable normalization Z(c),setting pre-trained models ppreinto , converting an exponential transform into an identitymapping, and considering a simplified upper-bound of negative log-likelihood. This may result in a practical minimization objective, such as the following:

[0046] Another category of algorithm may be a Reverse-BT-Proj ection. This category may parameterize the prior distribution as a function of the model output given acondition and the variational distribution as a function of a reference modeloutput given the condition (z.e., reverse KL-regularization). While on-policy policy gradient methods may optimize the unified objective, such methods may involve substantial computational costs and may be unstable for text-to-video models. Alternatively, a lightweight approach may be considered by optimizing a surrogate objective to extract the non-parametric optimal model into the parametric data distribution, for instance, as in direct preference optimization (DPO). An additional Bradley-Terry assumption may be introduced, where, if one video is more preferable than another the preference probabilitymay be a function of the reward r , as follows:where is the sigmoid function.

[0047] By transforming the analytical solution for the Lagrangian into a parameterized model a parameterized reward,may be obtained. The surrogate objective may be the maximization of the log-likelihood of the preference probability. This may be achieved by substituting the parameterized reward reinto the preference probability function p and considering a simplified lower-bound of the log-likelihood:

[0048] In some implementations, the following technique may be used to perform offline RL- finetuning to train a text-to-video / video generation model using Al feedback, with two different options for finetuning presented:Inputs: Pe , a dataset for pre-traininga set of conditional features for finetuningand a reward model1. Pretrain text-to-video model byminimizing IDDPM2. Generate videoconditioned on cJ eCfineto collect a dataset Dfine for finetuningPe3. Label Al feedback metric based reward4. finetunefe4A: For RWR, minimizewith4B: For DPO, minimize ^RL-KL-BT with paired dataset fromwhere BT stands for Bradley-Terry assumptions.

[0049] With regard to step 3, various types of VLMs may be used, including but not limited various versions of Gemini. One non-limiting example of the type of input prompt that might be used to prompt the VLM is set forth below:Task: You are a video reviewer evaluating a sequence of actions presented as eight consecutive images in the video below. You are going to accept the video if it completes the task and the video is consistent without glitches.Inputs Provided:Textual Prompt: Describes the task the video should accomplish.Sequence of Images (8 Frames): Represents consecutive moments in the video to be evaluated.Evaluation Process:View and Analyze Each Frame: Examine each of the eight images in sequence to understand the progression and continuity of actions.Assess Overall Coherence: Consider the sequence as a continuous scene to determine if the actions smoothly transition from one image to the next, maintaining logical progression.Check for Physical Accuracy: Ensure each frame adheres to the laws of physics, looking for any discrepancies in movement or positioning.Verify Task Completion: Check if the sequence collectively accomplishes the task described in the textual prompt.Identify Inconsistencies: Look for inconsistencies in object movement or overlaps that do not match the fixed scene elements shown in the first frame.Evaluation Criteria:Accept the sequence if it is as a coherent video which completes the task.Reject the sequence if any frame fails to meet the criteria, showing inconsistencies or not achieving the task. Reject even if there are the slightest errors. Do not be too strict in accepting the videos.Response Requirement:Provide a single-word answer: Accept or Reject. Do not give reasoning.Textual Prompt: {instruction} Video: {video tokens}

[0050] As set forth in this prompt, in various implementations, the output generated using the VLM may take a form requested in the VLM feedback-seeking prompt. For instance, if the VLM feedback-seeking prompt includes a command to simply accept or reject the sequence of synthetic digital images based on one or more of the criteria, as is the case with the example prompt above, then the evaluation output may be binary (e.g., “accept” or “reject”). On the other hand, if the VLM feedback-seeking prompt includes a command to generate natural language feedback about the sequence of synthetic digital images using one or more of the criteria, then the evaluation output may include one or more natural language statements orcritiques identifying one or more issues (e.g., violation of laws of physics, failure to accomplish high-level task or goal, rough transitions, etc.) in the synthetic video, and possible suggested fixes (e.g., “make sure the robot arm is moved around the doorframe, rather than through it”).

[0051] As is also demonstrated by the example prompt above, various criteria may be specified to evaluate a synthetic content. These criteria may include, but are not limited to, adherence of event(s) portrayed in synthetic content with law(s) of physics, adherence of event(s) portrayed in synthetic content with a request to simulate the event(s), consistency with conditioning digital real or synthetic image(s) used to condition the video generation model, preservation of visual feature(s) depicted in conditioning digital image(s), logical transition between a plurality of events, logical transition between a plurality of actions performed by an embodied agent in the synthetic content, and / or continuity between the plurality of actions, to name a few.

[0052] Another source of feedback for synthetic content is real life or simulated operation of an embodied agent, such as a robot, based on event(s) portrayed in synthetic content(s). In various implementations, synthetic content such as synthetic video may be processed to extract action(s) performed by an embodied agent in the synthetic video. For example, Cartesian coordinates of reference point(s) associated with joint(s) or other features of a robot may be tracked across the individual synthetic digital images of the synthetic video to build a trajectory or trace of the Cartesian coordinates over time. Inverse dynamics processing may then be performed on the trajectory of Cartesian coordinates to generate robot control data, such as a trajectory of joint commands. As another example, the final image of the synthetic video may be used as a goal image to condition a robot policy to generate the robot control data.

[0053] “Robot control data” may include, for instance, low-level actuator commands (also referred to as “joint commands,” and may include torque commands) that directly control the actuators / joints of a robot, cartesian commands that specify directi on(s) for an end effector, a target robot pose, code that specifies reward functions that a motion controller can optimize (e.g., using techniques such as receding horizon optimization) to find optimal low-level actuator commands, selected predefined robot primitives, and so forth. In some cases, robot logic may be configured to convert between joint commands and Cartesian commands, e.g., using forward and / or inverse kinematics.

[0054] Fig. 1 schematically depicts an example environment in which selected aspects of the present disclosure may be implemented. It will be appreciated that the environment of FIG. 1 is illustrative and not limiting of implementations disclosed herein. Many variations,modifications, and alternatives will be apparent in light of this disclosure. The components depicted in FIG. 1 may be configured and interconnected in various other ways, and some components may be omitted altogether.

[0055] Fig. 1 illustrates an example knowledge system 100 operably coupled to a client device 132 operated by a user 130 via one or more networks 199. Knowledge system 100 may be implemented in various ways, such as on one or more server computers, in a distributed computing environment, or using a cloud-based platform. For example, the components of knowledge system 100 may be executed by a single server or distributed across multiple servers that are geographically co-located or dispersed. Client device 132 may take any of a number of forms, such as a desktop computer, a laptop computer, a smartphone, a tablet, or a wearable device. Network(s) 199 can include any type of network, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, facilitating communication between knowledge system 100 and client device 132.

[0056] Knowledge system 100 includes a prompt assembly engine 102, a text-to-video engine 104 with access to one or more text-to-video (T2V) models 106, a VLM-powered evaluator 108 with access to one or more vision-language models (VLMs) 110, and a training module 112. Prompt assembly engine 102 is configured to receive a user query 112 from user 130 via client device 132. User query 112 can be a request to generate a video portraying one or more events. Prompt assembly engine 102 processes user query 112 to generate an input prompt 114, which is provided to text-to-video engine 104. Text-to-video engine 104 processes input prompt 114 using T2V model 106 to generate one or more initial synthetic videos 116. T2V model 106 may be, for instance, a denoising diffusion probabilistic model (DDPM). Initial synthetic video(s) 116 are temporal sequences of synthetic digital video frames that portray one or more simulated events, but may suffer from drawbacks such as unrealistic physics, poor adherence to the prompt, or logical inconsistencies.

[0057] Initial synthetic video(s) 116 may be processed by VLM-powered evaluator 108. VLM- powered evaluator 108 uses VLM(s) 110 to evaluate initial synthetic video(s) 116 based on one or more criteria and generate feedback 118. For example, the criteria may include adherence to physical laws. In a hypothetical synthetic video generated from the request "pushing a bottle so that it falls off the table," the VLM-powered evaluator 108 could assess whether the bottle's movement conforms to principles of gravity and momentum, such as accelerating as it falls and not passing through the solid table surface. Another possible criterion is adherence to thegeneration request. For a video meant to portray "taking one body spray of many similar," the VLM-powered evaluator 108 could determine if the video shows a single, specific can being removed from a group, rather than an incorrect number of cans or an entirely different object.

[0058] Logical transition between events is another potential criterion. For a video depicting a multi-step task like "opening a drawer, picking up a key, and closing the drawer," the VLM- powered evaluator 108 could verify that the actions occur in a coherent sequence. Consistency with a conditioning image could also be a criterion. If a video is generated based on an initial frame showing a red ball on a blue box, the VLM-powered evaluator 108 can check if subsequent frames maintain the color and shape of the ball and box, unless the prompt requests a change. Furthermore, continuity of action can be evaluated. In a video showing an embodied agent "folding a rag," the VLM-powered evaluator 108 may assess whether the rag's shape changes smoothly and plausibly from one frame to the next, without sudden jumps or illogical deformations.

[0059] Feedback 118 can take various forms. For instance, it can be a binary output, such as one or more tokens expressing "accept" or "reject," indicating whether the video meets the specified criteria. In other implementations, feedback 118 may comprise natural language statements that critique the video, identifying specific issues such as violations of physical laws or inconsistencies in object movement.

[0060] Feedback 118 is processed by training module 112 to adapt T2V model 106. This adaptation process can be performed using various techniques. For example, training module 112 can perform reinforcement learning, using feedback 118 as reward data. In another example, the adaptation can involve direct preference optimization (DPO), where feedback 118 for pairs of videos is used as preference data. Other techniques, such as reward-weighted regression (RWR), may also be used.

[0061] Through this adaptation process, T2V model 106 is updated (e.g., trained, fine-tuned) to generate improved synthetic videos. Subsequent synthetic videos generated using the adapted T2V model 106 exhibit increased adherence to the evaluation criteria. For example, if an initial video portrayed an object moving in a physically implausible way (e.g., passing through a solid object), after adaptation based on feedback indicating this flaw, a subsequent video generated from the same prompt is more likely to show the object moving in a physically plausible manner (e.g., moving around the solid object). As another real -world example, in a scenario involving multiple interacting objects where an initial video failed to maintain object consistency, a videogenerated by the adapted model would portray the interactions with improved logical consistency and object permanence. This iterative feedback and adaptation loop allows knowledge system 100 to produce synthetic videos with greater realism, coherence, and adherence to user requests.

[0062] Instead of, or in addition to, adapting T2V model 106, feedback 118 may be used in other ways to improve the quality of generated video content. For example, feedback 118 can be used to condition T2V model 106 during the video generation process itself, a technique known as "in context" learning. In such an implementation, text-to-video engine 104 can be configured to process not only input prompt 114 but also feedback 118 to generate a synthetic video. The feedback 118 may act as an additional conditioning signal, guiding T2V model 106 to generate a video that is more likely to satisfy the criteria evaluated by VLM-powered evaluator 108. For instance, if feedback 118 from a previous generation attempt indicates a physical implausibility (e.g., "the object passed through the wall"), this natural language statement can be incorporated into a subsequent prompt for T2V model 106, along with the original request, to steer the generation process towards a more physically plausible outcome (e.g., "the object moved around the wall"). This approach allows for real-time or near-real-time correction and refinement of the video generation process, potentially reducing the number of iterative generation and evaluation cycles.

[0063] Fig. 2 illustrates an example data flow for adapting a T2V model using VLM-generated feedback. The process begins at the left of Fig. 2, where a prompt for a dynamic scene 114 is provided to one or more pretrained T2V models 106A. For example, pretrained T2V model(s) 106A may be trained or fine-tuned on large-scale datasets comprising video content and associated textual descriptions, such as data sourced from the internet. The training process can involve learning to generate video frames that correspond to given textual prompts, often using architectures such as 3D-UNet within a diffusion model framework. This pretraining phase equips the model with a general capability to understand and visualize a wide range of concepts and actions.

[0064] Pretrained T2V model(s) 106A processes prompt 114 to sample a number of subpar videos 250. These subpar videos 250 may contain various artifacts and shortcomings. For example, if prompt 114 is "pushing a bottle so that it falls off the table," a resulting subpar video 250 may depict the bottle passing through the table surface, which is a violation of physical laws. In another example where the prompt is "taking one body spray of many similar," a subparvideo 250 might show the wrong number of sprays being moved or might fail to maintain the consistency of the other, unselected sprays in the background. Other issues could include implausible movements of deformable objects, such as a rag folding in an unnatural manner, or illogical transitions between actions in a multi-step task.

[0065] Subpar videos 250 are processed by VLM-powered evaluator 108, which generates feedback 118. In some implementations, feedback 118 may be a binary output, such as an "accept" or "reject" token, indicating whether a video meets certain criteria. During a reinforcement learning-based finetuning process, training module 112 uses feedback 118 to adapt pretrained T2V model(s) 106A, resulting in improved T2V model(s) 106B. This reinforcement learning may be implemented using several techniques. For example, feedback 118 can be used as reward data to perform the adaptation. One specific algorithm is reward- weighted regression (RWR), where the adaptation objective is a minimization function that includes the reward (feedback 118) and the difference between noise and the output of the denoising model. Another technique is direct preference optimization (DPO), where feedback 118 is used as preference data, for example, by comparing pairs of videos. In this case, the adaptation may be based on reward maximization with Bradley-Terry assumptions, where the preference probability between two videos is a function of their respective rewards. The finetuning can also be based on a unified objective with different Kullback-Leibler (KL) regularization and may be solved using expectation maximization. Once adapted, improved T2V model(s) 106B may be used to generate improved videos 252, which portray the events requested in prompt 114 without the shortcomings of subpar videos 250.

[0066] In a real-world example within a movie-making special effects setting, a director may need to generate a complex synthetic video for a science fiction film. The request, which becomes part of prompt 114, might be: "A futuristic chrome drone flies through a dense asteroid field, dodging several large, tumbling asteroids before landing on a moving platform." An initial pretrained T2V model 106A generates a set of subpar videos 250. One video might show the drone's reflection not matching the surrounding asteroids, violating physical laws of light reflection. Another might depict an asteroid suddenly vanishing or passing through the drone, which is not physically plausible. A third might show the drone's landing as a rough, illogical transition where it abruptly appears on the platform instead of smoothly descending.

[0067] These subpar videos 250 are fed to VLM-powered evaluator 108. The evaluator 108 processes each video against criteria such as "adherence with laws of physics" and "logicaltransition between events." For the video with incorrect reflections, feedback 118 would be a 'reject' token or a critique like "drone reflection is inconsistent with asteroid positions." For the video with the asteroid passing through the drone, the feedback would also indicate a rejection based on the violation of physical plausibility. This feedback 118 serves as reward data for training module 112. Using reinforcement learning, such as DPO, pairs of videos are evaluated. For example, a video where the drone narrowly avoids an asteroid would be preferred over one where it passes through it. The feedback acts as preference data to adapt pretrained T2V model 106A. Through this process, which may involve minimizing an objective based on Bradley- Terry assumptions, the model's parameters are updated, resulting in improved T2V model 106B. When the same prompt 114 is provided to this improved model 106B, it generates an improved video 252. This new video portrays the drone with accurate reflections, correctly navigating around the asteroids without any physically impossible interactions, and executing a smooth, continuous landing sequence on the moving platform, making it suitable for inclusion in the final film production.

[0068] Fig. 3 illustrates a flowchart of an example method 300 for practicing selected operations of the present disclosure. For convenience, method 300 is described as being performed by a system configured with selected aspects of the present disclosure, such as knowledge system 100 of Fig. 1. It should be understood that the operations of method 300 may be performed in a different order, and that certain operations may be added, omitted, combined, or rearranged.

[0069] At block 302, the system may assemble, as a video evaluation input prompt, data indicative of a synthetic digital video including a temporal sequence (e.g., 250) of synthetic digital video frames, and a request to evaluate the synthetic digital video based on one or more criteria. The synthetic digital video may be generated using a video generation model and may portray one or more events being simulated. In some implementations, prompt assembly engine 102 may be configured to assemble the video evaluation input prompt. The video generation model may include a denoising diffusion probabilistic model (DDPM), such as T2V model 106. The one or more events may include a plurality of actions, such as an embodied agent performing one or more actions.

[0070] At block 304, the system may process the video evaluation input prompt using a generative model to generate evaluation output indicative of an outcome of evaluating the synthetic digital video based on the one or more criteria. For example, VLM-powered evaluator 108 may be configured to process the prompt using a vision-language model (VLM), such asVLM(s) 110. The request to evaluate the synthetic digital video may include a request to generate binary output, and the evaluation output may include one or more tokens expressing binary output. The one or more criteria may include adherence of the one or more events portrayed in the synthetic digital video with one or more laws of physics. The criteria may also include adherence of the one or more events with a request to generate the synthetic digital video to portray the one or more events, where the request may be obtained from a video generation prompt used to condition the video generation model. Other criteria may include logical transition between a plurality of events, logical transition between a plurality of actions, continuity between a plurality of actions, consistency with one or more conditioning digital images used to condition the video generation model, and preservation of one or more visual features depicted in the one or more conditioning digital images.

[0071] At block 306, the system may, based on the evaluation output, adapt the video generation model. For example, training module 112 may be configured to adapt T2V model 106. In some implementations, the adapting may include reinforcement learning using the evaluation output as reward data. In some cases, the adapting includes direct preference optimization (DPO) using the evaluation output as preference data. The adapting may be based on reward-weighted regression (RWR), a unified objective with different Kullback-Leibler (KL) regularization, expectation maximization, or reward maximization with Bradley-Terry assumptions.

[0072] The adapted video generation model may then be used to generate new synthetic videos. For example, the system may assemble a new input prompt containing a request to generate a new synthetic digital video and process the new input prompt using the video generation model to generate the new synthetic video. Furthermore, in some use cases, the system may generate robot control data based on the new synthetic video and cause a real or simulated robot to be operated based on the robot control data.

[0073] Various other techniques may be used to train T2V model(s) 106. Fig. 4 illustrates a flowchart of an example method 400 for practicing selected operations of the present disclosure. For convenience, the operations of method 400 will be performed by a system configured with selected aspects of the present disclosure, such as knowledge system 100 of Fig. 1. It should be understood that various operations of method 400 may be added, omitted, reordered, combined, or rearranged.

[0074] At block 402, the system, e.g., by way of text-to-video engine 104, may generate, using a video generation model (e.g., 106), a plurality of synthetic digital videos. Each synthetic digitalvideo of the plurality of synthetic digital videos may portray one or more simulated events and include a temporal sequence of synthetic digital video frames. For example, text-to-video engine 104 may use T2V model 106 to generate the plurality of synthetic digital videos, such as subpar videos 250 shown in Fig. 2.

[0075] At block 404, the system, e.g., by way of VLM-powered evaluator 108, may obtain, from a vision-language model (e.g., VLM(s) 110), a plurality of evaluations corresponding to the plurality of synthetic digital videos. Each evaluation of the plurality of evaluations may be indicative of an outcome of an assessment of a corresponding synthetic digital video based on one or more criteria related to physical plausibility and logical consistency of the one or more simulated events portrayed therein. In some implementations, VLM-powered evaluator 108 may be configured to use VLM(s) 110 to produce the plurality of evaluations.

[0076] At block 406, the system, e.g., by way of training module 112, may create a plurality of video pairs from the plurality of synthetic digital videos. Each video pair may include a first synthetic digital video and a second synthetic digital video from the plurality of synthetic digital videos, wherein the evaluation corresponding to the first synthetic digital video indicates a higher adherence to the one or more criteria than the evaluation corresponding to the second synthetic digital video. For example, training module 112 may be configured to assemble these pairs based on feedback 118.

[0077] At block 408, the system, e.g., by way of training module 112, may adapt the video generation model by performing direct preference optimization using the plurality of video pairs and the plurality of evaluations as preference data. This may produce an adapted video generation model configured to generate subsequent synthetic digital videos exhibiting increased adherence to the one or more criteria. In some implementations, training module 112 may perform the direct preference optimization to adapt T2V model 106, resulting in an improved T2V model 106B as depicted in Fig. 2.

[0078] In a robotics and construction context, method 400 can be implemented to improve the generation of synthetic videos for training construction robots. Consider a scenario where an autonomous robotic arm is being trained to perform masonry tasks, such as precisely placing bricks to build a wall. The task involves picking up a brick, applying mortar, and placing it accurately on the wall being constructed.

[0079] At block 402, a video generation model generates a plurality of synthetic digital videos based on a prompt like "a robotic arm picks a brick from a stack, applies mortar to one side, andplaces it on a wall." Initial videos generated by a non-adapted model might contain physical implausibilities. For example, some videos might show the robotic gripper passing through other bricks in the stack, the mortar appearing on the brick without the application step, or the brick being placed in a position that defies gravity without support.

[0080] At block 404, these synthetic videos are provided to a vision-language model (VLM) for evaluation. The VLM assesses each video based on criteria such as physical plausibility (e.g., "Does the robotic arm avoid collisions with other objects?") and logical consistency of the construction task (e.g., "Is mortar applied before the brick is placed? Is the brick placed in a stable position?"). The VLM outputs a plurality of evaluations, for instance, a binary "accept" or "reject" for each video. A video where the brick is placed floating in mid-air would be rejected.

[0081] At block 406, the system creates a plurality of video pairs. Each pair consists of a "preferred" video (one that the VLM accepted or rated higher) and a "dispreferred" video (one that was rejected or rated lower). For example, a video showing the robot correctly picking a single brick from the top of a stack would be paired with, and preferred over, a video where the robot's gripper appears to phase through multiple bricks.

[0082] At block 408, the video generation model is adapted by performing direct preference optimization (DPO). The plurality of video pairs and their corresponding evaluations are used as preference data to update the parameters of the model. This adaptation process trains the model to generate videos that are more likely to be evaluated favorably by the VLM. The resulting adapted video generation model is now better configured to produce physically plausible and logically coherent synthetic videos of the masonry task. These higher-quality videos can subsequently be used to generate more effective and reliable robot control data for training real- world construction robots, improving their precision and safety on a job site.

[0083] Fig. 5 is a block diagram of an example computing device 510 that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device 510 typically includes at least one processor 514 which communicates with a number of peripheral devices via bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, user interface output devices 520, user interface input devices 522, and a network interface subsystem 516. The input and output devices allow user interaction with computing device 510. Network interface subsystem 516 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

[0084] User interface input devices 522 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device 510 or onto a communication network.

[0085] User interface output devices 520 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computing device 510 to the user or to another machine or computing device.

[0086] These software modules are generally executed by processor 514 alone or in combination with other processors. Memory 525 used in the storage subsystem 524 can include a number of memories including a main random-access memory (RAM) 530 for storage of instructions and data during program execution and a read only memory (ROM) 532 in which fixed instructions are stored. A file storage subsystem 526 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem 526 in the storage subsystem 524, or in other machines accessible by the processor(s) 514.

[0087] Bus subsystem 512 provides a mechanism for letting the various components and subsystems of computing device 510 communicate with each other as intended. Although bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0088] Computing device 510 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 510 depicted in Fig. 5 is intended only as a specific example for purposes ofillustrating some implementations. Many other configurations of computing device 510 are possible having more or fewer components than the computing device depicted in Fig. 5.

[0089] In some examples, the machine learning models described herein can be used for controlling a robotic device or a simulated robotic device.

[0090] The input to the machine learning model may comprise a natural language description of a task to be performed by the robotic device. For example the input may comprise speech or text data. Speech data may be captured by a microphone on the robotic device or on a separate device for example. Text data may be entered by a user through a keyboard or touchscreen on the robotic device or on a separate device for example, or may be generated from speech data captured by a microphone on the robotic device or on a separate device for example (for example using automatic speech recognition techniques). Thus the input may include textual or spoken instructions provided to the robotic device by a third-party (e.g., an operator). In particular, a user may control the robotic device using a client device such as a tablet computer or smart phone for example.

[0091] The input may additionally or alternatively comprise sensor data generated by one or more sensors on the robotic device or in the environment of the robotic device. For example, the input may comprise image data captured by one or more vision sensors such as one or more cameras (e.g., RGB, infrared). The input may comprise a three-dimensional (3D) digital representation of the environment captured by one or more sensors such as LIDAR sensors or depth cameras, for example point cloud data generated using a light detection and ranging (LIDAR) sensor. For example, the input may comprise sensor data from a distance or position sensor, or from an actuator. The input may include data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0092] The input may include data characterizing the current state of the robot, e.g., one or more of joint positionjoint velocity joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. The input data may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative data. The input may also include, for example, sensed electronic signals such as motor current or a temperature signal. The input may include data captured from e.g. one or more force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as “distance sensors”), depth sensors, torque sensors, barcode readers, radio frequency identification (“RFID”) readers, radars, range finders, accelerometers, gyroscopes, compasses, positioncoordinate sensors (e.g., global positioning system, or “GPS”), speedometers, edge detectors, Geiger counters, and so forth.

[0093] The output of the machine learning model may comprise data representing one or more tasks to be performed by the robotic device in order to perform the task.

[0094] For instance, the output may comprise natural language, for example text or speech, that describes steps or sub-tasks for completing a task. The output may define one or more low-level skills, e.g. from a vocabulary of previously learnt skills.

[0095] The output may comprise robot control data that is usable to control a robot to complete the task, for example. The robot control data may include, for instance, low-level actuator commands that directly control actuators of the robotic device, cartesian commands that specify direction(s) for an end effector of the robotic device, a target robot pose, selected predefined robot primitives, and so forth. As an illustration, the output may comprise action tokens, that can be converted into a control signal for the robotic device. For example, the action tokens may represent variables for arm movement (such as one or more of: x, y, z, roll, pitch, yaw, gripper opening), variables for base movement (such as one or more of: x, y, yaw), and variables to switch between modes (such as a variable to switch between controlling arm, controlling base, or terminating the episode). Each action dimension may be discretized, for example into 256 bins.

[0096] The output may comprise reward parameters that can be optimized by a low-level motion controller to determine low-level actuator commands.

[0097] The output may comprise robot policy code expressing functions or feedback loops that process perception outputs and parameterize control primitive APIs. For example, the output may comprise API calls to generate policy code.

[0098] The output may represent candidate robot or end effector trajectories, higher-level control commands, position, velocity, or force / torque / accel eration data for one or more joints, or electronic control data such as motor control data for example.

[0099] In various implementations, the robot may be simulated in a virtual environment. The input may comprise data representing the virtual environment in which the simulated robot operates, for example image data representing the virtual environment.

[0100] The robotic device may take various forms, including but not limited to a telepresence robot, a robotic arm, a bi-arm robotic device, a humanoid robot or other bipedal robot, a quadruped robot such as a “robot dog”, a wheeled robot, an aquatic robot, and so forth. Therobotic device may include control logic. Control logic may take various forms, such as a real time controller, one or more processors, one or more field-programmable gate arrays (“FPGA”), one or more application-specific integrated circuits (“ASIC”), and so forth. In some implementations, the logic may be operably coupled with memory. Memory may take various forms, such as random-access memory (“RAM”), dynamic RAM (“DRAM”), read-only memory (“ROM”), Magnetoresistive RAM (“MRAM”), resistive RAM (“RRAM”), NAND flash memory, and so forth. In some implementations, the control logic may be operably coupled with one or more joints, one or more end effectors, and / or one or more sensors. A joint of a robot may broadly refer to actuators, motors (e.g, servo motors), shafts, gear trains, pumps (e.g, air or liquid), pistons, drives, propellers, flaps, rotors, or other components that may create and / or undergo propulsion, rotation, and / or motion. An end effector may broadly refer to a variety of tools that may be operated by the robotic device in order to accomplish various tasks. For example, an end effector may take the form of a claw with two opposing “fingers” or “digits.” Such a claw is one type of “gripper” known as an “impactive” gripper. The gripper may have more than two digits, for example, three, four or five digits. Other types of grippers may include but are not limited to “ingressive” (e.g., physically penetrating an object using pins, needles, etc.), “astrictive” (e.g., using suction or vacuum to pick up an object), or “contigutive” (e.g., using surface tension, freezing or adhesive to pick up object). More generally, other types of end effectors may include but are not limited to drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, trays, and so forth. In some implementations, end effector may be removable, and various types of modular end effectors may be installed onto robot. Some robots, such as some telepresence robots, may not be equipped with end effectors.

[0101] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is,therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

CLAIMSWhat is claimed is:

1. A method implemented using one or more processors and comprising: assembling, as a video evaluation input prompt, data indicative of: a synthetic digital video comprising a temporal sequence of synthetic digital video frames, and a request to evaluate the synthetic digital video based on one or more criteria, wherein the synthetic digital video is generated using a video generation model and portrays one or more events being simulated; processing the video evaluation input prompt using a generative model to generate evaluation output indicative of an outcome of evaluating the synthetic digital video based on the one or more criteria; and based on the evaluation output, adapting the video generation model.

2. The method of claim 1, wherein the request to evaluate the synthetic digital video includes a request to generate binary output, and the evaluation output comprises one or more tokens expressing binary output.

3. The method of any of the preceding claims, wherein the one or more criteria comprise adherence of the one or more events portrayed in the synthetic digital video with one or more laws of physics.

4. The method of any of the preceding claims, wherein the one or more criteria comprise adherence of the one or more events portrayed in the synthetic digital video with a request to generate the synthetic digital video to portray the one or more events.

5. The method of claim 4, wherein the request is obtained from a video generation prompt used to condition the video generation model to generate the synthetic digital video.

6. The method of any of the preceding claims, wherein the one or more events comprise a plurality of events, and the one or more criteria comprise logical transition between the plurality of events.

7. The method of any of the preceding claims, wherein the one or more events comprise a plurality of actions.

8. The method of claim 7, wherein the one or more criteria comprise logical transition between the plurality of actions.

9. The method of claim 7 or 8, wherein the one or more criteria comprise continuity between the plurality of actions.

10. The method of any of the preceding claims, wherein the one or more criteria comprise consistency with one or more conditioning digital images used to condition the video generation model to generate the synthetic digital video.

11. The method of claim 10, wherein the one or more criteria comprise preservation of one or more visual features depicted in the one or more conditioning digital images.

12. The method of any of the preceding claims, wherein the one or more events comprises an embodied agent performing one or more actions.

13. The method of any of the preceding claims, wherein the adapting comprises reinforcement learning using the evaluation output as reward data.

14. The method of any of the preceding claims, wherein the adapting comprises direct preference optimization (DPO) using the evaluation output as preference data.

15. The method of any of the preceding claims, wherein the adapting is based on reward-weighted regression (RWR).

16. The method of any of the preceding claims, wherein the adapting is performed based on a unified objective with different Kullback-Leibler (KL) regularization.

17. The method of any of the preceding claims, wherein the adapting is based on expectation maximization.

18. The method of any of the preceding claims, wherein the adapting is based on reward maximization with Bradley-Terry assumptions.

19. The method of any of the preceding claims, wherein the generative model comprises a vision-language model (VLM).

20. The method of any of the preceding claims, wherein the video generation model comprises a denoising diffusion probabilistic model (DDPM).

21. The method of any of the preceding claims, further comprising: assembling a new input prompt containing a request to generate a new synthetic digital video; and processing the new input prompt using the video generation model to generate the new synthetic video.

22. The method of claim 21, further comprising: generating robot control data based on the new synthetic video; andcausing a real or simulated robot to be operated based on the robot control data.

23. A method implemented using one or more processors, comprising: assembling, as an input prompt, data indicative of one or more conditioning digital images and a request to generate a synthetic digital video that portrays one or more requested events; processing the input prompt using a video generation model to generate the requested synthetic digital video, wherein the requested synthetic video includes a temporal sequence of synthetic digital video frames that portray the one or more requested events; wherein the video generation model was previously adapted by performing the following operations: assembling, as a training input prompt, data indicative of: a training synthetic digital video comprising a training temporal sequence of synthetic digital video frames, and a request to evaluate the training synthetic digital video based on one or more criteria, wherein the training synthetic digital video was generated using the video generation model and portrays one or more training events being simulated; processing the training input prompt using a generative model to generate training output indicative of an outcome of evaluating the training synthetic digital video based on the one or more criteria; and based on the training output, adapting the video generation model.

24. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to perform any of the methods of claims 1-23.

25. At least one transitory or non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform any of the methods of claims 1-23.

26. A method implemented using one or more processors, comprising: generating a first synthetic digital video using a video generation model, the first synthetic digital video portraying one or more simulated events;obtaining an evaluation of the first synthetic digital video from a generative model, the evaluation based on one or more criteria related to physical plausibility of the one or more simulated events; adapting the video generation model based on the evaluation to produce an adapted video generation model, wherein adapting the video generation model comprises performing reinforcement learning using the evaluation as reward data; and generating a second synthetic digital video using the adapted video generation model, the second synthetic digital video portraying a second one or more simulated events exhibiting increased adherence to the one or more criteria relative to the first synthetic digital video.

27. A method implemented using one or more processors, comprising: generating, using a video generation model, a plurality of synthetic digital videos, each synthetic digital video of the plurality of synthetic digital videos portraying one or more simulated events and comprising a temporal sequence of synthetic digital video frames; obtaining, from a vision-language model, a plurality of evaluations corresponding to the plurality of synthetic digital videos, each evaluation of the plurality of evaluations indicative of an outcome of an assessment of a corresponding synthetic digital video based on one or more criteria related to physical plausibility and logical consistency of the one or more simulated events portrayed therein; creating a plurality of video pairs from the plurality of synthetic digital videos, each video pair comprising a first synthetic digital video and a second synthetic digital video from the plurality of synthetic digital videos, wherein the evaluation corresponding to the first synthetic digital video indicates a higher adherence to the one or more criteria than the evaluation corresponding to the second synthetic digital video; and adapting the video generation model by performing direct preference optimization using the plurality of video pairs and the plurality of evaluations as preference data to produce an adapted video generation model configured to generate subsequent synthetic digital videos exhibiting increased adherence to the one or more criteria.

Images