Training method, video generation method and apparatus, electronic device, program product

Abstract

The method comprises the following steps: inputting input data into a simulator to output a first prediction value; inputting the input data into a neural network to be trained to output a second prediction value; calculating a value of a first loss function according to the second prediction value and the first prediction value; updating trainable parameters in the neural network according to the value of the first loss function until the neural network converges to obtain a target neural network; keeping the trainable parameters in the target neural network unchanged; inputting the input data into the target neural network to output a third prediction value; calculating a value of a second loss function according to the third prediction value and an observation value, and calculating a gradient; updating a physical parameter according to the gradient to obtain a target physical parameter; and configuring the target physical parameter into the simulator to obtain a target simulator.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a training method, a video generation method and apparatus, electronic devices, and program products. Background Technology

[0002] A policy model refers to the rules by which an intelligent agent, such as a robot, chooses actions under specific conditions. The policy is a core component that drives the robot; its inputs are various sensory information, such as video captured by a camera and the positions of the robot's joints, and its output is the action the robot must perform.

[0003] Training a policy requires a large amount of training data; however, collecting such data in the real world is generally expensive, time-consuming, and often limited in terms of scalability and repeatability. Acquiring training data in a simulation environment can reduce costs, but it may present other problems, such as the low quality of training data obtained in a simulation environment, which differs from the quality of training data obtained from real-world observations and thus lacks training value. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a training method, a video generation method and apparatus, an electronic device, and a program product.

[0005] According to a first aspect of the present disclosure, a training method is provided, comprising: inputting input data to a simulator, and outputting a first predicted value of the state of an agent at time step t via the simulator; wherein the input data includes sampled physical parameters and observed values ​​of the agent's actions and states at time step t-1; the input data is observation data obtained by observing a physical device including the agent in a real environment; the physical parameters include at least one of friction coefficient, stiffness, or damping coefficient; t ≥ 2, and t is an integer; inputting the input data to a neural network to be trained, and outputting a second predicted value of the state of the agent at time step t via the neural network; calculating the value of a first loss function based on the second predicted value and the first predicted value corresponding to the state of the agent at time step t; and calculating the value of a first loss function based on the first loss function. The value of the loss function is used to update the trainable parameters in the neural network until the neural network converges, resulting in the target neural network. Keeping the trainable parameters in the target neural network unchanged, the input data is fed into the target neural network, which outputs the third predicted value of the agent's state at time step t. Based on the third predicted value and the observed value corresponding to the agent's state at time step t, the value of the second loss function is calculated. The observed value corresponding to the agent's state at time step t is the observation data obtained from observing the physical device, including the agent, in a real environment. The gradient is calculated based on the value of the second loss function. The sampled physical parameters are updated based on the gradient until the second loss function converges, resulting in the target physical parameters. The target physical parameters are then configured into the simulator to obtain the target simulator.

[0006] According to a second aspect of the present disclosure, a training method is provided, the method comprising: determining a reference image from a sample video; the sample video being a photograph of an entity moving in a real environment; acquiring an image sequence corresponding to the reference image; the image sequence being obtained based on multiple frames of images following the reference image in the sample video; inputting the reference image and the image sequence as conditions for generating a video into a video generation model to be trained, thereby obtaining a first video through the video generation model; calculating the value of a third loss function based on the first video and the sample video; updating the trainable parameters in the video generation model based on the value of the third loss function until the video generation model converges, thereby obtaining a target video generation model.

[0007] According to a third aspect of the present disclosure, a video generation method is provided, the method comprising: determining a reference image; inputting the reference image into a policy model, and outputting an action sequence through the policy model; inputting the action sequence into a target simulator, and outputting an image sequence corresponding to the reference image through the target simulator; the target simulator is used to predict the actions of an entity device in the reference image, and the target simulator is trained using the method of the first aspect; inputting the image sequence and the reference image as conditions for generating a video into a target video generation model, and outputting a target video through the target video generation model; the target video is used to train the policy model, and the target video generation model is trained using the method of the second aspect.

[0008] Based on the training method of the first aspect provided in the above embodiments of this disclosure, a physical calibration model is set as a proxy model of the simulator. By training the proxy model, the proxy model learns the simulation process of the simulator. After the proxy model is trained, the physical parameters obtained by random sampling are updated by gradient through backpropagation according to the trained physical calibration model to obtain the calibrated physical parameters. The simulator uses the calibrated physical parameters to perform simulation prediction. In this way, by calibrating the physical parameters used in the simulator, the differences in physical dynamics caused by unreasonable physical parameter settings in the Real2Sim process can be reduced.

[0009] Based on the training method of the second aspect provided in the above embodiments of this disclosure, when inputting a reference image, the image sequence is also input as an additional condition into the video generation model. This allows the video generation model to generate a video that is more in line with expectations based on the reference image and the image sequence. For example, the image sequence may contain motion trajectory information or deformation information of physical devices such as robots. This enables the trained video generation model to generate a video that is more in line with expectations and avoids situations such as broken or distorted robotic arms / foreground objects caused by random generation. The generated video is more realistic, of higher quality, and reduces the differences in appearance and visual aspects during the Sim2Real process in the simulation environment.

[0010] Based on the training method of the third aspect provided in the above embodiments of this disclosure, the target simulator obtained in the first aspect is used for simulation, and an image sequence is output. Then, the image sequence and the reference image are used as conditions for the video generation model and input into the video generation model trained in the second aspect. Based on the target simulator, the differences in physical dynamics caused by unreasonable physical parameter settings in the Real2Sim process are reduced. Based on the video generation model, the differences in appearance and visual aspects in the Sim2Real process under the simulation environment are reduced, thereby reducing the overall Real2Sim2Real difference and obtaining more realistic videos as training sample videos for the policy.

[0011] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0014] Figure 1 This is a schematic diagram of the system architecture of one embodiment of the present disclosure.

[0015] Figure 2 This is a schematic diagram illustrating the three-stage execution of a training method for training a simulator in this disclosure.

[0016] Figure 3 This is a flowchart of one embodiment of the training method for training simulators disclosed herein.

[0017] Figure 4 This is a schematic diagram illustrating the use of MSE as the loss function in one embodiment of the training method for training simulators disclosed herein.

[0018] Figure 5 This is a flowchart of one embodiment of the training method disclosed herein for training a video generation model.

[0019] Figure 6 This is a schematic diagram of the system framework of an embodiment of the training method for training video generation models disclosed herein.

[0020] Figure 7 This is a schematic diagram illustrating two conditions input through multiple channels in one embodiment of the training method disclosed herein for training a video generation model.

[0021] Figure 8 This is a flowchart of one embodiment of the video generation method disclosed herein.

[0022] Figure 9 This is a schematic diagram of the system architecture of one embodiment of the video generation method disclosed herein.

[0023] Figure 10 This is a schematic diagram of the structure of one embodiment of the training apparatus for training simulators disclosed herein.

[0024] Figure 11 This is a schematic diagram of one embodiment of the training apparatus disclosed herein for training a video generation model.

[0025] Figure 12 This is a schematic diagram of one embodiment of the video generation apparatus disclosed herein.

[0026] Figure 13 This is a schematic diagram of the structure of an electronic device in one embodiment of the present disclosure. Detailed Implementation

[0027] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0028] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0029] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0030] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0031] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0032] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0033] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0034] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0035] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0036] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0037] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0038] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0039] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0040] SUMMARY

[0041] With the continuous development of artificial intelligence (AI) and embodied intelligence technologies, robotics has made significant progress in recent years. Embodied intelligence refers to the ability of robots to autonomously perceive their environment, make decisions, and perform actions in the physical world.

[0042] The development of embodied intelligence technology has driven significant progress in robot learning, enabling agents to perform complex manipulation, navigation, and decision-making tasks in real-world environments. At the heart of this advancement is the availability of large-scale, high-quality observational data that captures the underlying relationships between the sensory inputs and movements of agents operating in different physical environments. For example, policies, a core component used to drive robots, require vast amounts of observational data for training. However, collecting such data in the real world is costly, time-consuming, and often limited in terms of scalability and repeatability.

[0043] Acquiring training data in a simulation environment has become a viable solution. For example, some open-source robot learning benchmark frameworks in this field provide standardized testing environments for research such as reinforcement learning, imitation learning, and multi-task learning, and can be used as practical alternatives for training and evaluating robot policies.

[0044] In the process of implementing this disclosure, the inventors discovered that simulation environments in related technologies exhibit a Real2Sim2Real difference. This Real2Sim2Real difference includes differences in the process from reality to simulation (Real2Sim) and differences in the process from simulation to reality (Sim2Real). Two key sources of this difference are differences in physical dynamics and visual appearance.

[0045] For example, discrepancies in physical dynamics may arise from overly simplistic assumptions or unmodeled parameters such as joint stiffness, actuator delay, or surface friction. The physical behavior simulated in a virtual environment often differs from real-world dynamics. These mismatches lead to inaccurate motion trajectories and undermine the reliability of simulation-based control.

[0046] On the other hand, the difference in visual appearance may be due to the fact that the rendered appearance of simulated scenes often lacks the visual fidelity required to generalize to real-world sensory input. Therefore, training perceptual models such as policy models using training datasets lacking visual fidelity may result in perceptual models such as policy models that may not function effectively in real-world environments.

[0047] Furthermore, in related technologies, video generation models can be used to generate videos for training policies. The robot's motion trajectory in the generated videos is random and uncontrollable, and there may be situations where the movement does not conform to expectations, as well as situations where the robotic arm or foreground objects are broken or distorted. In other words, the video quality generated by the video generation models in related technologies is poor, and there is a significant difference in quality compared with the sample videos obtained by filming robots moving in the real world in the observation dataset.

[0048] These differences pose a significant challenge to policies trained in simulation environments, especially for tasks requiring precise physical interaction or high-quality visual perception. For example, robot policy models trained in simulation environments have a high success rate, but the success rate drops sharply when deployed on real machines.

[0049] Exemplary System

[0050] The training method and video generation method provided in this exemplary embodiment can be applied to... Figure 1 The exemplary system architecture shown includes a policy model, a simulator, a physical calibration model (PhysAligner), and a visual calibration model (VisAligner).

[0051] The training method and video generation method provided in an exemplary embodiment of this disclosure may specifically include a training method for training a physical calibration model, a training method for training a visual calibration model, and a method based on... Figure 1 The video generation method of the system architecture described above can generate videos that can be used as training samples to construct a training dataset. This training dataset can be used to train a policy model, enabling robots and other physical devices equipped with the policy model to make more accurate action decisions in the real world based on the real-time objective environment.

[0052] Figure 1 In the system architecture shown, the physical calibration model is a trained model. This trained physical calibration model is used to calibrate the physical parameters used in the simulator. This calibration process can also be understood as training the simulator using the physical calibration model. Furthermore, the visual calibration model is also a trained model. The training process aims to improve the quality of the videos generated by the visual calibration model, for example, generating more realistic videos that better match the expected motion trajectories.

[0053] Specifically, a reference image can be input into the policy model. This reference image can be an image containing part or all of a physical device such as a robot. The policy model outputs a motion sequence. The simulator, based on the motion sequence output by the policy model and using calibrated physical parameters, simulates real-world physical behavior and outputs an image sequence containing robot motion trajectory information. The visual calibration model uses the image sequence output by the simulator and the aforementioned reference image as conditions to generate a high-quality video. The generated video can be used as a training sample for training the policy model. Optionally, the last frame of the generated video, or a frame selected from the last few frames, can be used as the next reference image. In this way, multiple high-quality videos can be generated to construct a training dataset for training the policy model.

[0054] Among them, the physical calibration model is used to calibrate the physical parameters, which can make up for the differences in physical dynamics between reality and simulation by setting the physical parameters reasonably; the trained visual calibration model can generate more realistic or higher quality videos, reducing the difference between videos generated based on the simulation environment and videos collected in the real world.

[0055] Figure 1 The strategy model shown can be applied to robots, wearable devices, smart home devices, or other electronic devices (physical devices) that support interaction with the environment.

[0056] Exemplary Method

[0057] This disclosure provides an exemplary embodiment of a training method for training a physical calibration model. After obtaining the trained physical calibration model, the physical parameters are updated using gradients through the physical calibration model, and the updated physical parameters are configured into a simulator to obtain a trained simulator (target simulator). This training method can be applied to electronic devices.

[0058] like Figure 2 As shown, in some optional embodiments, the training process may include the following three stages:

[0059] Phase 1, Dataset generation: In this phase, the simulator 100 generates a dataset for training the physical calibration model.

[0060] Phase Two, Surrogate Modeling: The physical calibration model 110 is trained using the dataset generated in Phase One. The physical parameters used in both Phase Two and Phase One are the physical parameters 101 to be calibrated, which can be obtained through random sampling in the real world.

[0061] Phase 3, Gradient-based Refinement: The trained physical calibration model 111 is used to update the physical parameters 101 to be calibrated using gradients to obtain the calibrated physical parameters. The calibrated physical parameters are then configured into the simulator, thus obtaining the trained simulator.

[0062] Figure 2 The flame markers in the diagram indicate that the parameters of the object are being optimized in the current stage, while the snowflake markers indicate that the trainable parameters of the object are kept unchanged (frozen) in the current stage. For example, the training target of stage two is physical calibration model 110, and the trainable parameters of physical calibration model 110 are continuously optimized in this stage; in stage three, the trainable parameters of the trained physical calibration model 111 remain unchanged or are in a frozen state. For clarity, different states of the same model are distinguished by different labels; physical calibration model 110 is the model during the training process, and physical calibration model 111 is the model obtained after training is completed.

[0063] Specifically, Figure 3 This is a flowchart illustrating a training method provided in an exemplary embodiment of this disclosure, combined with... Figure 2 and Figure 3 This training method may include the following process:

[0064] Step 201: Input the input data to the simulator, and the simulator outputs the first predicted value of the agent's state at time step t.

[0065] For example, such as Figure 2 As shown in Phase 1, the input data includes sampled physical parameters ( f, p, d ) and the observed values ​​of the agent's actions at time step t-1. a t-1 The observed state of the agent at time step t-1 S t-1 The input data is fed into the simulator 100, which then outputs the first predicted value of the agent's state at time step t. P t t≥2, and t is an integer.

[0066] State refers to the state of an agent relative to its environment. For example, state can include the agent's current location in the environment. The state output by the simulator represents the state of the agent relative to the simulation environment.

[0067] Physical parameters ( f, p, d This includes at least one of the following: coefficient of friction, stiffness, or damping coefficient. f This represents the coefficient of friction.d This represents the damping coefficient. p It represents stiffness.

[0068] In Phase 1, a large amount of random sampling was used. f, p, d The simulator is driven to generate a first predicted value of the state at time step t. P t First predicted value P t This indicates that the simulator performs a certain operation on each sample. f, p, d The predicted value is obtained by predicting the actual state at time step t.

[0069] The input data consists of observation data obtained from observing physical devices, including the agent, in a real environment. The input data also includes the observed action of the agent at time step (t-1). a t-1 The observed state of the agent at time step t-1 S t-1 These can be obtained from the observation dataset. The physical parameters in Phase 1 can be the initial values ​​of a large number of physical parameters obtained through random sampling in the real world.

[0070] An agent can be a software system or entity capable of sensing its environment, making autonomous decisions, and taking actions. An entity that includes an agent can be a robot, or the agent itself can represent a robot.

[0071] Combination Figure 2 and Figure 3 Phase 1 may include step 201, Phase 2 may include steps 202-204, and Phase 3 may include steps 205-208.

[0072] Step 202: Input the input data into the neural network to be trained, and the neural network outputs the second predicted value of the agent's state at time step t. .

[0073] For example, such as Figure 2 In stage two, the same input data as in stage one is input to the simulator and then input to the physical calibration model 110 (the neural network to be trained). The physical calibration model 110 outputs a second predicted value of the agent's state at time step t. That is, the physical calibration model 110 outputs a second predicted value of the agent's state based on the input data. ,predict .

[0074] The physical calibration model 110 (111) can be implemented using a neural network. The neural network to be trained is the physical calibration model to be trained, such as a physical calibration model that is waiting to be trained or is in the process of training or waiting to continue training.

[0075] It should be noted that the physical calibration model can be implemented based on a differentiable neural network. Specifically, it can be based on the ability to calculate gradients using a loss function.

[0076] The physical calibration model 110 (the neural network to be trained) can be understood as a proxy model for the simulator 100. The reason for setting up a proxy model is that the simulator 100 cannot calculate the gradient. Calculating the gradient requires the objective function to be differentiable. The simulator is not differentiable, so it is impossible to directly update the physical parameters based on the gradient of the simulator.

[0077] Specifically, the gradient is calculated based on the loss function. The gradient is a vector of partial derivatives of the loss function with respect to the model parameters, representing the rate of change of the loss function in the parameter space. More specifically, the gradient is the partial derivative of the loss function with respect to the model parameters, reflecting the degree of influence of each trainable parameter in the model on the loss function during current training. By calculating the gradient, the direction and step size of parameter adjustment can be determined, allowing the parameters to gradually approach the optimal solution.

[0078] In this embodiment, a differentiable surrogate model is used to simulate the simulator. By training this surrogate model, it learns the simulation process of the simulator 100, thus solving the problem that the non-differentiable simulator cannot calculate gradients. The trained surrogate model is further used to update physical parameters using gradients. These physical parameters are then configured in the simulator, which uses the updated physical parameters for simulation.

[0079] Step 203: Calculate the value of the first loss function based on the second predicted value and the first predicted value corresponding to the state of the agent at time step t.

[0080] For example, such as Figure 2 As shown in Phase Two, and P t The value is substituted into the first loss function to calculate the value of the first loss function.

[0081] As a proxy model for simulator 100, physical calibration model 110 needs to learn the simulation performance of simulator. Therefore, it needs to use the output of simulator in stage one as the learning label. By calculating the difference between the output of physical calibration model 110 and the output of simulator 100, the trainable parameters in physical calibration model 110 are adjusted so that the prediction performance of physical calibration model 110 is closer to that of simulator.

[0082] The first loss function is the loss function used when training the physical calibration model.

[0083] Step 204: Update the trainable parameters in the neural network according to the value of the first loss function until the neural network converges to obtain the target neural network.

[0084] The target neural network is the trained physical calibration model 111.

[0085] Step 205: Keeping the trainable parameters in the target neural network unchanged, input the input data into the target neural network, and output the third predicted value of the agent's state at time step t. .

[0086] It should be noted that in stages two and three, the model parameters may change, and the outputs may differ for the same input data, as shown in stage two. As shown in Phase Three It can represent different values.

[0087] like Figure 2 The third stage shown involves sampling the physical parameters ( f, p, d ) and the observed values ​​of the agent's actions at time step t-1. a t-1 The observed state of the agent at time step t-1 S t-1 The input is fed into the target neural network 111 (i.e., the trained physical calibration model), and the target neural network 111 outputs the third prediction value at the t-th time step.

[0088] The training objective in Phase 2 is the physical calibration model 110. Unlike Phase 2, the training objective in Phase 3 is the physical parameter 102. In this Phase 3, the parameters in the target neural network 111 are frozen, and the physical parameters are updated by gradient. The optimized physical parameters are then configured in the simulator. The training process in Phase 3 with the physical parameters as the training objective is also the training process with the simulator as the training objective.

[0089] Step 206: Calculate the value of the second loss function based on the third predicted value and the observed value corresponding to the state of the agent at time step t.

[0090] For example, such as Figure 2 Phase three, as shown, will be the third predicted value. Compared with observed values S t Substitute the values ​​into the second loss function to calculate the loss value of the second loss function.

[0091] Wherein, the observed value corresponding to the state of the agent at time step t. S t This refers to the observational data obtained from observing physical devices, including intelligent agents, in a real environment (i.e., the real world). The training objective of Phase 3 is to adjust the physical parameters so that the simulator can achieve a simulation effect that is closer to the real world, and therefore uses the observational data collected in the real world as labels.

[0092] Step 207: Calculate the gradient based on the value of the second loss function.

[0093] Calculating the gradient, which is to calculate the vector of partial derivatives of the second loss function with respect to the physical parameters, represents the rate of change of the second loss function in the physical parameter space. Specifically, the gradient is the partial derivative of the loss function with respect to the physical parameters, reflecting the degree of influence of each physical parameter on the second loss function during the current training.

[0094] Step 208: Update the sampled physical parameters according to the gradient until the second loss function converges to obtain the target physical parameters.

[0095] By calculating the gradient, the direction and step size of the physical parameter adjustment can be determined, so that the physical parameters gradually approach the optimal solution.

[0096] Step 209: Configure the target physical parameters into the simulator to obtain the target simulator.

[0097] The simulator that uses the updated physical parameters described above is the target simulator.

[0098] Figure 3 In the illustrated embodiment, a physical calibration model is set as a proxy model for the simulator. By training the proxy model, the proxy model learns the simulation process of the simulator. After the proxy model is trained, the physical parameters obtained by random sampling are updated by backpropagation based on the trained physical calibration model to obtain calibrated physical parameters. The simulator uses the calibrated physical parameters to perform simulation prediction. In this way, by calibrating the physical parameters used in the simulator, the differences in physical dynamics caused by unreasonable physical parameter settings in the Real2Sim process can be reduced.

[0099] Among these measures, setting up a proxy model for the simulator also solved the problem that the simulator could not calculate gradients and did not support gradient updates of physical parameters.

[0100] In one alternative embodiment, based on Figure 3 Based on the illustrated embodiment, the neural network used to implement the physical calibration model can be a multilayer perceptron (MLP), for example... Figure 2The physical calibration model 110 shown is an example of an MLP.

[0101] In the embodiment of implementing the physical calibration model using MLP, MLP is a lightweight model that can reduce the complexity of the model and has the characteristics of efficient computation and simple structure. As a physical calibration model, MLP can provide good performance in resource-constrained environments.

[0102] In one alternative embodiment, based on Figure 3 Based on the illustrated embodiment, the neural network used to implement the physical calibration model can be a neural network including an attention mechanism, such as a neural network based on a transformer architecture, specifically a BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-trained Transformer) series model.

[0103] In the embodiment that uses a neural network including an attention mechanism to implement the physical calibration model, the bottlenecks such as low efficiency of sequence processing and difficulty in modeling long-distance dependencies are overcome by using the attention mechanism.

[0104] In one alternative embodiment, based on Figure 3 Based on the illustrated embodiment, the first loss function and the second loss function can be the same type of loss function or different types of loss functions. For example, as shown... Figure 4 As shown, the first loss function and the second loss function can both be the mean squared error loss function (MSE). Figure 4 The MSE Loss shown represents the mean squared error loss function. In other embodiments, both the first and second loss functions can be cross-entropy loss functions. Alternatively, in other embodiments, one loss function is the mean squared error loss function and the other is the cross-entropy loss function.

[0105] By calculating the square of the error, the MSE loss function can significantly amplify larger error values. This characteristic prompts the model to pay more attention to samples with larger prediction errors during training, thereby accelerating the convergence efficiency of the gradient descent process.

[0106] An exemplary embodiment of this disclosure also provides a training method for training a video generation model. The trained video generation model can generate more realistic videos using simulated videos (image sequences) output in a simulated environment as conditions. The generated videos can be used as training samples to train a policy. This training method can be applied to electronic devices.

[0107] Figure 5 This is a flowchart illustrating a training method for training a video generation model according to an exemplary embodiment of this disclosure. The training method may include the following steps:

[0108] Step 301: Determine the reference image from the sample video.

[0109] Sample videos are captured by filming physical devices moving in a real-world environment, such as robots. A sample video can be a single video obtained from an observation dataset.

[0110] Determining a reference image from a sample video can be done by randomly selecting a frame from the sample video, or by selecting the first frame or one of the first few frames as the reference image.

[0111] Step 302: Obtain the image sequence corresponding to the reference image.

[0112] The image sequence contains at least the motion trajectory information of the physical device in multiple frames following the reference image.

[0113] During the training phase of training the video generation model, the image sequence can be obtained based on multiple frames following a reference image frame in the sample video. For example, if the sample video is a video obtained from the observation dataset, and the first frame of the sample video is selected as the reference image, then the image sequence is the sequence of multiple frames following the first frame. For instance, an image sequence can be obtained by taking one frame at intervals from the multiple frames following the first frame.

[0114] Step 303: The reference image and image sequence are used as conditions for generating the video and input into the video generation model to be trained. The first video is obtained through the video generation model.

[0115] The reference image determined in step 301 is used as one condition for generating the video, and the image sequence obtained in step 302 is used as another condition for generating the video. Both are input into the video generation model to be trained, and the first video is output.

[0116] Step 304: Calculate the value of the third loss function based on the first video and the sample video.

[0117] Using sample videos as labels, the difference between the first video output and the sample videos is calculated. Then, the trainable parameters in the video generation model are adjusted based on the difference, so that the video generation model can learn to generate more realistic videos.

[0118] Step 305: Update the trainable parameters in the video generation model according to the value of the third loss function until the video generation model converges, and obtain the target video generation model.

[0119] Target video generation model, i.e. Figure 1 The video calibration model shown.

[0120] In related technologies, a common method for obtaining generated videos through video generation models is to input only a single reference image into the model. The model then generates a video based on this reference image. However, due to the limited information in the reference image, the output video may contain unexpected motion or exhibit broken or distorted robotic arms / foreground objects, resulting in low-quality videos that are unsuitable for use as training samples. This disclosure... Figure 5 In the illustrated embodiment, the image sequence is input as an additional condition along with the reference image into the video generation model. This allows the video generation model to generate a more realistic video based on the reference image and the image sequence. For example, the image sequence can contain motion trajectory information or deformation information of physical devices such as robots. This enables the trained video generation model to generate a more realistic video and avoids situations such as broken or distorted robotic arms / foreground objects caused by random generation. The generated video is more realistic and of higher quality, and can be used as a training sample to further train the policy, thus having training value.

[0121] In one alternative embodiment, based on Figure 5 The illustrated embodiment, as Figure 6 As shown, Figure 6 An example system architecture for training the video generation model in this alternative embodiment is shown. In this alternative embodiment, the video generation model is a Stable Video Diffusion (SVD) model, which includes a spatio-temporal attention mechanism, or more specifically, SVD employs a spatio-temporal block attention mechanism. For example, as... Figure 6 As shown, SVD includes a spatial attention layer and a temporal attention layer, which can be alternated. In this exemplary system architecture, an encoder and a decoder are also configured.Figure 6 In the diagram, Enc represents the encoder, and Dec represents the decoder. For example, the encoder can be a variational autoencoder (VAE), and the decoder can be a VAE decoder.

[0122] Specifically, a reference image containing initial background and robot appearance information can be used as the first frame of the condition. Subsequent frames (i.e., the image sequence) are generated by pixel-wise addition of the robot's motion observations in the simulated environment (a sequence of robot segmentation images) with the segmentation mask of the foreground object. For example, Figure 6 In the reference image shown, the robot's robotic arm is attempting to pick up an apple from a table; the robotic arm is the physical device, and the apple is the foreground object. Figure 6 In the process, the robot-segmented image and the mask image of the foreground object segmentation image are overlaid pixel by pixel to obtain an image sequence. Figure 6 The multiple small squares following the reference image represent the image sequence. After the reference image and the image sequence are concatenated in the temporal dimension, they are encoded into a latent representation by an encoder, and then concatenated with random noise in the channel dimension. This latent representation is then input into the VisAligner model for denoising. The target video can be obtained by decoding the denoised latent representation.

[0123] It should be noted that during the training phase of the video generation model, the image sequence corresponding to the reference image is obtained by segmenting and masking multiple frames of images from the sample video. For example, the image sequence corresponding to the reference image can be obtained in the following way:

[0124] The sample video is segmented into multiple frames following the reference image to obtain segmented images including foreground objects (e.g., ...). Figure 6 The image consists of multiple frames of a first segmented image (shown as an apple) and multiple frames of a second segmented image (robot segmentation image) including the physical device. Then, the multiple frames of the first segmented image are masked to obtain a multiple frame of the first masked image corresponding to the first segmented image. Finally, the multiple frame of the first masked image and the multiple frame of the second segmented image are pixel-wise superimposed to obtain the image sequence corresponding to the reference image.

[0125] After training the video generation model, during the inference (or application) phase, the robot segmentation image can come from one interface of the simulator, while the foreground object segmentation image comes from another interface of the simulator or from the observation dataset. In other words, the segmentation images of the physical device (e.g., the robot) and the segmentation images of the foreground objects come from different data sources. These two segmentation images need to be merged to obtain an image sequence. The foreground object segmentation image can be masked before being merged with the robot segmentation image.

[0126] The reference image and image sequence are concatenated in the time dimension and then input into the encoder. Random noise is then superimposed in the channel dimension and input into SVD. After denoising by SVD, the video is decoded (Dec) to obtain the generated video.

[0127] exist Figure 7 In the illustrated embodiment, the video generation model is implemented using Spatiotemporal Attention (SVD). SVD employs a spatiotemporal attention mechanism. By training SVD, it can capture long-distance dependencies in both spatial and temporal dimensions. For example, it can learn dependencies between adjacent or separated pixels in the spatial dimension and dependencies between pixels in different frames in the temporal dimension, thereby improving the consistency and visual quality of the generated video. The image sequence is obtained by pixel-stacked foreground object segmentation images and robot segmentation images to adapt to the simulation environment. In the simulation environment, the mask images for the robot segmentation image sequence and the foreground object segmentation image come from different data sources. Therefore, during the training phase, to better simulate the simulation environment, the images in the sample video are segmented into foreground object segmentation images and robot segmentation images before pixel-stacked to obtain the image sequence. This allows for the training of a video generation model more suitable for application in simulation environments.

[0128] In an optional embodiment, since this disclosure adds conditions for video generation—that is, in addition to the first condition for video generation, a reference image, an image sequence is also used as a second condition for video generation—the SVD needs to be expanded in channels. For example, the original 4 channels need to be expanded to 8 channels. After expansion, the reference image and the image sequence can be used together as conditions for video generation, and input to the encoder through multiple channels. The encoder outputs a first feature image sequence. The first feature image sequence includes the reference image and the encoded images corresponding to the image sequence. Random noise is superimposed on the first feature image sequence through multiple channels, input to the video generation model, and outputs a denoised second feature image sequence. The second feature image sequence is input to the decoder, and the decoder obtains the generated video.

[0129] like Figure 7 As shown, in this optional embodiment, the expanded multiple channels may include at least one first channel and at least one second channel, that is, SVD includes at least one first channel and at least one second channel for input conditions. The reference image is used as a first condition for generating the video and is input to the video diffusion model through at least one first channel; the reference image and the image sequence are stitched together in the time dimension as a second condition for generating the video and are input to the video diffusion model through at least one second channel. The input to the video diffusion model can be an encoder configured before being input to the video diffusion model.

[0130] Figure 8 The example only shows one first channel and one second channel. In actual applications, there can be multiple first channels and two second channels, for example, four first channels and four second channels.

[0131] An exemplary embodiment of this disclosure also provides a video generation method, wherein the video generation model can generate high-quality videos based on simulated videos (image sequences) output by a simulator, and the generated videos can be used as training samples to train a policy.

[0132] Figure 3 This is a flowchart illustrating a video generation method provided in an exemplary embodiment of the present disclosure. The method may include the following steps:

[0133] Step 401: Determine the reference image.

[0134] The reference image can be determined from a sample video, such as selecting a frame from a sample video in the observation dataset. Alternatively, it can be the last frame of the video generated by the previous video generation model.

[0135] Step 402: Input the reference image into the policy model, and the policy model outputs the action sequence.

[0136] The strategy model is used to predict the subsequent actions of entities, such as robots, that appear in the current reference image. The multiple predicted actions arranged in chronological order constitute the action sequence.

[0137] Step 403: Input the action sequence into the target simulator, and the target simulator outputs the image sequence corresponding to the reference image.

[0138] The target simulator adopts the above-mentioned method based on... Figure 5 The simulator trained by the training method of any of the illustrated embodiments uses calibrated physical parameters in the target simulator to generate a simulation video of a robot or other physical device in a simulation environment based on the action sequence output by the policy. The simulation video includes at least a sequence of segmented images of the physical device, for example, at least a sequence of segmented images of the robot.

[0139] Step 404: The image sequence and reference image are used as conditions for generating the video and input into the target video generation model. The target video is then output by the target video generation model.

[0140] The target video is used to train the policy, and the target video generation model adopts a method based on... Figure 8 The method of any of the embodiments described herein is used for training.

[0141] The inventors also discovered that variations in the texture of foreground objects and the background lead to a significant decrease in policy performance, indicating that the policy is sensitive to changes in visual appearance. To address this issue, embodiments of this disclosure propose generating augmented training data with modified object appearance and environmental conditions, eliminating the need for physical scene configuration or real-world data acquisition, and then using the augmented training data to train the policy model.

[0142] In one alternative embodiment, based on Figure 9 The illustrated embodiments, such as Figure 9 As shown, the reference image can be one of the following: the original image determined from the sample video, the foreground image obtained by performing foreground transformation on the original image, or the background image obtained by performing background transformation on the original image.

[0143] For example, tools with textured images can be used to modify the reference image (original image) in each sample of the observation dataset, altering the foreground and background content, such as randomly placing an A4 sheet of white paper on a table or changing the color of a cup. The resulting different reference images can correspond to a consistent image sequence; that is, only the foreground or background of the reference image needs modification, without requiring modification to each frame in subsequent image sequences. Subsequently, videos are generated under these modified conditions, and these simulated clips are ultimately converted into realistic visual demonstrations. Sim2Real is implemented through a video generation model to generate a set of high-quality, diverse demonstration clips. These numerous demonstration clips are then used as augmentation data to train the policy.

[0144] Specifically, in step 402, the reference image is input into the policy model, which can be the original image being input into the policy model, for example... Figure 3 The Robot Policy shown is the robot's strategy model. The strategy model outputs a sequence of actions, which is then input into the simulator. The simulator uses the aforementioned... Figure 9 The calibrated physical parameters shown in the embodiment are simulated in a simulation environment based on the input action sequence, outputting a simulation video. The simulation video includes at least segmented images of the physical device (e.g., robot segmentation images). Based on the simulation video output by the simulator, the image sequence corresponding to the original image is concatenated with the original image and its corresponding image sequence in the time dimension, and then input into the visual calibration model. The visual calibration model outputs the video corresponding to the original image. Similarly, the foreground image and background image are input into the policy model respectively to obtain the video corresponding to the foreground image and the video corresponding to the background image, respectively. In this way, three sets of videos can be generated based on a single reference image.

[0145] That is, the original image, foreground image, and background image are input into the policy model, and the model outputs the first target video corresponding to the original image, the second target video corresponding to the foreground image, and the third target video corresponding to the background image, respectively. The second and third target videos are the augmented training data obtained by modifying the appearance of the object and the environmental conditions.

[0146] For example, the policy takes a reference image as input, generates an action sequence of length T, and feeds this action sequence into a simulator to obtain a corresponding simulated video. Then, using the simulated video as conditions, a high-quality video is generated and output through a visual calibration model (i.e., the trained video generation model). Optionally, the last frame of the video can be used as a new reference image for the next iteration, and this process repeats to generate new actions and their corresponding simulated and real videos. This loop continues for multiple cycles before termination, thereby generating new trajectories with realistic appearances. The policy is trained using multiple new trajectories through reinforcement learning or imitation learning.

[0147] Figure 9 The illustrated embodiment demonstrates that by performing foreground and background transformations on the original image, three reference images can be generated based on one reference image, resulting in three sets of corresponding videos. This allows for the generation of as many realistic sets of sample videos as possible as training material for the policy, even when the sample videos in the observation dataset are limited. This not only enhances the training data but also enables the policy to learn coping strategies under different foreground or background conditions, thus addressing the performance degradation caused by changes in background texture or foreground.

[0148] It should be noted that, Figure 6 The "C" in the image indicates that the reference image and the image sequence are stitched together in the time dimension. The stitching can be done according to the frame order. Figure 7 and Figure 9 The "C" in the image indicates that the image and random noise are spliced ​​together in the channel dimension.

[0149] also, Figure 9 In the illustrated embodiment, the encoder and decoder are not shown. Figure 9 The illustrated embodiment may include an encoder and a decoder. Figure 9 The document simultaneously displays three reference images: the original image, the image obtained from foreground transformation, and the image obtained from background transformation. In reality, only one of these images is input when generating the video; it should not be interpreted as inputting all three images at the same time. Figure 9 After stitching the reference image and the image sequence (the sequence obtained by pixel overlay of the robot segmentation image and the foreground segmentation image) in the time dimension, random noise can be further superimposed in the channel dimension. Due to the limitations of view length, Figure 9The process of superimposing random noise is not shown, and should not be construed as meaning that the process cannot superimpose random noise, nor should it be construed as meaning that the architecture cannot contain encoders and decoders.

[0150] It should also be noted that the method proposed in this disclosure supports training the policy model based on reinforcement learning (RL) or IL (imitation learning (IL). When training the policy model using RL, in the application stage of the video generation model, including segmented images of entity devices, for example... Figure 9 The robot segmentation image shown is output from one interface of the simulator, while the foreground segmentation image (or foreground object segmentation image) comes from another interface of the simulator. The two images need to be merged to obtain the image sequence. When training the policy model based on IL, the robot segmentation image can come from the simulator, while the foreground segmentation image comes from the observation dataset.

[0151] in, Exemplary Apparatus The diagram shows Traj1 and TrajM, representing the M trajectories generated by the Policy through Regression Learning (RL). These M trajectories are then scored, for example, based on preferences, such as trajectory length. Shorter trajectories receive higher scores, and the scores are sorted from highest to lowest. During RL training, two trajectories can be selected from the M trajectories, one with a high score and the other with a low score. The difference between these two trajectories is used to obtain gradient information, which is then used to update the Policy. This allows the Policy to learn how to generate trajectories that are biased towards higher scores.

[0152] When training a policy using IL, a new training dataset can also be generated by changing the foreground or background.

[0153] The computer language used to implement the above scheme can be one or more of Python, C, C++, and CUDA.

[0154] It should be further noted that directly training a policy using reinforcement learning in a simulation environment requires manually building the simulation environment, which is costly, introduces a Sim2Real gap, and reduces the success rate of the trained policy when transferred to the real environment. On the other hand, manually collecting multiple data points and using imitation learning to train the policy is labor-intensive and inefficient. The solution proposed in this disclosure uses a differentiable process to quickly estimate the physical parameters of the simulation environment, making the simulation system and the real system behave similarly. Furthermore, a diffusion model is used to achieve simulation-to-real transfer, obtaining the robot trajectory and corresponding real-world observation video.

[0155] Any of the training methods and video generation methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any of the training methods and video generation methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the training methods and video generation methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0156] Figure 10

[0157] Figure 10 This is a schematic diagram of one embodiment of the training apparatus for training simulators disclosed herein. The apparatus of this embodiment can be used to implement the corresponding training method embodiments of this disclosure. Figure 11 The device shown includes: a first acquisition module 1000, a first training module 1001, and a second training module 1002.

[0158] The first acquisition module 1000 can input input data to a simulator, and the simulator outputs a first predicted value of the agent's state at time step t. The input data includes sampled physical parameters and observed values ​​of the agent's actions and states at time step t-1. The input data is observation data obtained from observing the physical device, including the agent, in a real environment. The physical parameters include at least one of the friction coefficient, stiffness, or damping coefficient. t ≥ 2, and t is an integer.

[0159] The first training module 1001 can input input data into the neural network to be trained, and output a second predicted value of the state of the agent at time step t through the neural network; calculate the value of the first loss function based on the second predicted value and the first predicted value corresponding to the state of the agent at time step t; update the trainable parameters in the neural network based on the value of the first loss function until the neural network converges to obtain the target neural network.

[0160] The second training module 1002 can input input data into the target neural network while keeping the trainable parameters in the target neural network unchanged, and output the third predicted value of the agent's state at time step t. Based on the third predicted value and the observed value corresponding to the agent's state at time step t, the value of the second loss function is calculated. The observed value corresponding to the agent's state at time step t is the observation data obtained by observing the physical device including the agent in the real environment. The gradient is calculated based on the value of the second loss function. The sampled physical parameters are updated based on the gradient until the second loss function converges to obtain the target physical parameters. The target physical parameters are configured into the simulator to obtain the target simulator.

[0161] Figure 11 This is a schematic diagram illustrating the structure of one embodiment of the training apparatus for training a video generation model according to the present disclosure. The apparatus of this embodiment can be used to implement the corresponding training method embodiments of the present disclosure. Figure 12 The device shown includes: a first determining module 2000, a second acquiring module 2001, a first output module 2002, and a first updating module 2003.

[0162] The first determining module 2000 can determine a reference image from a sample video. The sample video is captured by filming a physical device moving in a real environment.

[0163] The second acquisition module 2001 can acquire the image sequence corresponding to the reference image. The image sequence is obtained based on multiple frames of images following the reference image in the sample video.

[0164] The first output module 2002 can take the reference image and image sequence as conditions for generating the video, input them into the video generation model to be trained, and obtain the first video through the video generation model.

[0165] The first update module 2003 can calculate the value of the third loss function based on the first video and the sample video; and update the trainable parameters in the video generation model based on the value of the third loss function until the video generation model converges to obtain the target video generation model.

[0166] Figure 12 This is a schematic diagram illustrating the structure of one embodiment of the video generation apparatus of this disclosure. The apparatus of this embodiment can be used to implement the corresponding video generation method embodiments of this disclosure. Figure 3 The device shown includes: a second determining module 3000, a calling module 3001, and a second output module 3002.

[0167] The second determining module 3000 can be used to determine the reference image.

[0168] Module 3001 can input a reference image into a policy model, which outputs an action sequence; and input the action sequence into a target simulator, which outputs an image sequence corresponding to the reference image. The target simulator is used to predict the actions of the entity devices in the reference image. The target simulator employs a method such as... Figure 5 The method of any of the embodiments shown is used for training.

[0169] The second output module 3002 is used to input the image sequence and reference image as conditions for video generation into the target video generation model, and output the target video after passing through the target video generation model; the target video is used to train the policy model, and the target video generation model adopts a strategy based on... Exemplary Electronic Device The method of any of the embodiments shown is used for training.

[0170] Figure 13

[0171] Below, for reference Figure 13 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0172] Figure 13 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0173] like Figure 13 As shown, the electronic device includes one or more processors and memory.

[0174] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0175] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the training methods, video generation methods, and / or other desired functions of the various embodiments of this disclosure described above.

[0176] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0177] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0178] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0179] Of course, for the sake of simplicity, ​Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0180] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training methods and video generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0181] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0182] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the training methods and video generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0183] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0184] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0185] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0186] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0187] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0188] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0189] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0190] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A training method, characterized in that, The method includes: The input data is fed into the simulator, which then outputs a first predicted value of the agent's state at time step t. The input data includes sampled physical parameters and observed values ​​of the agent's actions and states at time step t-1. The input data is observational data obtained from observing a physical device, including the agent, in a real environment. The physical parameters include at least one of friction coefficient, stiffness, or damping coefficient; t ≥ 2, and t is an integer. The input data is fed into the neural network to be trained, and the neural network outputs a second predicted value of the state of the agent at time step t. Calculate the value of the first loss function based on the second predicted value and the first predicted value corresponding to the state of the agent at time step t; Based on the value of the first loss function, the trainable parameters in the neural network are updated until the neural network converges, thus obtaining the target neural network. Keeping the trainable parameters in the target neural network unchanged, the input data is input into the target neural network, and the target neural network outputs the third predicted value of the agent's state at time step t. The value of the second loss function is calculated based on the third predicted value and the observed value corresponding to the state of the agent at time step t; the observed value corresponding to the state of the agent at time step t is the observation data obtained by observing the physical device including the agent in the real environment. Calculate the gradient based on the value of the second loss function; The sampled physical parameters are updated according to the gradient until the second loss function converges to obtain the target physical parameters. The target physical parameters are configured into the simulator to obtain the target simulator.

2. The method according to claim 1, characterized in that, The neural network is a multilayer perceptron or a neural network that includes an attention mechanism.

3. The method according to any one of claims 1-2, characterized in that, The first loss function and / or the second loss function are mean squared error loss function or cross-entropy loss function.

4. A video generation method, characterized in that, The method includes: Identify a reference image; The reference image is input into the policy model, and the policy model outputs an action sequence. The action sequence is input to the target simulator, which then outputs an image sequence corresponding to the reference image. The target simulator is used to predict the actions of the entity device in the reference image, and the target simulator is trained using the method described in any one of claims 1-3. The image sequence and the reference image are used as conditions for generating a video and input into the target video generation model. The target video generation model outputs a target video. The target video is used to train the policy model. The target video generation model is trained using the following method: a reference image is determined from a sample video; the sample video is captured by a physical device moving in a real environment; an image sequence corresponding to the reference image is obtained; the image sequence is obtained based on multiple frames of images following the reference image in the sample video; the reference image and the image sequence are used as conditions for generating a video and input into the video generation model to be trained. The first video is obtained through the video generation model; the value of a third loss function is calculated based on the first video and the sample video; the trainable parameters in the video generation model are updated based on the value of the third loss function until the video generation model converges, thus obtaining the target video generation model.

5. The method according to claim 4, characterized in that, Obtaining the image sequence corresponding to the reference image includes: Based on multiple frames following the reference image in the sample video, a segmented first multi-frame segmented image including the foreground object and a second multi-frame segmented image including the physical device are obtained. Obtain the multi-frame first mask image corresponding to the multi-frame first segmentation image; The first mask image and the second segmented image of multiple frames are superimposed pixel by pixel to obtain the image sequence corresponding to the reference image.

6. The method according to claim 4 or 5, characterized in that, The video generation model is a video diffusion model, which includes at least one first channel and at least one second channel for input conditions; The reference image and the image sequence are used as conditions for generating the video and input into the video generation model to be trained, including: The reference image is used as a first condition for generating the video and is input into the video diffusion model through the at least one first channel. The reference image and the image sequence are spliced ​​together in the time dimension as a second condition for generating the video, and then input into the video diffusion model through the at least one second channel.

7. The method according to claim 6, characterized in that, The video diffusion model includes a spatiotemporal attention mechanism.

8. The method according to claim 4, characterized in that, The reference image is one of the following: an original image determined based on a sample video, a foreground image obtained by performing foreground transformation on the original image, or a background image obtained by performing background transformation on the original image; Inputting the reference image into the policy model includes: The original image, the foreground image, and the background image are respectively input into the strategy model; Output target video, including: The first target video corresponding to the original image, the second target video corresponding to the foreground image, and the third target video corresponding to the background image are output respectively.

9. A training device, characterized in that, The device includes: The first acquisition module is used to input input data to the simulator, and the simulator outputs a first predicted value of the state of the agent at time step t; wherein, the input data includes sampled physical parameters and observed values ​​of the agent's actions and states at time step t-1; the input data is observation data obtained by observing the physical device including the agent in a real environment; the physical parameters include at least one of friction coefficient, stiffness or damping coefficient; t≥2, and t is an integer; The first training module is used to input the input data into the neural network to be trained, and output a second predicted value of the state of the agent at time step t through the neural network; calculate the value of a first loss function based on the second predicted value and the first predicted value corresponding to the state of the agent at time step t; update the trainable parameters in the neural network based on the value of the first loss function until the neural network converges to obtain the target neural network; The second training module is used to keep the trainable parameters in the target neural network unchanged; input the input data into the target neural network, and output the third predicted value of the agent's state at time step t; calculate the value of the second loss function based on the third predicted value and the observed value corresponding to the agent's state at time step t; the observed value corresponding to the agent's state at time step t is the observation data obtained by observing the entity device including the agent in the real environment; calculate the gradient based on the value of the second loss function; update the sampled physical parameters based on the gradient until the second loss function converges to obtain the target physical parameters; configure the target physical parameters into the simulator to obtain the target simulator.

10. A video generation apparatus, characterized in that, The device includes: The second determining module is used to determine the reference image; The module is configured to input the reference image into a policy model and output an action sequence through the policy model; and to input the action sequence into a target simulator and output an image sequence corresponding to the reference image through the target simulator; the target simulator is configured to predict the actions of the entity device in the reference image, and the target simulator is trained using the method described in any one of claims 1-3; The second output module is used to input the image sequence and the reference image as conditions for generating a video into a target video generation model, and output a target video through the target video generation model. The target video is used to train the policy model, which is trained using the following method: determining a reference image from a sample video; the sample video is captured by a physical device moving in a real environment; obtaining an image sequence corresponding to the reference image; the image sequence is obtained based on multiple frames of images after the reference image in the sample video; inputting the reference image and the image sequence as conditions for generating a video into a video generation model to be trained, and obtaining a first video through the video generation model; calculating the value of a third loss function based on the first video and the sample video; updating the trainable parameters in the video generation model based on the value of the third loss function until the video generation model converges, thus obtaining the target video generation model.

11. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-8.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

13. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

Images