Training method, video generation method and device, electronic equipment and program product

Abstract

The embodiment of the invention discloses a training method, a video generation method and device, electronic equipment and a program product, and the method comprises the steps: inputting input data into a simulator, and outputting a first prediction value; inputting the input data into a neural network to be trained, and outputting a second predicted value through the neural network; calculating a value of a first loss function according to the second predicted value and the first predicted value; according to the value of the first loss function, trainable parameters in the neural network are updated until the neural network converges, and a target neural network is obtained; keeping trainable parameters in the target neural network unchanged; inputting the input data into a target neural network, and outputting a third predicted value through the target neural network; calculating a value of a second loss function according to the third predicted value and the observed value, and calculating a gradient; according to the gradient, updating the physical parameter to obtain a target physical parameter; and configuring the target physical parameter into the simulator to obtain a target simulator.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology, in particular to a training method, a video generation method and device, an electronic device, and a program product. Background Art

[0002] A policy model refers to the rules that an intelligent agent, such as a robot, uses to select actions in a specific state. The policy is the core component that drives the robot. Its input is various sensory information, such as camera video and the positions of the robot's joints. Its output is the action the robot is supposed to perform.

[0003] Training policies requires a large amount of training data. However, collecting such data in the real world is generally expensive, time-consuming, and often limited in scalability and repeatability. Acquiring training data in a simulation environment can reduce costs, but this can also lead to other issues. For example, the quality of training data obtained in simulation can be low, differing from the quality of training data obtained from real-world observations, making it less valuable for training. Summary of the Invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a training method, a video generation method and apparatus, an electronic device, and a program product.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method is provided, comprising: inputting input data into a simulator, and outputting a first predicted value of the state of an agent at the t-th time step through the simulator; wherein the input data comprises sampled physical parameters and observation values ​​of the action and state of the agent at the t-1th time step; the input data is observation data obtained by observing a physical device including the agent in a real environment; the physical parameters comprise at least one of a friction coefficient, a stiffness or a damping coefficient; t≥2, and t is an integer; inputting the input data into a neural network to be trained, and outputting a second predicted value of the state of the agent at the t-th time step through the neural network; calculating the value of a first loss function according to the second predicted value and the first predicted value respectively corresponding to the state of the agent at the t-th time step; and calculating the value of a first loss function according to the first loss function. The value of the loss function is used to update the trainable parameters in the neural network until the neural network converges to obtain the target neural network; the trainable parameters in the target neural network are kept unchanged, the input data is input into the target neural network, and the target neural network outputs the third predicted value of the state of the agent at the t-th time step; the value of the second loss function is calculated according to the third predicted value and the observed value corresponding to the state of the agent at the t-th time step; the observed value corresponding to the state of the agent at the t-th time step is the observation data obtained by observing the physical device including the agent in the real environment; according to the value of the second loss function, the gradient is calculated; according to the gradient, the sampled physical parameters are updated until the second loss function converges to obtain the target physical parameters; the target physical parameters are configured in the simulator to obtain the target simulator.

[0006] According to a second aspect of an embodiment of the present disclosure, a training method is provided, which includes: determining a reference image from a sample video; the sample video is obtained by shooting a physical device moving in a real environment; obtaining an image sequence corresponding to the reference image; the image sequence is obtained based on multiple frames of images after the reference image in the sample video; the reference image and the image sequence are used as conditions for generating a video, and are input into a video generation model to be trained, and a first video is obtained through the video generation model; based on the first video and the sample video, a value of a third loss function is calculated; based on the value of the third loss function, the trainable parameters in the video generation model are updated until the video generation model converges to obtain a target video generation model.

[0007] According to the third aspect of the embodiments of the present disclosure, a video generation method is provided, which includes: determining a reference image; inputting the reference image into a strategy model, and outputting an action sequence through the strategy model; inputting the action sequence into a target simulator, and outputting an image sequence corresponding to the reference image through the target simulator; the target simulator is used to predict the action of the physical device in the reference image, and the target simulator is trained using the method of the first aspect; inputting the image sequence and the reference image as conditions for generating a video into a target video generation model, and outputting a target video through the target video generation model; the target video is used to train the strategy model, and the target video generation model is trained using the method of the second aspect.

[0008] Based on the training method of the first aspect provided by the above-mentioned embodiment of the present disclosure, a physical calibration model is set as a proxy model of the simulator. By training the proxy model, the proxy model learns the simulation process of the simulator. After the training of the proxy model is completed, the physical parameters obtained by random sampling are gradient updated by back propagation according to the trained physical calibration model to obtain calibrated physical parameters. The simulator uses the calibrated physical parameters for simulation prediction. In this way, by calibrating the physical parameters used in the simulator, the differences in physical dynamics caused by unreasonable physical parameter settings in the Real2Sim process can be reduced.

[0009] Based on the training method of the second aspect provided by the above-mentioned embodiment of the present disclosure, while inputting the reference image, the image sequence is input into the video generation model as an additional condition, so that the video generation model can generate a video that is more in line with expectations based on the reference image and the image sequence. For example, the image sequence can include motion trajectory information or deformation information of physical equipment such as robots, so that the trained video generation model can generate a video that is more in line with expectations, and avoids the breakage and distortion of the robotic arm / foreground object due to random generation. The generated video is more realistic and of higher quality, reducing the difference in appearance visual aspects during the Sim2Real process in the simulation environment.

[0010] Based on the training method of the third aspect provided by the above-mentioned embodiment of the present disclosure, the target simulator obtained by the first aspect is used for simulation, and an image sequence is output. The image sequence and the reference image are then used as conditions of the video generation model and input into the video generation model obtained by training in the second aspect. Based on the target simulator, the differences in physical dynamics caused by unreasonable physical parameter settings in the Real2Sim process are reduced. Based on the video generation model, the differences in appearance visual aspects in the Sim2Real process under the simulation environment are reduced, thereby reducing the Real2Sim2Real difference as a whole, and being able to obtain more realistic videos as training sample videos for the policy.

[0011] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0013] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 A schematic diagram of the system architecture of an embodiment of the present disclosure.

[0014] Figure 2 FIG. 1 is a schematic diagram showing an embodiment of a training method for training a simulator in the present disclosure, which is performed in three stages.

[0015] Figure 3 The present invention is a flowchart of an embodiment of a training method for training a simulator.

[0016] Figure 4 Schematic diagram of using MSE as a loss function in one embodiment of the training method for training a simulator disclosed herein.

[0017] Figure 5 The present invention is a flowchart of an embodiment of a training method for training a video generation model.

[0018] Figure 6 A schematic diagram of a system framework of an embodiment of the training method for training a video generation model disclosed herein.

[0019] Figure 7 This is a schematic diagram of inputting two conditions through multiple channels in one embodiment of the training method for training a video generation model disclosed herein.

[0020] Figure 8 The flowchart of one embodiment of the video generation method disclosed herein.

[0021] Figure 9 A schematic diagram of the system architecture of an embodiment of the video generation method disclosed herein.

[0022] Figure 10 The figure is a schematic structural diagram of an embodiment of a training device for training a simulator disclosed herein.

[0023] Figure 11 The figure is a structural diagram of an embodiment of a training device for training a video generation model disclosed in the present invention.

[0024] Figure 12 The figure is a structural diagram of an embodiment of the video generating device disclosed herein.

[0025] Figure 13 FIG. 1 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0027] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0028] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0029] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0030] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0031] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0032] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0033] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0034] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0035] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0036] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0037] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0038] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by the computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, and the like that perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked via a communications network. In distributed cloud computing environments, program modules can be located on local or remote computer system storage media, including storage devices.

[0039] Application Overview Robotics has made significant progress in recent years with the continuous development of artificial intelligence (AI) and embodied intelligence. Embodied intelligence refers to the ability of robots to autonomously perceive their environment, make decisions, and perform actions in the physical world.

[0040] The development of embodied intelligence technologies has driven significant progress in robot learning, enabling intelligent agents to perform complex manipulation, navigation, and decision-making tasks in real-world environments. Central to this progress is the availability of large-scale, high-quality observational data that captures the underlying relationships between the interactions between sensory inputs and movements of agents operating in diverse physical environments. For example, policies are the core component used to drive robots, and training them requires a large amount of observational data. However, collecting observational data in the real world is costly and time-consuming, and is often limited in scalability and repeatability.

[0041] Obtaining training data in a simulation environment has become a viable solution. For example, some open-source robotics learning benchmark frameworks in this field provide standardized testing environments for research in reinforcement learning, imitation learning, and multi-task learning. These testing environments can be used as practical alternatives for training and evaluating robotic policies.

[0042] During the implementation of this disclosure, the inventors discovered that simulation environments in related technologies suffer from Real-to-Sim-to-Real discrepancies. These discrepancies include differences in the transition from reality to simulation (Real-to-Sim) and from simulation to reality (Sim-to-Real). The two key sources of these discrepancies are differences in physical dynamics and visual appearance.

[0043] For example, differences in physical dynamics, perhaps due to overly simplistic assumptions or unmodeled parameters such as joint stiffness, actuator delays, or surface friction, often lead to physical behaviors simulated in a virtual environment that differ from real-world dynamics. These mismatches lead to inaccurate motion trajectories and undermine the reliability of simulation-based control.

[0044] On the other hand, the difference in visual appearance may be due to the fact that the rendered appearance of simulated scenes often lacks the visual fidelity required to generalize to real-world sensory input. Therefore, using training datasets lacking visual fidelity to train perception models such as policies may not be able to operate effectively in real environments.

[0045] In addition, in the related art, a video generation model can be used to generate videos for training policies. The motion trajectory of the robot in the generated video is random and uncontrollable, and may not conform to the expected motion, and the robotic arm or foreground objects may be broken or distorted. That is, the video quality generated by the video generation model in the related art is poor, and there is a significant difference in the quality of the sample videos obtained by shooting robots moving in the real world in the observation dataset.

[0046] These differences pose significant challenges to deploying policies trained in simulation, especially for tasks requiring precise physical interactions or high-quality visual perception. For example, training a robot policy model in simulation yields a high success rate, but the success rate plummets when deployed on a real machine.

[0047] Exemplary Systems The training method and video generation method provided by an exemplary embodiment of the present disclosure can be applied to Figure 1 The exemplary system architecture shown in Figure 1 includes a policy model (Policy), a simulator (Simulator), a physical calibration model (PhysAligner), and a visual calibration model (VisAligner).

[0048] The training method and video generation method provided by an exemplary embodiment of the present disclosure may specifically include a training method for training a physical calibration model, a training method for training a visual calibration model, and a training method based on Figure 1 The video generation method of the system architecture and the videos generated by the video generation method can be used as training samples to construct a training data set, which can be used to train a strategy model so that physical devices such as robots deployed with the strategy model can make more accurate action decisions in the real world based on the real-time objective environment.

[0049] Figure 1 In the illustrated system architecture, the physical calibration model is a trained model. This trained model is used to calibrate the physical parameters used in the simulator. This calibration process can also be understood as training the simulator using the physical calibration model. Furthermore, the visual calibration model is also a trained model. The training process aims to improve the quality of the videos it generates, for example, to produce more realistic videos that better align with the intended motion trajectory.

[0050] Specifically, a reference image can be input into the policy model. This reference image can be an image that partially or completely displays a physical device such as a robot. After the policy model outputs a motion sequence, the simulator uses calibrated physical parameters based on the action sequence output by the policy model to simulate physical behavior in the real world and output an image sequence containing information about the robot's motion trajectory. The visual calibration model generates a high-quality video based on the image sequence output by the simulator and the reference image. The generated video can be used as a training sample for training the policy model. Optionally, the last frame of the generated video, or a frame selected from the last multiple frames, is used as the next reference image. In this way, multiple high-quality videos can be generated to construct a training dataset for training the policy model.

[0051] Among them, the use of a physical calibration model to calibrate physical parameters can make up for the differences in physical dynamics from reality to simulation through the rationality of physical parameter settings; the trained visual calibration model can generate more realistic or higher-quality videos, reducing the differences between videos generated based on simulation environments and videos collected in the real world.

[0052] Figure 1 The policy model shown in can be applied to robots, wearable devices, smart home devices, or other electronic devices (physical devices) that support interaction with the environment.

[0053] Exemplary Methods An exemplary embodiment of the present disclosure provides a training method for training a physical calibration model. After obtaining the trained physical calibration model, the physical parameters are gradient-updated using the physical calibration model. The updated physical parameters are then configured in a simulator to obtain the trained simulator (target simulator). This training method can be applied to electronic devices.

[0054] like Figure 2 As shown, in some optional embodiments, the training process may include the following three stages: Phase 1: Dataset generation: In this phase, the simulator 100 generates a dataset for training the physical calibration model.

[0055] Phase 2, Surrogate Modeling: The physical calibration model 110 is trained using the dataset generated in Phase 1. The physical parameters used in both Phase 2 and Phase 1 are the physical parameters 101 to be calibrated, which can be obtained by random sampling in the real world.

[0056] Phase 3: Gradient-based Refinement: The trained physical calibration model 111 is used to perform gradient updates on the physical parameters to be calibrated 101, obtaining the calibrated physical parameters. The calibrated physical parameters are then configured in the simulator, resulting in a trained simulator.

[0057] Figure 2 The flame symbol in the figure indicates that the parameters of the object are being optimized in the current stage, and the snowflake symbol indicates that the trainable parameters of the object are kept unchanged (frozen) in the current stage. For example, the training target in stage two is physical calibration model 110, and the trainable parameters of physical calibration model 110 are continuously optimized during this stage. In stage three, the trainable parameters of physical calibration model 111 after training remain unchanged or are frozen. For clarity, different states of the same model are distinguished by different labels. Physical calibration model 110 is the model during training, and physical calibration model 111 is the model after training.

[0058] Specifically, Figure 3 This is a flow chart of a training method provided by an exemplary embodiment of the present disclosure, combined with Figure 2 and Figure 3 , this training method can include the following process: Step 201: Input data to the simulator, and output the first predicted value of the state of the agent at the tth time step through the simulator.

[0059] For example, Figure 2 As shown in stage 1, the input data includes the sampled physical parameters ( f,p,d ) and the observed value of the agent’s action at the t-1th time step a t-1 and the observed value of the agent's state at time step t-1 S t-1 , input data to the simulator 100, and the simulator 100 outputs the first predicted value of the state of the agent at the tth time step P t . t≥2, and t is an integer.

[0060] The state refers to the state of the agent relative to the environment. For example, the state may include the location of the agent in the current environment. The state output by the simulator represents the state of the agent relative to the simulated environment.

[0061] Physical parameters ( f,p,d ) includes at least one of a friction coefficient, a stiffness or a damping coefficient, wherein, f represents the coefficient of friction (friction), drepresents the damping coefficient (damping), p Indicates stiffness.

[0062] In the first stage, a large number of random samples ( f, p, d ) drives the simulator to generate a first prediction value of the state at the t-th time step through the simulator P t The first predicted value P t Indicates the simulator's response to each sample ( f, p, d ) is the predicted value obtained by predicting the true state at the t-th time step.

[0063] The input data is the observation data obtained by observing the physical equipment including the agent in the real environment. The observation value of the action of the agent at the t-1th time step is a t-1 and the observed value of the agent's state at time step t-1 S t-1 The physical parameters in the first stage can be obtained from the observation data set, which can be the initial values ​​of a large number of physical parameters obtained by random sampling in the real world.

[0064] An agent is a software system or entity that can perceive its environment, make decisions, and take actions autonomously. The physical device that includes an agent can be a robot, or the agent itself can represent a robot.

[0065] Combine Figure 2 and Figure 3 , stage one may include step 201 , stage two may include steps 202 - 204 , and stage three may include steps 205 - 208 .

[0066] Step 202: Input the input data to the neural network to be trained, and output the second predicted value of the state of the agent at the tth time step through the neural network. .

[0067] For example, Figure 2 In the second stage shown, the same input data as that input to the simulator in the first stage is input to the physical calibration model 110 (neural network to be trained), and the physical calibration model 110 outputs the second predicted value of the state of the agent at the tth time step, that is, the physical calibration model 110 outputs the second predicted value of the state of the agent at the tth time step according to the input ,predict .

[0068] The physical calibration model 110 (111) can be implemented using a neural network. The neural network to be trained is the physical calibration model to be trained, for example, it can be a physical calibration model waiting for training or in the training process, waiting for continued training.

[0069] It should be noted that the physical calibration model can be implemented based on a differentiable neural network, and the ability to differentiate can specifically be the ability to calculate gradients based on a loss function.

[0070] The physical calibration model 110 (the neural network to be trained) can be understood as a proxy model of the simulator 100. The reason for setting up the proxy model is that the simulator 100 cannot calculate the gradient. When calculating the gradient, the objective function is required to be differentiable. The simulator is not differentiable, and the gradient of the physical parameters cannot be directly updated based on the simulator.

[0071] Specifically, the gradient is calculated based on the loss function. The gradient is the vector of partial derivatives of the loss function with respect to the model parameters, representing the rate of change of the loss function in the parameter space. Specifically, the gradient is the partial derivative of the loss function with respect to the model parameters, reflecting the degree of influence of each trainable parameter in the model on the loss function during training. By calculating the gradient, we can determine the direction and step size of parameter adjustment, so that the parameters gradually approach the optimal solution.

[0072] In the disclosed embodiments, a differentiable proxy model is used to simulate the simulator. This model is trained to learn the simulator 100's simulation process, thereby resolving the simulator's inability to calculate gradients due to its non-differentiable nature. The trained proxy model is further used to gradient-update physical parameters, which are then configured in the simulator. The simulator then uses these updated physical parameters for simulation.

[0073] Step 203 , calculating the value of the first loss function according to the second predicted value and the first predicted value corresponding to the state of the agent at the t-th time step.

[0074] For example, Figure 2 As shown in Phase 2, and P t The value of is substituted into the first loss function to calculate the value of the first loss function.

[0075] As a proxy model of the simulator 100, the physical calibration model 110 needs to learn the simulation performance of the simulator. Therefore, it needs to use the output of the simulator in stage one as the learning label. By calculating the difference between the output of the physical calibration model 110 and the output of the simulator 100, the trainable parameters in the physical calibration model 110 are adjusted so that the prediction performance of the physical calibration model 110 is closer to the simulator.

[0076] The first loss function is the loss function used when training the physical calibration model.

[0077] Step 204: Update the trainable parameters in the neural network according to the value of the first loss function until the neural network converges to obtain the target neural network.

[0078] The target neural network is the obtained trained physical calibration model 111.

[0079] Step 205: Keep the trainable parameters in the target neural network unchanged, input the input data into the target neural network, and output the third predicted value of the state of the agent at the tth time step through the target neural network. .

[0080] It should be noted that in phases 2 and 3, since the model parameters may have changed, the output obtained under the same input data may be different, that is, As shown in Phase 3 Can represent different values.

[0081] like Figure 2 In the third stage shown, the physical parameters obtained by sampling ( f,p,d ) and the observed value of the agent’s action at the t-1th time step a t-1 and the observed value of the agent's state at time step t-1 S t-1 , is input to the target neural network 111 (i.e., the trained physical calibration model), and the target neural network 111 outputs the third predicted value of the t-th time step.

[0082] The training target in stage two is the physical calibration model 110. Unlike stage two, the training target in stage three is the physical parameters 102. In this stage three, the parameters in the target neural network 111 are frozen, and the physical parameters are gradient updated. The optimized physical parameters are configured in the simulator. The training process in stage three with physical parameters as the training target is also the training process with the simulator as the training target.

[0083] Step 206 , calculating the value of the second loss function according to the third predicted value and the observed value corresponding to the state of the agent at the t-th time step.

[0084] For example, Figure 2 The third stage shown is the third predicted value With the observed value S t Substitute into the second loss function and calculate the loss value of the second loss function.

[0085] Among them, the observation value corresponding to the state of the agent at the tth time step is S t is observational data obtained from observing physical devices, including intelligent agents, in a real environment (i.e., the real world). The goal of Phase 3 training is to adjust physical parameters so that the simulator can achieve simulation results that are closer to the real world. Therefore, observational data collected in the real world is used as labels.

[0086] Step 207: Calculate the gradient according to the value of the second loss function.

[0087] Calculate the gradient, that is, calculate the partial derivative vector of the value of the second loss function with respect to the physical parameter, which represents the rate of change of the second loss function in the physical parameter space. Specifically, the gradient is the partial derivative of the loss function with respect to the physical parameter, which reflects the degree of influence of each physical parameter on the second loss function in the current training.

[0088] Step 208: Update the sampled physical parameters according to the gradient until the second loss function converges to obtain the target physical parameters.

[0089] By calculating the gradient, the direction and step size of physical parameter adjustment can be determined, so that the physical parameters gradually approach the optimal solution.

[0090] Step 209: configure the target physical parameters into the simulator to obtain the target simulator.

[0091] The simulator that uses the updated physical parameters to perform simulation is the target simulator.

[0092] Figure 3 In the embodiment shown, a physical calibration model is set as a proxy model of the simulator. By training the proxy model, the proxy model learns the simulation process of the simulator. After the training of the proxy model is completed, the physical parameters obtained by random sampling are gradient updated by back propagation according to the trained physical calibration model to obtain calibrated physical parameters. The simulator uses the calibrated physical parameters for simulation prediction. In this way, by calibrating the physical parameters used in the simulator, the differences in physical dynamics caused by unreasonable physical parameter settings in the Real2Sim process can be reduced.

[0093] Among them, setting up the simulator's proxy model also solves the problem that the simulator cannot calculate gradients and does not support gradient updates for physical parameters.

[0094] In an alternative embodiment, based on Figure 3 Based on the embodiment shown, the neural network used to implement the physical calibration model can be a multilayer perceptron (MLP), for example Figure 2The physical calibration model 110 shown in FIG. 1 is an example of an MLP.

[0095] In the embodiment of using MLP to implement the physical calibration model, MLP is a lightweight model that can reduce the complexity of the model and has characteristics such as efficient calculation and simple structure. As a physical calibration model, MLP can provide good performance in a resource-constrained environment.

[0096] In an alternative embodiment, based on Figure 3 Based on the illustrated embodiment, the neural network used to implement the physical calibration model can be a neural network including an attention mechanism, for example, it can be a neural network based on a transformer architecture, for example, it can be a BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-trained Transformer) series model.

[0097] In an embodiment of implementing a physical calibration model using a neural network including an attention mechanism, the attention mechanism overcomes bottlenecks such as low sequence processing efficiency and difficulty in modeling long-distance dependencies.

[0098] In an alternative embodiment, based on Figure 3 Based on the embodiment shown, the first loss function and the second loss function can be the same loss function or different loss functions. Figure 4 As shown, the first loss function and the second loss function can be both mean squared error loss functions (MSE). Figure 4 The MSE Loss shown in represents the mean squared error loss function. In other embodiments, the first loss function and the second loss function may both be cross-entropy loss functions. Alternatively, in other embodiments, one loss function is the mean squared error loss function and the other is the cross-entropy loss function.

[0099] By calculating the square of the error, the MSE loss function can significantly amplify larger error values. This feature prompts the model to pay more attention to samples with larger prediction errors during training, thereby accelerating the convergence efficiency of the gradient descent process.

[0100] An exemplary embodiment of the present disclosure also provides a training method for training a video generation model. The trained video generation model can generate more realistic videos based on simulated videos (image sequences) output in a simulated environment. The generated videos can be used as training samples to train a policy. This training method can be applied to electronic devices.

[0101] Figure 5 FIG. 1 is a flow chart of a training method for training a video generation model provided by an exemplary embodiment of the present disclosure. The training method may include the following process: Step 301: Determine a reference image from a sample video.

[0102] A sample video is captured by capturing a physical device moving in a real environment (the real world), such as a robot. A sample video can be a video obtained from an observation dataset.

[0103] The reference image is determined from the sample video by randomly selecting a frame of image from the sample video as the reference image, or selecting a first frame or a frame of image from the first few frames as the reference image.

[0104] Step 302: Acquire an image sequence corresponding to the reference image.

[0105] The image sequence at least includes motion trajectory information of the physical device in multiple frames of images following the reference image.

[0106] During the training phase of a video generation model, an image sequence can be derived from multiple frames following a reference image frame in a sample video. For example, if the sample video is a video acquired from an observation dataset, and the first frame of the sample video is selected as the reference image, then the image sequence is the image sequence corresponding to the multiple frames following the first frame. For example, an image sequence can be derived by taking a frame at intervals of several frames from the multiple frames following the first frame.

[0107] In step 303 , the reference image and the image sequence are used as conditions for generating a video and input into a video generation model to be trained, and the first video is obtained through the video generation model.

[0108] The reference image determined in step 301 is used as one condition for generating a video, and the image sequence obtained in step 302 is used as another condition for generating a video. Both are input into the video generation model to be trained, and the first video is output.

[0109] Step 304: Calculate the value of a third loss function based on the first video and the sample video.

[0110] Using the sample video as a label, the gap between the first video output by video generation and the sample video is calculated, and then the trainable parameters in the video generation model are adjusted according to the gap, so that the video generation model can generate more realistic videos through learning.

[0111] Step 305: Update the trainable parameters in the video generation model according to the value of the third loss function until the video generation model converges to obtain the target video generation model.

[0112] The target video generation model, i.e. Figure 1 The video calibration model shown in .

[0113] In the related art, a common method for obtaining a generated video through a video generation model is to input only one reference image into the video generation model, and the video generation model generates a video based on the reference image. Due to the limited information in the reference image, the video output by the video generation model may have motion that does not conform to the expected situation, or the robot arm / foreground object may be broken and distorted. The generated video quality is not high and it is difficult to use it as a sample video for training policy. Figure 5 In the embodiment shown, while inputting the reference image, the image sequence is input into the video generation model as an additional condition, so that the video generation model can generate a video that is more in line with the expectations based on the reference image and the image sequence. For example, the image sequence can include motion trajectory information or deformation information of physical equipment such as robots, so that the trained video generation model can generate a video that is more in line with the expectations, and avoid the breakage and distortion of the robotic arm / foreground object due to random generation. The generated video is more realistic and of higher quality. The generated video can be used as a training sample to further train the policy and has training value.

[0114] In an alternative embodiment, based on Figure 5 The embodiment shown, as Figure 6 As shown, Figure 6 An example of a system architecture for training a video generation model in this optional embodiment is shown. In this optional embodiment, the video generation model is a Stable Video Diffusion (SVD) model. SVD includes a spatio-temporal attention mechanism, or in other words, SVD uses a spatio-temporal block attention mechanism. For example, Figure 6 As shown in the figure, SVD includes a spatial attention layer and a temporal attention layer, which can be set alternately. In this exemplary system architecture, an encoder and a decoder are also configured. Figure 6 The Enc shown in is the encoder, and the Dec is the decoder. For example, the encoder can be a variational autoencoder (VAE), and the decoder can be a VAE decoder.

[0115] Specifically, a reference image containing the initial background and robot appearance information can be used as the first frame of the condition. Subsequent frames (i.e., image sequences) are generated by pixel-wise addition of the robot's motion observations in the simulated environment (sequence of robot segmentation images) and the segmentation mask of the foreground object. For example, Figure 6 In the reference image shown, the robot's arm is trying to pick up an apple on the table, where the robot's arm is the physical device and the apple is the foreground object. Figure 6 In the image sequence, the robot segmentation image and the mask image of the foreground object segmentation image are superimposed together to obtain the image sequence ( Figure 6 The multiple small squares after the reference image in the image represent the image sequence. After the reference image and the image sequence are concatenated in the temporal dimension, they are encoded into a latent representation by the encoder. This is then concatenated with random noise in the channel dimension and input into the visual calibration model (VisAligner) for denoising. The target video is then obtained by decoding the denoised latent representation.

[0116] It should be noted that during the training phase of the video generation model, the image sequence corresponding to the reference image is obtained by segmenting and masking multiple frames in the sample video. For example, the image sequence corresponding to the reference image can be obtained as follows: Segment the multiple frames of images after the reference image in the sample video to obtain the segmented images including the foreground objects (such as Figure 6 The first segmented images (the apple shown in the figure) and the second segmented images (the robot segmented images) of the physical device are then masked to obtain a plurality of first masked images corresponding to the first segmented images. The first masked images and the second segmented images are pixel-wise superimposed to obtain an image sequence corresponding to the reference image.

[0117] After training the video generation model, during the inference phase (or application phase) of the video generation model, the robot segmentation image can come from one simulator interface, while the foreground object segmentation image can come from another simulator interface or from an observation dataset. In other words, the segmentation image of the physical device (e.g., robot) and the segmentation image of the foreground object come from different data sources. These two segmentation images need to be merged to obtain an image sequence. The foreground object segmentation image can be masked and then merged with the robot segmentation image.

[0118] The reference image and the image sequence are spliced ​​in the time dimension and input into the encoder. After that, random noise is superimposed in the channel dimension and then input into SVD. After SVD denoising and decoding (Dec), the generated video is obtained.

[0119] exist Figure 6 In the illustrated embodiment, the video generation model is implemented using SVD, which uses a spatiotemporal attention mechanism. Training the SVD allows it to capture long-range dependencies in both spatial and temporal dimensions. For example, it can learn the dependencies between pixels that are adjacent or several pixels apart in the spatial dimension, as well as the dependencies between pixels in different frames in the temporal dimension, thereby improving the consistency and visual quality of the generated video. The image sequence is obtained by pixel-wise superposition of foreground object segmentation images and robot segmentation images to adapt to the simulation environment. In the simulation environment, the mask images of the robot segmentation image sequence and the foreground object segmentation image sequence come from different data sources. Therefore, during the training phase, to better match the simulation environment, the images in the sample video are segmented into foreground object segmentation images and robot segmentation images, and then pixel-wise superposition is performed to obtain the image sequence. This allows for training a video generation model that is more suitable for application in a simulation environment.

[0120] In an optional embodiment, since the disclosed embodiment adds a condition for video generation, that is, not only the first condition for video generation - the reference image - is input, but also the image sequence is used as the second condition for video generation, it is necessary to perform channel expansion on the SVD, for example, the original 4 channels need to be expanded to 8 channels. After the expansion, the reference image and the image sequence can be used together as the condition for video generation, and input to the encoder through multiple channels, and the encoder outputs a first feature image sequence. The first feature image sequence includes the encoded images corresponding to the reference image and the image sequence respectively; random noise is superimposed on the first feature image sequence in multiple channels, input into the video generation model, and the video generation model outputs the denoised second feature image sequence, which is input into the decoder, and the decoder obtains the generated video.

[0121] like Figure 7 As shown, in this optional embodiment, the expanded multiple channels may include at least one first channel and at least one second channel, that is, the SVD includes at least one first channel and at least one second channel for input conditions. A reference image is used as the first condition for generating a video and is input into the video diffusion model via the at least one first channel. The reference image and the image sequence are spliced ​​temporally as the second condition for generating a video and are input into the video diffusion model via the at least one second channel. The input to the video diffusion model may be an encoder configured prior to inputting the video diffusion model.

[0122] Figure 7 Only one first channel and one second channel are shown as examples. In actual applications, the number of first channels and second channels can be multiple, for example, four first channels and four second channels are set.

[0123] An exemplary embodiment of the present disclosure further provides a video generation method, wherein the video generation model can generate high-quality videos based on the simulated videos (image sequences) output by the simulator. The generated videos can be used as training samples to train the policy.

[0124] Figure 8 FIG. 1 is a flow chart of a video generation method provided by an exemplary embodiment of the present disclosure. The method may include the following process: Step 401: Determine a reference image.

[0125] The reference image can be determined from a sample video, for example, by selecting a frame from a sample video in an observation dataset as the reference image. Alternatively, the last frame of a video generated by the previous round of the video generation model can be used as the reference image.

[0126] In step 402, a reference image is input into a policy model, and an action sequence is outputted by the policy model.

[0127] The policy model is used to predict the subsequent actions of the physical device, such as a robot, that appears in the reference image based on the current reference image. The predicted multiple actions are arranged in chronological order as an action sequence.

[0128] Step 403: input the action sequence into the target simulator, and the target simulator outputs an image sequence corresponding to the reference image.

[0129] The target simulator uses the Figure 3 In a simulator trained using the training method of any of the illustrated embodiments, the target simulator uses calibrated physical parameters and generates a simulated video of a physical device, such as a robot, in a simulated environment based on the action sequence output by the policy. The simulated video includes at least a sequence of segmented images of the physical device, such as at least a sequence of segmented images of a robot.

[0130] In step 404 , the image sequence and the reference image are input into a target video generation model as conditions for generating a video, and the target video is outputted through the target video generation model.

[0131] The target video is used to train the policy, and the target video generation model is based on Figure 5 The method of any embodiment of the embodiments is trained.

[0132] The inventors also discovered that texture variations in foreground objects and backgrounds can significantly degrade policy performance, demonstrating a degree of sensitivity to changes in visual appearance. To address this issue, the disclosed embodiments generate augmented training data with modified object appearance and environmental conditions, eliminating the need for physical scene configuration or real-world data collection. The augmented training data is then used to train the policy model.

[0133] In an alternative embodiment, based on Figure 8 The embodiment shown, as Figure 9 As shown, the reference image may be one of an original image determined from a sample video, a foreground image obtained by performing a foreground transformation on the original image, or a background image obtained by performing a background transformation on the original image.

[0134] For example, tools with native images can be used to modify the reference image (original image) in each sample in the observation dataset, changing the foreground and background content. For example, randomly placing A4 white paper on a table or changing the color of a cup. The image sequences corresponding to the different modified reference images can be consistent. In other words, only the foreground or background of the reference image needs to be modified, and there is no need to modify every frame in the subsequent image sequence. Subsequently, videos are generated under these modified conditions, and ultimately these simulated clips are converted into realistic visual demonstrations. Sim2Real is implemented through a video generation model to generate a set of high-quality and diverse demonstration clips. These large number of demonstration clips are then used as augmented data to train the policy.

[0135] Specifically, in step 402, the reference image is input into the policy model, which may be inputting the original image into the policy model, for example Figure 9 The Robot Policy shown in the figure is the robot's policy model. The policy model outputs a set of action sequences, which are input into the simulator. The simulator uses the above-mentioned Figure 3 The calibrated physical parameters of the illustrated embodiment are simulated in a simulation environment based on an input action sequence, outputting a simulated video that includes at least a segmented image of the physical device (e.g., a segmented image of a robot). Based on the simulated video output by the simulator, the image sequence corresponding to the original image is spliced ​​temporally with its corresponding image sequence and then input into a visual calibration model, which then outputs a video corresponding to the original image. Similarly, the foreground image and background image are input into the policy model, respectively, to produce a video corresponding to the foreground image and a video corresponding to the background image. In this way, three sets of videos can be generated based on a single reference image.

[0136] Specifically, the original image, foreground image, and background image are fed into the policy model, which then outputs a first target video corresponding to the original image, a second target video corresponding to the foreground image, and a third target video corresponding to the background image. The second and third target videos serve as enhanced training data obtained by modifying the object's appearance and environmental conditions.

[0137] For example, a policy takes a reference image as input and generates an action sequence of length T. This action sequence is fed into a simulator, resulting in a corresponding simulated video. The simulated video is then conditioned on the visual calibration model (i.e., the trained video generation model) to output a high-quality generated video. Optionally, the last frame of the video can be used as the new reference image for the next iteration, and the process is repeated to generate new actions and their corresponding simulated and real videos. This cycle continues for multiple cycles before terminating, generating new trajectories with realistic appearance. The policy is trained using these multiple new trajectories using reinforcement learning or imitation learning.

[0138] Figure 9 In the embodiment shown, by performing foreground transformation and background transformation on the original image, three reference images can be generated based on one reference image, and then three corresponding groups of videos can be obtained. This can achieve the situation where the sample videos in the observation data set are limited, and as many groups of relatively realistic sample videos as possible can be generated as training materials for the policy. While obtaining enhanced training data, the policy can also learn response strategies when the foreground or background changes, solving the problem of performance degradation of the policy due to changes in background texture or foreground.

[0139] It should be noted that Figure 9 The “C” in represents the splicing of the reference image and the image sequence in the time dimension, and the splicing can be performed in frame order. Figure 6 and Figure 7 The “C” in represents the concatenation of the image and random noise in the channel dimension.

[0140] also, Figure 9 In the illustrated embodiment, the encoder and decoder are not shown. Figure 9 The illustrated embodiment may include an encoder and a decoder. Figure 9 Three reference images are shown at the same time: the original image, the image obtained by foreground transformation, and the image obtained by background transformation. In fact, when generating a video, only one of the images is input, and it cannot be understood that three images are input at the same time. Figure 9 After the reference image and the image sequence (the sequence obtained by pixel superposition of the robot segmentation image and the foreground segmentation image) are spliced ​​in the time dimension, random noise can be superimposed in the channel dimension. However, due to the length of the view, Figure 9The process of superimposing random noise is not shown, which should not be understood to mean that random noise cannot be superimposed in this process, nor should it be understood to mean that the architecture cannot include an encoder and a decoder.

[0141] It should also be noted that the method proposed in the embodiment of the present disclosure supports the training of the policy model based on reinforcement learning (RL) or IL (Imitation learning, IL). When using RL to train the policy model, in the application stage of the video generation model, including the segmented image of the physical device, such as Figure 9 The robot segmentation image shown in the figure is output from one simulator interface, while the foreground segmentation image (also called foreground object segmentation image) comes from another simulator interface. These two images need to be merged to form an image sequence. When training a policy model based on IL, the robot segmentation image can come from the simulator, while the foreground segmentation image comes from the observation dataset.

[0142] in, Figure 9 Traj1 and TrajM in the figure represent M trajectories generated by the policy through RL. These trajectories are then scored, for example, based on a preference for trajectory length. Shorter trajectories receive higher scores, with scores arranged from high to low. During RL training, two trajectories can be extracted from the M trajectories, one with a high score and one with a low score. The difference between these two trajectories is then used to obtain gradient information, which is then used to update the policy in reverse order. This allows the policy to learn how to generate trajectories that tend to have higher scores.

[0143] When using IL training policy, new training data sets can also be generated by transforming the foreground or background.

[0144] The computer language used to implement the above solution can be one or more of Python, C, C++, and CUDA.

[0145] It should be further noted that directly using reinforcement learning to train policies in a simulation environment requires manual construction of the simulation environment, which is costly and results in a Sim2Real gap. Furthermore, the success rate decreases after the trained policy is transferred to the real environment. Manually collecting multiple data points and using imitation learning to train policies also carries high labor costs and low efficiency. The above-mentioned solution proposed in the disclosed embodiments uses a differentiable process to rapidly estimate the physical parameters of the simulation environment, ensuring that the performance of the simulation system and the real system are similar. Furthermore, a diffusion model is used to achieve simulation-to-real migration, resulting in robot trajectories and corresponding real-world observation videos.

[0146] Any training method or video generation method provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server. Alternatively, any training method or video generation method provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any training method or video generation method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be further described below.

[0147] Exemplary devices Figure 10 FIG. 1 is a structural diagram of an embodiment of a training device for training a simulator disclosed herein. The device of this embodiment can be used to implement the corresponding training method embodiment of the present disclosure. Figure 10 The device shown includes: a first acquisition module 1000 , a first training module 1001 , and a second training module 1002 .

[0148] The first acquisition module 1000 may input input data into the simulator, and the simulator may output a first predicted value of the state of the agent at time step t. The input data may include sampled physical parameters and observed values ​​of the agent's actions and states at time step t-1. The input data may be observation data obtained by observing a physical device including the agent in a real environment. The physical parameters may include at least one of the friction coefficient, stiffness, or damping coefficient. t ≥ 2, where t is an integer.

[0149] The first training module 1001 can input input data into the neural network to be trained, and output the second predicted value of the state of the intelligent agent at the tth time step through the neural network; calculate the value of the first loss function according to the second predicted value and the first predicted value corresponding to the state of the intelligent agent at the tth time step; and update the trainable parameters in the neural network according to the value of the first loss function until the neural network converges to obtain the target neural network.

[0150] The second training module 1002 can input input data into the target neural network while keeping the trainable parameters in the target neural network unchanged, and output the third predicted value of the state of the intelligent agent at the t-th time step through the target neural network; calculate the value of the second loss function according to the third predicted value and the observation value corresponding to the state of the intelligent agent at the t-th time step; the observation value corresponding to the state of the intelligent agent at the t-th time step is the observation data obtained by observing the physical equipment including the intelligent agent in the real environment; calculate the gradient according to the value of the second loss function; update the sampled physical parameters according to the gradient until the second loss function converges to obtain the target physical parameters; configure the target physical parameters into the simulator to obtain the target simulator.

[0151] Figure 11FIG. 1 is a structural diagram of an embodiment of a training device for training a video generation model disclosed herein. The device of this embodiment can be used to implement the corresponding training method embodiment of the present disclosure. Figure 11 The device shown includes: a first determining module 2000 , a second acquiring module 2001 , a first output module 2002 , and a first updating module 2003 .

[0152] The first determining module 2000 can determine a reference image from a sample video. The sample video is obtained by shooting a physical device moving in a real environment.

[0153] The second acquisition module 2001 can acquire an image sequence corresponding to the reference image. The image sequence is obtained based on multiple frames of images following the reference image in the sample video.

[0154] The first output module 2002 can use the reference image and the image sequence as conditions for generating a video, input them into the video generation model to be trained, and obtain the first video through the video generation model.

[0155] The first update module 2003 can calculate the value of the third loss function based on the first video and the sample video; update the trainable parameters in the video generation model according to the value of the third loss function until the video generation model converges to obtain the target video generation model.

[0156] Figure 12 FIG. 1 is a schematic diagram of a structure of an embodiment of a video generation device disclosed in the present invention. The device of this embodiment can be used to implement the corresponding video generation method embodiment of the present invention. Figure 12 The device shown includes: a second determining module 3000 , a calling module 3001 , and a second output module 3002 .

[0157] The second determining module 3000 may be used to determine a reference image.

[0158] The calling module 3001 can input the reference image into the strategy model, and output the action sequence through the strategy model; and input the action sequence into the target simulator, and output the image sequence corresponding to the reference image through the target simulator; the target simulator is used to predict the action of the entity device in the reference image, and the target simulator adopts the method based on Figure 3 The method of any of the embodiments shown is trained.

[0159] The second output module 3002 is used to input the image sequence and the reference image as the conditions for generating a video into the target video generation model, and output the target video through the target video generation model; the target video is used to train the strategy model, and the target video generation model adopts a Figure 5 The method of any of the embodiments shown is trained.

[0160] Exemplary electronic devices Below, reference Figure 13 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be either or both of the first device and the second device, or a standalone device independent of them, and the standalone device may communicate with the first device and the second device to receive collected input signals from them.

[0161] Figure 13 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0162] like Figure 13 As shown, the electronic device includes one or more processors and memory.

[0163] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0164] The memory may store one or more computer program products. The memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may execute the computer program products to implement the training method, video generation method, and / or other desired functions of the various embodiments of the present disclosure described above.

[0165] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0166] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0167] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0168] Of course, to simplify, Figure 13 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0169] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the training method and video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0170] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0171] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the training method and video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0172] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0173] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0174] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0175] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0176] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0177] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0178] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0179] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A training method, characterized in that: The method comprises: Input data is input into a simulator, and the simulator outputs a first predicted value of the state of the agent at time step t; wherein the input data includes sampled physical parameters and observed values ​​of the action and state of the agent at time step t-1; the input data is observation data obtained by observing a physical device including the agent in a real environment; the physical parameters include at least one of a friction coefficient, a stiffness, or a damping coefficient; t ≥ 2, and t is an integer; Inputting the input data into a neural network to be trained, and outputting a second predicted value of the state of the agent at the tth time step through the neural network; Calculating a value of a first loss function according to the second predicted value and the first predicted value corresponding to the state of the agent at the tth time step; updating the trainable parameters in the neural network according to the value of the first loss function until the neural network converges to obtain a target neural network; Keeping the trainable parameters in the target neural network unchanged, inputting the input data into the target neural network, and outputting a third predicted value of the state of the agent at the tth time step through the target neural network; Calculating a value of a second loss function based on a third predicted value and an observed value corresponding to the state of the agent at time step t, wherein the observed value corresponding to the state of the agent at time step t is observation data obtained by observing a physical device including the agent in a real environment; Calculating a gradient according to a value of the second loss function; updating the sampled physical parameters according to the gradient until the second loss function converges to obtain target physical parameters; The target physical parameters are configured into the simulator to obtain a target simulator.

2. The method according to claim 1, characterized in that The neural network is a multi-layer perceptron or a neural network including an attention mechanism.

3. The method according to any one of claims 1-2, characterized in that: The first loss function and / or the second loss function is a mean square error loss function or a cross entropy loss function.

4. A training method, characterized in that: The method comprises: Determining a reference image from a sample video; the sample video is captured of a physical device moving in a real environment; Acquire an image sequence corresponding to the reference image; the image sequence is obtained based on multiple frames of images following the reference image in the sample video; The reference image and the image sequence are used as conditions for generating a video, and are input into a video generation model to be trained, and a first video is obtained through the video generation model; Calculating a value of a third loss function based on the first video and the sample video; According to the value of the third loss function, the trainable parameters in the video generation model are updated until the video generation model converges to obtain a target video generation model.

5. The method according to claim 4, characterized in that Acquiring an image sequence corresponding to the reference image, comprising: Obtaining, based on a plurality of frames of images following the reference image in the sample video, a plurality of first segmented images including foreground objects and a plurality of second segmented images including the physical device after segmentation; Acquire multiple frames of first mask images corresponding to the multiple frames of first segmented images; Multiple frames of the first mask image and multiple frames of the second segmented image are pixel-superimposed to obtain an image sequence corresponding to the reference image.

6. The method according to claim 4 or 5, characterized in that The video generation model is a video diffusion model, and the video diffusion model includes at least one first channel and at least one second channel for input conditions; The reference image and the image sequence are used as conditions for generating a video and input into a video generation model to be trained, including: Using the reference image as a first condition for generating a video, and inputting the reference image into the video diffusion model through the at least one first channel; The reference image and the image sequence are spliced ​​in a time dimension as a second condition for generating a video, and are input into the video diffusion model through the at least one second channel.

7. The method according to claim 6, characterized in that The video diffusion model includes a spatiotemporal attention mechanism.

8. A video generation method, characterized in that: The method comprises: Determine the reference image; Inputting the reference image into a strategy model, and outputting an action sequence via the strategy model; Inputting the action sequence into a target simulator, and outputting an image sequence corresponding to the reference image via the target simulator; the target simulator is used to predict the action of the physical device in the reference image, and the target simulator is trained using the method according to any one of claims 1 to 3; The image sequence and the reference image are input into a target video generation model as conditions for generating a video, and the target video is output through the target video generation model; the target video is used to train the strategy model, and the target video generation model is trained using the method described in any one of claims 4-7.

9. The method according to claim 7, characterized in that The reference image is one of an original image determined based on a sample video, a foreground image obtained by performing a foreground transformation on the original image, or a background image obtained by performing a background transformation on the original image; Inputting the reference image into a policy model comprises: inputting the original image, the foreground image and the background image into the strategy model respectively; Output target video, including: A first target video corresponding to the original image, a second target video corresponding to the foreground image, and a third target video corresponding to the background image are output respectively.

10. A training device, characterized in that: The device comprises: A first acquisition module is configured to input input data into a simulator, and output, via the simulator, a first predicted value of the state of the agent at time step t; wherein the input data includes sampled physical parameters and observed values ​​of the action and state of the agent at time step t-1; the input data is observation data obtained by observing a physical device including the agent in a real environment; the physical parameters include at least one of a friction coefficient, a stiffness, or a damping coefficient; and t ≥ 2, where t is an integer; A first training module is configured to input the input data into a neural network to be trained, and output a second predicted value of the state of the agent at time step t via the neural network; calculate a value of a first loss function based on the second predicted value and the first predicted value corresponding to the state of the agent at time step t; and update trainable parameters in the neural network based on the value of the first loss function until the neural network converges to obtain a target neural network; The second training module is used to keep the trainable parameters in the target neural network unchanged; input the input data into the target neural network, and output the third predicted value of the state of the intelligent agent at the tth time step through the target neural network; calculate the value of the second loss function according to the third predicted value and the observation value corresponding to the state of the intelligent agent at the tth time step; the observation value corresponding to the state of the intelligent agent at the tth time step is the observation data obtained by observing the physical device including the intelligent agent in the real environment; calculate the gradient according to the value of the second loss function; update the sampled physical parameters according to the gradient until the second loss function converges to obtain the target physical parameters; configure the target physical parameters into the simulator to obtain the target simulator.

11. A training device, characterized in that: The device comprises: A first determination module is configured to determine a reference image from a sample video, wherein the sample video is captured by a physical device moving in a real environment; A second acquisition module is configured to acquire an image sequence corresponding to the reference image; the image sequence is obtained based on a plurality of frames of images following the reference image in the sample video; A first output module is configured to input the reference image and the image sequence as conditions for generating a video into a video generation model to be trained, and obtain a first video through the video generation model; The first update module is used to calculate the value of a third loss function based on the first video and the sample video; and update the trainable parameters in the video generation model according to the value of the third loss function until the video generation model converges to obtain a target video generation model.

12. A video generating device, characterized in that: The device comprises: A second determining module, configured to determine a reference image; A calling module, configured to input the reference image into a policy model, and output an action sequence via the policy model; and input the action sequence into a target simulator, and output an image sequence corresponding to the reference image via the target simulator; the target simulator is configured to predict the action of the physical device in the reference image, and the target simulator is trained using the method according to any one of claims 1 to 3; The second output module is used to input the image sequence and the reference image as conditions for generating a video into a target video generation model, and output a target video through the target video generation model; the target video is used to train the strategy model, and the target video generation model is trained using the method described in any one of claims 4-7.

13. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, implements the method described in any one of claims 1 to 9.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 9 is implemented.

15. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images