A method for robot operation strategy migration from simulation to reality

Abstract

The application relates to the technical field of robot operation skill learning, and discloses a robot operation strategy migration method from simulation to reality, which comprises the following steps: acquiring an image sequence collected when a robot executes a task in a real scene and a corresponding action sequence of the robot as real teaching data; constructing a simulation environment according to the real teaching data, optimizing the pose of a target object when a task execution simulation is performed in the simulation environment, and obtaining a dynamics model by taking the action sequence in the real teaching data as the driving; constructing a hybrid reward model according to the similarity between the image sequence when the task execution simulation is performed in the simulation environment and the image sequence in the real teaching data and the judgment result of whether the task is truly completed; training the operation strategy of the robot through the dynamics model and the hybrid reward model; and deploying the trained operation strategy to a real robot to migrate the operation strategy of the robot from the simulation environment to the real scene.

Description

Technical Field

[0001] This invention relates to the field of robot operation skill learning technology, and in particular to a method, system, device and medium for transferring robot operation strategies from simulation to reality. Background Technology

[0002] In industrial and home environments, robots need to perform various visual manipulation tasks such as grasping, inserting, stacking, and tipping. Currently, a core engineering problem in the field of robot vision manipulation is how to enable robots to learn and perform manipulation tasks in real physical environments. The current mainstream technical approach is "Sim-to-Real Transfer": that is, first training the manipulation strategy in a simulation environment at low cost using reinforcement learning, and then deploying the strategy to a real robot for execution. The advantage of this approach is that the simulation environment can provide massive amounts of training data and safe trial-and-error opportunities.

[0003] The systematic differences between simulation environments and the real world in terms of visual appearance and physical dynamics (i.e., the Sim-to-Real Gap) are the core obstacle hindering the practical implementation of this technology. To narrow the simulation-reality gap, existing methods typically use manually predefined scene parameters or perform full-scene 3D reconstruction of real scenes when constructing simulation environments. However, these methods only consider the differences between simulation and reality at the visual level, without addressing the differences between simulation and reality in terms of physical dynamics (such as object contact and collision response). When robots perform actions in simulation environments, the trajectory of objects is inconsistent with that in the real world, causing strategies that perform well in simulation to be unable to be stably executed directly on real robots, resulting in a significant decrease in task success rate. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, device, and medium for transferring robot operation strategies from simulation to reality, which can solve the problem that the difference between the physical dynamics of the simulation environment and the real scene affects the deployment of the strategy on the real robot.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for transferring robot operation strategies from simulation to reality, comprising: Acquire image sequences and corresponding action sequences of the robot when it performs tasks in real-world scenarios, as real teaching data; A simulation environment is constructed based on real teaching data, and the pose of the target object is optimized when the task is executed in the simulation environment by using the action sequence in the real teaching data as the driving force, so as to obtain the dynamic model of the simulation environment. Based on the similarity between the image sequences collected by the simulated robot during task execution simulation in the simulation environment and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed, a hybrid reward model for the simulation environment is constructed. Based on reinforcement learning algorithms, the robot's operating strategy is trained using a dynamic model of the simulated environment and a hybrid reward model. The trained operational strategies are deployed onto real robots to transfer the robot's operational strategies from the simulation environment to the real scene.

[0006] Furthermore, the step of optimizing the pose of the target object during task execution simulation in a simulation environment by using action sequences from real teaching data as the driving force to obtain a dynamic model of the simulation environment includes: Driven by the action sequences in the real teaching data, the pose of the target object is optimized when performing task execution simulation in the simulation environment, so that the position of the target object at key moments when executing the action sequences in the real teaching data in the simulation environment is consistent with the position of the corresponding target object in the real teaching data. In optimizing the pose of the target object, environmental constraints and trajectory constraints are applied. The environmental constraints are constructed based on the physical rationality of the pose of the target object when performing task execution simulation in the simulation environment, and the trajectory constraints are constructed based on the alignment degree between the position of the target object at key moments when executing the action sequences in the real teaching data and the position of the corresponding target object in the real teaching data.

[0007] Furthermore, the construction of a hybrid reward model for the simulation environment, based on the similarity between the image sequences collected by the simulated robot during task execution simulation and the image sequences in the real teaching data, and the judgment result of whether the task has been truly completed, includes: Based on the enhanced invariant distance metric, the similarity between the image sequence acquired by the simulated robot and the image sequence of the real teaching data in the visual embedding space is determined. Based on the spatial relationship of multiple target objects on the two-dimensional image plane when observed from a preset perspective during task execution simulation in a simulation environment, it is determined whether the task has been truly completed. Based on the similarity between the image sequences collected by the simulated robot during task execution simulation in the simulation environment and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed, a hybrid reward model for the simulation environment is constructed.

[0008] Furthermore, the robot's operational strategy, trained using a reinforcement learning algorithm and a dynamic model of the simulated environment combined with a hybrid reward model, includes: Based on reinforcement learning algorithms, the robot's operation strategy is trained through a dynamic model and a hybrid reward model of the simulation environment. During the training process, a curriculum-based domain randomization strategy is used to gradually apply visual and physical perturbations to the dynamic model. The visual perturbations include adjusting the background appearance of the image sequence during simulation and frequency-based data augmentation, while the physical perturbations include changes in the pose of the virtual camera and changes in the initial position of the target object during simulation.

[0009] Furthermore, the construction of the simulation environment based on real teaching data includes: Input the textual description of the task execution and the image sequence from the real teaching data into the visual language model to determine the target object related to the task execution; Match a 3D mesh model to the identified target object from a pre-built library of 3D models; Estimate the six-degree-of-freedom pose of the target object relative to the real robot base coordinate system in the initial frame of the image sequence; A simulation environment is constructed based on the 3D mesh model and the six-degree-of-freedom pose of the target object.

[0010] Furthermore, the following steps are used to determine whether the task has truly been completed: The pre-defined spatial relationships described manually are used as text prompts, and together with the observation images captured from multiple virtual cameras in the simulation from multiple perspectives, a visual language model is used to determine whether the spatial relationships of the target objects in each perspective satisfy the pre-defined object spatial relationships; if the pre-defined object spatial relationships are satisfied, it is determined that the task has been truly completed. Specifically, the visual language model is used only in the pre-execution and post-execution stages to determine whether the preset object spatial relationships are satisfied.

[0011] Furthermore, the robot's operational strategy, trained using a reinforcement learning algorithm and a dynamic model of the simulated environment combined with a hybrid reward model, includes: The DrQ-v2-based reinforcement learning algorithm progressively expands the simulation replay buffer through interaction with a dynamics model and a hybrid reward model to train the policy network and obtain the robot's operation policy.

[0012] Embodiments of the present invention also provide a system for transferring robot operation strategies from simulation to reality, comprising: The data acquisition module is used to acquire image sequences and corresponding action sequences of the robot when the real robot performs tasks in real scenarios, as real teaching data; The dynamic alignment module is used to construct a simulation environment based on real teaching data, and optimize the pose of the target object when performing task simulation in the simulation environment by using the action sequence in the real teaching data as the driving force, so as to obtain the dynamic model of the simulation environment. The reward learning module is used to construct a hybrid reward model for the simulation environment based on the similarity between the image sequences collected by the simulated robot during task execution simulation and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed. The strategy training module is used to train the robot's operation strategy based on reinforcement learning algorithms, through a dynamic model of the simulation environment and a hybrid reward model. The strategy transfer module is used to deploy the trained operation strategies to the real robot, so as to transfer the robot's operation strategies from the simulation environment to the real scene.

[0013] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for transferring robot operation strategies from simulation to reality.

[0014] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for transferring robot operation strategies from simulation to reality.

[0015] The method for transferring robot operation strategies from simulation to reality provided by this invention has at least the following beneficial effects: This invention does not use a predefined simulation environment. Instead, it uses the robot's observed images and actions in real-world scenarios as teaching data to establish the simulation environment. Simultaneously, it constructs a dynamic model so that the poses of objects in the simulation environment are continuously optimized based on the actions in the teaching data during task execution. Through this action-centric pose optimization mechanism, a simulation environment precisely aligned with the dynamics of the real-world scenario is automatically constructed, fundamentally narrowing the gap between simulation and reality at the dynamics level. Then, by combining similarity rewards (guiding the strategy to approach real-world behavior) and task completion rewards in a hybrid reward model, the robot's operational strategy is trained. This ensures that when real-world teaching actions are replayed in the simulation, the object's motion trajectory in the real-world scenario can be reproduced. Therefore, the operational strategy trained in the simulation has the ability to be directly deployed to the real environment, resulting in stable execution and improved task success rate. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0017] Figure 1 A schematic diagram illustrating the process of transferring robot operation strategies from simulation to reality, as provided by this invention; Figure 2 A schematic diagram of a hybrid reward model provided by the present invention; Figure 3 This invention provides a schematic diagram of a two-stage VLM query process; Figure 4 A schematic diagram illustrating the framework of a robot operation strategy transfer method from simulation to reality provided by the present invention; Figure 5 This invention provides a schematic diagram of a multi-task real-world experimental scenario. Figure 6 This is a schematic diagram illustrating a generalized scenario to unseen objects provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] Currently, existing technologies are exploring the differences between simulation and reality mainly from the following directions: Direction 1: Domain Randomization. This approach randomizes visual parameters such as lighting, texture, and object color during simulation training, aiming to teach appearance-insensitive features under a large number of random visual variations, thus enabling transfer to real-world environments. The limitations of this method are: it only addresses visual differences, failing to address the differences between simulation and the real world in terms of physical dynamics (such as object contact and collision responses); and the randomization range lacks guidance from real-world geometric constraints, potentially generating a large number of invalid training samples.

[0020] Direction Two: Full-Scene 3D Reconstruction of Real-World Scenarios. This approach utilizes methods such as NeRF, 3DGS, or multi-view scanning to perform complete 3D reconstruction of real-world operational scenarios, generating high-fidelity digital twin environments for policy training. The limitations of this method are: the reconstruction process is time-consuming and computationally expensive, and the reconstruction results are typically uneditable and non-interactive, making it difficult to support the training requirements of reinforcement learning.

[0021] Option 3: Manually Designed Reward Functions. Reward functions are manually written for each task in a simulation environment to provide training signals for reinforcement learning. The limitations of this method are: different tasks require redesigning different reward functions; the design process relies on domain expert experience; and improper design can lead to policy training failure or the learning of incorrect behaviors.

[0022] Direction 4: Imitation Learning Driven by Large-Scale Real-World Teaching Data. This approach utilizes a large amount of real-world robot teaching data to directly learn the mapping from observation to action through behavior cloning or a vision-language-action (VLA) model. The limitations of this method are: it requires thousands or even more high-quality real-world teaching data points; data acquisition involves human teleoperation of the robot, resulting in extremely high costs; and it exhibits poor generalization ability with limited data, being sensitive to changes in object position, background appearance, and camera perspective.

[0023] The existing technology has the following specific drawbacks: (1) The misalignment between the physical dynamics of the simulation environment and the real scene is the root cause of policy deployment failure. Existing methods typically use manually set object poses and physical parameters when constructing simulation environments, or use full scene reconstruction but lack physical interaction capabilities. This results in the object motion trajectories experienced by the robot in actions such as grasping and inserting in the simulation being inconsistent with the real world. For example, a policy that successfully completes "pours the can into the cup" in the simulation may fail completely when deployed to the real environment because the initial pose deviation of the can causes the gripper to be unable to align with the can. In existing methods, even if the simulation environment is directly initialized using a six-DOF pose estimation method such as FoundationPose, the success rate of replaying expert actions in the simulation is only 24.4% due to the presence of pose estimation errors, which is far from sufficient to support effective policy training.

[0024] (2) The reward function relies on manual design, which limits the scalability of the method. Traditional reinforcement learning requires the design of a specific reward function for each task. For example, "grabbing a banana" requires defining conditions such as "whether the hand is closed" and "whether the banana leaves the table", while "inserting a pen" requires defining different conditions such as "whether the pen tip enters the pen holder" and "whether the pen remains vertical". This task-by-task design approach is labor-intensive, difficult to reuse, and reinforcement learning is difficult to converge when the rewards are too sparse.

[0025] (3) Existing methods cannot maintain both the accuracy and generalization ability of the policy under limited data conditions. Although large-scale pre-trained VLA models have a certain generalization ability, their success rate on specific tasks is low. Fine-tuning these models with a small amount of data can improve the success rate on specific tasks, but after introducing perturbations such as changes in object position and background, the success rate drops sharply from 70% to 20%, indicating that fine-tuning destroys the original generalization representation.

[0026] In summary, existing technologies have their own limitations in narrowing the gap between simulation and reality. There is a lack of a complete technical solution that can start from a small number of real teaching demonstrations, automatically construct a simulation environment that is aligned with the dynamics of real scenes, generate automatically expandable reward signals, and ultimately achieve zero-sample deployment of policies.

[0027] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0028] One embodiment of the present invention relates to a method for transferring robot operation strategies from simulation to reality. The specific process of the method for transferring robot operation strategies from simulation to reality in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain the image sequence and the corresponding action sequence of the robot when it performs a task in a real scene, as real teaching data; Step 102: Construct a simulation environment based on real teaching data, and optimize the pose of the target object when performing task simulation in the simulation environment by using the action sequence in the real teaching data as the driving force, so as to obtain the dynamic model of the simulation environment. Step 103: Based on the similarity between the image sequence collected by the simulated robot during task execution simulation in the simulation environment and the image sequence in the real teaching data, as well as the judgment result of whether the task has been truly completed, construct a hybrid reward model for the simulation environment. Step 104: Based on the reinforcement learning algorithm, the robot's operation strategy is trained using the dynamic model of the simulation environment and the hybrid reward model. Step 105: Deploy the trained operation strategy onto the real robot to transfer the robot's operation strategy from the simulation environment to the real scene.

[0029] The following details the implementation of the robot operation strategy transfer method from simulation to reality in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0030] In step 101, expert teaching data (i.e., real teaching data) from real-world scenarios are collected.

[0031] Using a robotic system equipped with an RGB-D camera and a robotic arm, a small number of expert teaching trajectories were acquired through manual teleoperation in real-world operating scenarios. Each teaching trajectory consists of a series of temporally continuous RGB-D image frames as an observation image sequence, and the corresponding robotic arm end effector movements as an action sequence. Simultaneously, during the acquisition process, the initial placement position of the object and the grasping entry angle were intentionally varied to cover different operational initiation conditions.

[0032] The collected teaching dataset is denoted as . Each trajectory From length of The system consists of RGB-D observation-action pairs. Simultaneously, the precise pose relationship between the camera and the robot base is obtained through extrinsic parameter calibration, and this pose relationship will be reproduced in the subsequent simulation environment construction.

[0033] In step 102, a motion-centered simulation dynamics model is constructed (equivalent to providing a motion-centered simulation environment construction method). The goal of this step is to automatically construct a simulation environment based on the real teaching data collected in step 101, so that when the expert's actions are replayed in this environment, the motion trajectory of the object is highly consistent with the motion trajectory observed in the real world.

[0034] This step specifically includes the following sub-steps: (1) Task decomposition and target object recognition.

[0035] Given a textual description of the task execution and an initial frame RGB image of the taught trajectory, both are input into a Visual Language Model (VLM). The VLM analyzes the task content and outputs a list of target objects directly related to the task execution. Subsequently, an open vocabulary detector is used to locate these target objects in the initial frame image, and pixel-level masks of the objects are obtained through a segmentation algorithm.

[0036] (2) Obtaining the three-dimensional model of the object and estimating the initial pose.

[0037] From a pre-built 3D model library In this process, a 3D mesh model is matched for each identified target object. The 3D meshes in the model library are retrieved from online 3D model resource libraries or obtained by offline scanning of real objects using a mobile phone. Matching does not require the 3D model to be completely identical to the real object; the matching criteria are relaxed to accept category-level texture similarity matching, without requiring exact consistency with real-world instances. Subsequently, combining the SAM segmentation results and the FoundationPose algorithm, the six-DOF pose of each target object relative to the robot's base coordinate system in the initial frame of the image sequence is estimated. .

[0038] (3) Simulation environment initialization.

[0039] In the Robosuite simulation framework based on the MuJoCo physics engine, the robot base is used as the origin of the world coordinate system. A virtual camera is placed in the simulation according to the calibrated camera pose, and the three-dimensional mesh model obtained in step (2) is placed in the simulation scene with the estimated initial pose.

[0040] The initial simulation environment is defined as follows: ; Each tuple consists of a grid model. (Associated with predefined physical and semantic properties) and its estimated initial six-degree-of-freedom pose composition.

[0041] (4) Action-centered pose optimization.

[0042] When directly using the initial pose from step (3) to replay expert actions in a simulation, problems such as object collision penetration or excessive positional deviation often occur due to pose estimation errors, leading to playback failure. To solve this problem, this embodiment proposes a pose optimization method centered on actions. Specifically, driven by the action sequence in the real teaching data, the initial pose of the target object in the simulation environment is optimized so that when these actions are performed in the simulation, the position of the target object at key moments (such as the moment the gripper closes to grasp, the moment the task is completed) is consistent with the position observed in the real teaching.

[0043] Two key constraints are introduced in this construction process to optimize the pose of the target object: the first is the environmental constraint, which is constructed based on the physical reasonableness of the target object's pose during task execution simulation in the simulation environment. It is used to ensure that the recovered object pose is physically feasible, including avoiding objects penetrating the tabletop, avoiding unreasonable collisions between objects, and maintaining its stability (no collisions and maintaining a stable configuration). The second is the trajectory constraint, which is constructed based on the alignment degree between the position of the target object at key moments when executing the action sequence in the real teaching data and the corresponding position of the target object in the real teaching data. By back-reasoning from the expert teaching actions, the pose of the object in the simulation environment is made consistent with the position of the hand at key moments during the expert's execution, including the hand alignment constraint at the moment of grasping and the spatial consistency constraint of the final task completion state (such as insertion, stacking, placement, etc.).

[0044] For each teaching data Optimize the initial object pose to meet these constraints, and generate a task-specific simulation environment and high-fidelity data. This is used for subsequent reward model learning and reinforcement learning training.

[0045] The pose optimization problem can be formalized as follows: ; This optimization is constrained by simulation dynamics and environmental conditions. From initial configuration under expert action Obtained through forward simulation. Indicating instruction A set of objects and key state pairs, each key state... This corresponds to critical moments such as changes in the gripper's state or task completion. The first item is forced trajectory alignment. Experts in critical situations Object The target location This corresponds to the position of the object in the simulation. Represents the physical cost function, used for penalty time. The configuration is unstable. It is a weighting factor that balances physical plausibility and imitation accuracy.

[0046] Traditional methods either initialize the simulation with manually preset object poses or reconstruct the entire real-world scene in 3D. This embodiment identifies and models only the key objects relevant to the task, and ensures that the simulation dynamics align with the real world through action-centric pose optimization, rather than pursuing visual consistency. This design prioritizes modeling the dynamic relationship established between the robot and the target object through actions, rather than background visual consistency.

[0047] In step 103, a hybrid reward model is learned (equivalent to providing a reusable automatic reward generation method based on object projection relationships and visual language models). The goal of this step is to automatically generate reward signals for reinforcement learning training based on the simulation environment built in step 102, without the need for manually designing reward functions for each task.

[0048] This step specifically includes the following sub-steps: (1) Calculate teaching similarity reward: Based on the enhanced invariant distance metric, determine the similarity between the image sequence collected by the simulated robot and the image sequence of the real teaching data in the visual embedding space when performing task execution simulation in the simulation environment.

[0049] The reward provides dense training signals, guiding the policy to produce behavior that visually approximates expert teaching. Specifically, it uses an augmented invariant distance metric (ADM) to calculate the distance between the current simulated observation and all expert teaching observations in the learned visual embedding space, finding the nearest teaching observation as a reference.

[0050] At each state transition, the agent's next observation is sampled from the replay buffer. With simulation data All teaching observations Comparison. Recent teaching observations. Determined in the following ways:

[0051] ; Similarity reward Through evaluation and The degree of closeness between them and the threshold of dynamic estimation The threshold is determined by the relationship between consecutive teaching observations, and is defined as the average distance between them. ; To adapt to changes in the learned representation, this threshold is updated periodically during training as the encoder and dynamic model evolve, ensuring temporal smoothness and representation consistency.

[0052] (2) Calculate the projection reward based on the visual language model: determine whether the task has been truly completed based on the spatial relationship of multiple target objects on the two-dimensional image plane when observed from a preset perspective during task execution simulation in the simulation environment.

[0053] This reward provides a sparse but explicit signal of task completion. Unlike traditional methods that require manually defining task-specific conditions such as "object A reaches location B," this method utilizes manually guided object projection relationships as the basis for judging task completion. Object projection relationships refer to the spatial relationships, such as occlusion and overlap, of multiple three-dimensional objects on a two-dimensional image plane when viewed from a specific perspective. For example, "an apple being caught by a hand" is represented from a bird's-eye view as "the apple's projection mostly falls within the hand's projection area." These projection relationships are intuitive and universal; operators only need to describe the expected projection relationships between objects using natural language to define the conditions for task success.

[0054] The specific implementation is as follows: The manually described object projection relationships (i.e., spatial relationships) are used as text prompts and input into the VLM along with multi-view observation images captured from multiple virtual cameras in the simulation. The VLM determines whether the object projection relationships at each viewpoint satisfy the described conditions and outputs binary labels. Only when all viewpoints satisfy the condition ( Only after this process is completed is the task deemed complete and a projection reward awarded. This multi-view consistency check effectively reduces visual ambiguity under a single viewpoint. Experiments show that when using four views, the accuracy of VLM's judgment can reach the level required for effective annotation.

[0055] like Figure 2 The diagram illustrates the hybrid reward proposed in the apple-picking example. This hybrid reward integrates dense similarity rewards and sparse projection rewards. The similarity reward is calculated by comparing the agent's next-time observation with expert instruction using an enhanced invariant distance metric in the learned embedding space. The projection reward is generated by a Virtual Model (VLM) that evaluates multi-view observations based on manually guided object projection relationships.

[0056] To reduce VLM call overhead, the visual language model is used only in the pre-task and post-task stages to determine whether the preset object spatial relationships are met. That is, projection rewards are calculated only at critical moments (gripper state changes or end of a round). This embodiment designs a two-stage VLM query process: in the pre-task stage, the VLM evaluates whether the current state meets the prerequisites for executing the next action; in the post-task stage, the VLM evaluates whether the task has been successfully completed. Each prompt template explicitly specifies the target object and the moving object, and can be reused for similar tasks by simply changing the object names.

[0057] Figure 3 A two-stage VLM query process is illustrated. For each task, a two-stage VLM query process is used to generate reward labels for training the reward function. In the pre-task stage, the VLM is queried using manually guided object projection relationships to determine whether the current state is suitable for executing the next action. In the post-task stage, the VLM evaluates whether the task has been successfully completed based on another set of manually guided object projection relationships. The cue template is defined at the skill level and can be instantiated for new tasks by simply replacing the names of the moving object and the target object.

[0058] By combining projection rewards with teaching similarity rewards, a hybrid learning objective is formed. Defined as:

[0059] ; Each component is defined as follows: ; in, This represents the sparse reward signal from the VLM. The similarity reward is given only if two conditions are met simultaneously: the agent's next observation... Observation must be conducted under expert instruction. threshold Within the specified range, and the projected reward must be zero. This condition ensures that similarity rewards are not redundantly allocated after the task is completed. Similarity rewards are also weighted by time decay. Modulation, and These represent the length of the matched expert trajectory and the matching time step, respectively. This time decay encourages the agent to perform tasks sequentially. It is a constant scalar that controls the nominal magnitude of the similarity reward.

[0060] Traditional methods require manually writing reward functions for each task, resulting in significant differences between tasks and making reuse impossible. This embodiment proposes a hybrid reward model consisting of two automatically calculated reward components: a dense teaching similarity reward that continuously guides the strategy towards expert behavior, and a sparse projection reward that explicitly determines whether the task has been completed. The two components complement each other: the similarity reward addresses the question of "how to gradually approach the goal," while the projection reward addresses the question of "whether the task has actually been completed." More importantly, the prompt template for the projection reward is defined at the skill level, allowing reuse for similar new tasks simply by replacing the target object name, without the need for redesign.

[0061] In step 104, the dynamic model constructed in step 102 and the hybrid reward model learned in step 103 are combined into a complete world model, and a deployable policy is trained in the combined world model. The goal of this step is to use the dynamic model constructed in step 102 and the hybrid reward model learned in step 103 to train an operational policy through reinforcement learning that can accurately perform tasks and cope with various perturbations in the real environment.

[0062] In one example, the parameters of the hybrid reward model are frozen, and a curriculum-based domain randomization is applied to the dynamics model: the magnitudes of visual and physical perturbations are gradually increased from small to large according to an exponential scheduler. Initially, the policy learns basic operational capabilities under near-realistic conditions, and later, robustness is enhanced under large perturbations. Visual perturbations include changes in background appearance and frequency-based data augmentation; physical perturbations include changes in camera pose and changes in the initial position of objects.

[0063] Policy training employs a reinforcement learning algorithm based on DrQ-v2. The simulation replay buffer is progressively expanded through interaction with a combinatorial world model. For each transformation, Stored in In, among them, This represents a new observation generated by the dynamic model. To maintain consistency with the hybrid reward model, hybrid reward... The Q function is calculated from unenhanced observations. Use the enhanced input Training was conducted, including... This indicates a frequency-based data augmentation operation.

[0064] The final temporal difference loss formula for Q-learning is as follows: ; in, from Mid-sampling, Indicates the target Q-network, It is a reward discount factor. Utilizing curriculum-based domain randomization helps robust policy learning, improves convergence and generalization ability, thereby achieving effective simulation-to-real-world transfer.

[0065] Traditional methods train policies in fixed simulation environments. The training and deployment environments differ significantly in visual appearance, and direct fine-tuning can easily disrupt the rich visual representations acquired during pre-training, thus weakening the policy's robustness to visual perturbations. This embodiment demonstrates that, based on a simulation environment already aligned with real-world scene dynamics, visual and physical perturbations are gradually introduced through curriculum-based domain randomization. This allows the policy to learn to ignore task-irrelevant visual changes while maintaining sensitivity to task-related information. Since the dynamic foundation is already aligned, domain randomization only needs to handle visual differences, significantly reducing the difficulty of policy transfer.

[0066] In step 105, the trained policy is deployed to the real robot with zero samples. This step directly loads the policy trained in step 104 into the control system of the real robot without any fine-tuning or online adaptation in the real environment.

[0067] During deployment, the real robot's RGB-D camera captures scene images in real time with the same pose as in the simulation. The images are then input into a trained policy network, which outputs the target pose and gripper action of the robotic arm's end effector, which is then executed by the robotic arm controller.

[0068] The zero-shot deployment is feasible because this embodiment systematically eliminates various sources of the simulation-to-reality gap in the preceding steps: Step 102 ensures that the physical dynamics in the simulation are aligned with the real world through action-centric pose optimization, eliminating the dynamics gap; Step 103 automatically generates reward signals consistent with the real task objective through VLM-based projection rewards and teaching similarity rewards, eliminating the reward function mismatch problem; Step 104 makes the policy robust to changes in visual appearance through curriculum-based domain randomization, eliminating the visual gap. The gaps at the dynamics, reward, and visual levels are addressed one by one, thus enabling the policy to be executed directly in the real environment.

[0069] The above steps provide a complete method for zero-shot simulation-to-reality transfer of strategies based on dynamic alignment simulation environment. It eliminates the simulation-to-reality gap at the dynamic level by action-centered pose optimization, eliminates the artificial design dependence of reward function by hybrid reward model, and eliminates the simulation-to-reality gap at the visual level by curriculum-based domain randomization. The three work together to enable the trained strategy to be directly deployed to real robot to perform operation tasks without any fine-tuning in real environment.

[0070] The robot operation strategy transfer system corresponding to the above method from simulation to reality may include: a data acquisition module (including an RGB-D camera, a robotic arm, and a teleoperation interface), a simulation environment construction module (including a visual language model-driven task decomposition and object recognition unit, a six-degree-of-freedom pose estimation unit, a 3D model library, a motion-centered pose optimization unit, and a physics engine-based simulation environment generation unit), a reward model learning module (including a teaching similarity reward calculation unit and a VLM-based projection reward calculation unit), a policy training module (including a domain randomization unit and a reinforcement learning training unit), and a policy deployment module (including a real-time image acquisition, policy inference, and robotic arm control execution unit).

[0071] As can be seen, the robot operation strategy transfer method of this invention from simulation to reality is a combinatorial world model framework for few-shot generalizable policy learning. For example... Figure 4 As shown, (a) uses real-world expert demonstrations and a 3D model library to reconstruct an action-centric simulation environment as a dynamic model; (b) trains a hybrid reward model by integrating trajectory similarity and manually guided projection relationships; (c) freezes the hybrid reward model under a combined world model and applies domain randomization to the dynamic model for policy learning; and (d) deploys the learned policy to the real world through zero-shot simulation to real-world transfer.

[0072] In summary, this invention presents a complete method for automatically constructing a simulation environment aligned with the dynamics of a real scene from a small amount of real expert teaching data. Within this environment, an operational strategy is trained using automatically generated hybrid reward signals. This allows the trained strategy to be deployed directly to a real robot with zero-shot execution without any fine-tuning in the real environment. The method involves: extracting scene and motion information from a small amount of real expert teaching data; automatically constructing a simulation environment precisely aligned with the real scene at the physical and dynamic level through a motion-centric pose optimization mechanism, thus fundamentally reducing the gap between simulation and reality at the dynamic level; automatically generating task reward signals in this environment without manual design through a hybrid reward mechanism combining projection rewards based on a visual language model and dense rewards based on teaching similarity, which can be extended to new tasks by simply replacing object names, forming a combined world model; training the operational strategy within this world model using reinforcement learning, introducing curriculum-based randomization, and training the strategy on the aligned dynamic model, enabling the strategy to maintain task accuracy while gaining robustness to visual and physical perturbations; and finally, directly deploying the trained strategy onto a real robot system equipped with a camera and robotic arm to achieve zero-shot execution of the target operation task.

[0073] Traditional reinforcement learning trains policies in predefined simulation environments where object positions and physical parameters are manually set, resulting in inherent deviations from real-world scenarios. Policy deployment requires additional fine-tuning in the real-world environment or online adaptation. This invention, however, does not use predefined simulation environments. Instead, it starts with limited teaching data from real-world scenarios and automatically generates a simulation environment aligned with the dynamics of the real-world environment through action-centric pose optimization. This ensures that replaying expert actions in the simulation faithfully reproduces the object motion trajectories of the real world, thereby enabling the simulation-trained policy to be directly deployed to real-world environments.

[0074] Compared with existing methods, the present invention has the following significant technical advantages: (1) Zero-shot simulation to real-world transfer of operational strategies was achieved. The trained strategies do not require any fine-tuning or online adaptation in the real environment and can be directly deployed to real robots to perform operational tasks. On five types of real-world operational tasks, the average success rate under standard conditions reached 86%, and even after introducing random perturbations such as object position, background appearance, and camera perspective, the average success rate remained at 66%. In contrast, the optimal baseline method (π0) fine-tuned with the same amount of teaching data achieved 70% under standard conditions, but plummeted to 20% under perturbation conditions.

[0075] (2) Significantly reduces the amount of real teaching data required. Only 5 to 10 real expert teachings are needed to complete the entire process from simulation environment construction to policy training. In contrast, even with 20 teachings, the average success rate of the behavior cloning method is only 8% to 34%; although the large-scale pre-trained VLA model uses massive amounts of data for pre-training, its success rate on specific tasks is still lower than that of the policy trained with only 10 teachings in this invention.

[0076] (3) It eliminates the need for manually designed reward functions. Both components of the hybrid reward model can be calculated automatically, without the need to write specific reward code for each task. The cue template for projected rewards is defined at the skill level and can be extended to new tasks by changing the object name. Experiments show that the signal trend generated by this reward model is highly consistent with manually designed dense rewards.

[0077] (4) Action-centric pose optimization improved the success rate of simulated action playback from 24.4% to 91.7%. This improvement is the basis for achieving zero-shot policy transfer and proves the accurate alignment between simulated dynamics and the real world.

[0078] (5) Supports zero-shot generalization across object categories. By replacing the 3D mesh model of the object in the simulation environment and reusing the existing motion trajectory, the strategy can be extended to new, unseen objects (such as from "grabbing bananas" to "grabbing strawberries") without the need for additional real teaching data, with an average success rate of 75%.

[0079] To verify the effectiveness of the technical solution described in this invention, experiments were conducted on various real-world robot operation tasks, including grasping, tipping, insertion, and generalization ability tests across object categories. The implementation process of this invention is described in detail below with reference to different scenarios, but this is not intended to limit the scope of protection of this invention.

[0080] like Figure 5 The image shows a real-world multi-task experimental scenario. The top-right image shows the original expert-collected scenario. To evaluate generalization ability, randomization of initial object positions, background appearance, and camera poses was introduced relative to these baseline configurations during policy testing.

[0081] Example 1: Lift Banana Task In this task, the robot needs to identify banana objects in the scene, grasp them, and lift them to a target height. This embodiment uses only 10 expert teaching videos from real-world operations as input data. Each video contains an RGB-D image sequence and the corresponding hand trajectory. The banana's 3D mesh model comes from an online resource library. The initial pose is obtained through object segmentation and six-degree-of-freedom pose estimation methods, and further optimized using the environmental and trajectory constraints proposed in this invention. This ensures that the banana's position, pose, and interaction with the robot's hand in the simulation environment are highly consistent with the real-world teaching.

[0082] Subsequently, the actions were replayed in the constructed simulation scenario to verify the executability of the trajectory, and the stability of the banana being grasped by the gripper was determined through multi-view projection relationships. After constructing a reward model based on teaching similarity and projection geometry, a grasping strategy was trained using reinforcement learning methods. Random lighting, background texture perturbation, and random spectral occlusion were added during training to enhance generalization ability. Finally, the trained strategy was directly deployed on a real robot, achieving a real zero-shot success rate of over 80% without any real fine-tuning, fully validating the effectiveness of this invention in grasping tasks.

[0083] Example 2, Pour Can Task: In this task, the robot needs to pick up a metal can and pour its contents into a designated container. This task involves complex object rotation, posture control, and determination of the pouring angle. First, the three-dimensional posture of the can and container is recovered from the teaching video according to the method of this invention. The teaching actions are then replayed in a simulation environment, and the can's posture is constrained and optimized based on the gripper trajectory to ensure consistency with the actual teaching in terms of contact point, rotation axis, and center of gravity position.

[0084] To accurately evaluate the success of the tipping action, this invention simultaneously sets up three virtual cameras at different locations in the simulation to capture the projection relationship between the can, the content flow line, and the container opening. A visual language model then determines whether conditions such as "the tipping angle of the can exceeds a preset threshold" and "the projection of the can falls into the container area" are met. These projection constraints are used to construct a reward signal, enabling the reinforcement learning to converge stably. The final trained strategy can complete the tipping action on a real robot without any adjustments, demonstrating the strong feasibility of this invention in tasks involving complex rotation and geometric relationship judgments.

[0085] Example 3: Insert Pen Task This embodiment verifies the performance of the invention in fine manipulation and contact-sensitive tasks. The robot needs to insert a slender cylindrical pen into an upright pen holder. First, a precise 3D model of the pen and pen holder is obtained using a lightweight 3D scanning method. Then, keyframe poses during the insertion process, including the spatial alignment between the pen tip and the pen holder opening, are reconstructed using teaching data. Through the trajectory constraints proposed in this invention, an insertion trajectory that can be stably replayed in simulation is optimized and generated.

[0086] To determine the success of the insertion action, this invention utilizes a multi-view camera to acquire the projection overlap, occlusion relationship, and depth changes of the pen and pen holder, thereby inferring whether the pen tip has entered the container. Combined with teaching similarity rewards, a dense and distinguishable task progress signal can be constructed, enabling reinforcement learning to stably train the insertion strategy. Ultimately, the trained strategy achieves zero-sample insertion with approximately 80% success rate in a real-world environment, validating the applicability of this invention in tasks involving tightly coupled and precisely aligned operations.

[0087] Figure 6 Describe the generalization to unseen objects. Evaluate DeCoW's ability to generalize to new tasks involving objects not present in the original expert teaching dataset, including Lift Strawberry and Pick Place Green Apple. Replace the object mesh model in a simulation environment without acquiring additional real-world teaching data.

[0088] Example 4: Generalization test of new objects (e.g., strawberries, green apples): To further verify the invention's ability to generalize across object categories, this embodiment tests the robot's performance on unfamiliar objects (such as strawberries and green apples). This invention does not require any additional teaching videos; it constructs a new simulation environment simply by replacing the object's 3D mesh and reusing motion trajectories optimized for similar tasks. Because the motion center simulation method does not depend on object category but rather on the geometric relationships of the objects required by the task, the robot can directly use existing strategies for operation.

[0089] Tests show that this invention maintains stable zero-shot performance in new object tasks, far exceeding the performance of current mainstream methods such as OCTO and OpenVLA in cross-object generalization tasks. This result demonstrates that this invention achieves excellent task transfer capabilities without additional teaching, possessing extremely high practical value.

[0090] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0091] Another embodiment of the present invention relates to a system for transferring robot operation strategies from simulation to reality. This embodiment of the system for transferring robot operation strategies from simulation to reality includes: The data acquisition module is used to acquire image sequences and corresponding action sequences of the robot when the real robot performs tasks in real scenarios, as real teaching data; The dynamic alignment module is used to model the target objects related to task execution in the real teaching data to build a simulation environment. By using the action sequence in the real teaching data as a driver, it optimizes the pose of the target objects when simulating task execution in the simulation environment to obtain the dynamic model of the simulation environment. The reward learning module is used to construct a hybrid reward model for the simulation environment based on the similarity between the image sequences collected by the simulated robot during task execution simulation and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed. The strategy training module is used to train the robot's operation strategy based on reinforcement learning algorithms, through a dynamic model of the simulation environment and a hybrid reward model. The strategy transfer module is used to deploy the trained operation strategies to the real robot, so as to transfer the robot's operation strategies from the simulation environment to the real scene.

[0092] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0093] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0094] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the robot operation strategy transfer method from simulation to reality in the above embodiments.

[0095] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0096] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0097] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0098] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for transferring robot operation strategies from simulation to reality, characterized in that, include: Acquire image sequences and corresponding action sequences of the robot when it performs tasks in real-world scenarios, as real teaching data; A simulation environment is constructed based on real teaching data, and the pose of the target object is optimized when the task is executed in the simulation environment by using the action sequence in the real teaching data as the driving force, so as to obtain the dynamic model of the simulation environment. Based on the similarity between the image sequences collected by the simulated robot during task execution simulation in the simulation environment and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed, a hybrid reward model for the simulation environment is constructed. Based on reinforcement learning algorithms, the robot's operating strategy is trained using a dynamic model of the simulated environment and a hybrid reward model. The trained operational strategies are deployed onto real robots to transfer the robot's operational strategies from the simulation environment to the real scene.

2. The method for transferring robot operation strategies from simulation to reality according to claim 1, characterized in that, The process involves optimizing the pose of the target object during task execution simulation in a simulation environment, driven by action sequences from real teaching data, to obtain a dynamic model of the simulation environment. This includes: Driven by the action sequences in the real teaching data, the pose of the target object is optimized when performing task execution simulation in the simulation environment, so that the position of the target object at key moments when executing the action sequences in the real teaching data in the simulation environment is consistent with the position of the corresponding target object in the real teaching data. In optimizing the pose of the target object, environmental constraints and trajectory constraints are applied. The environmental constraints are constructed based on the physical rationality of the pose of the target object when performing task execution simulation in the simulation environment, and the trajectory constraints are constructed based on the alignment degree between the position of the target object at key moments when executing the action sequences in the real teaching data and the position of the corresponding target object in the real teaching data.

3. The method for transferring robot operation strategies from simulation to reality according to claim 1, characterized in that, The hybrid reward model for the simulation environment is constructed based on the similarity between the image sequences collected by the simulated robot during task execution simulation and the image sequences in the real teaching data, as well as the judgment result of whether the task has been truly completed. This includes: Based on the enhanced invariant distance metric, the similarity between the image sequence acquired by the simulated robot and the image sequence of the real teaching data in the visual embedding space is determined. Based on the spatial relationship of multiple target objects on the two-dimensional image plane when observed from a preset perspective during task execution simulation in a simulation environment, it is determined whether the task has been truly completed. Based on the similarity between the image sequences collected by the simulated robot during task execution simulation in the simulation environment and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed, a hybrid reward model for the simulation environment is constructed.

4. The method for transferring robot operation strategies from simulation to reality according to claim 1, characterized in that, The robot's operational strategy, trained using a reinforcement learning algorithm and a dynamic model of the simulated environment combined with a hybrid reward model, includes: Based on reinforcement learning algorithms, the robot's operation strategy is trained through a dynamic model and a hybrid reward model of the simulation environment. During the training process, a curriculum-based domain randomization strategy is used to gradually apply visual and physical perturbations to the dynamic model. The visual perturbations include adjusting the background appearance of the image sequence during simulation and frequency-based data augmentation, while the physical perturbations include changes in the pose of the virtual camera and changes in the initial position of the target object during simulation.

5. The method for transferring robot operation strategies from simulation to reality according to claim 2, characterized in that, The construction of the simulation environment based on real teaching data includes: Input the textual description of the task execution and the image sequence from the real teaching data into the visual language model to determine the target object related to the task execution; Match a 3D mesh model to the identified target object from a pre-built library of 3D models; Estimate the six-degree-of-freedom pose of the target object relative to the real robot base coordinate system in the initial frame of the image sequence; A simulation environment is constructed based on the 3D mesh model and the six-degree-of-freedom pose of the target object.

6. The method for transferring robot operation strategies from simulation to reality according to claim 3, characterized in that, Determine whether the task has been truly completed by following these steps: The preset spatial relationships described manually are used as text prompts, and together with the observation images captured from multiple virtual cameras in the simulation from multiple perspectives, a visual language model is used to determine whether the spatial relationships of the target objects in each perspective satisfy the preset object spatial relationships. If the preset spatial relationships between objects are satisfied, then the task has been truly completed. Specifically, the visual language model is used only in the pre-execution and post-execution stages to determine whether the preset object spatial relationships are satisfied.

7. The method for transferring robot operation strategies from simulation to reality according to claim 1, characterized in that, The robot's operational strategy, trained using a reinforcement learning algorithm and a dynamic model of the simulated environment combined with a hybrid reward model, includes: The DrQ-v2-based reinforcement learning algorithm progressively expands the simulation replay buffer through interaction with a dynamics model and a hybrid reward model to train the policy network and obtain the robot's operation policy.

8. A system for transferring robot operation strategies from simulation to reality, characterized in that, include: The data acquisition module is used to acquire image sequences and corresponding action sequences of the robot when the real robot performs tasks in real scenarios, as real teaching data; The dynamic alignment module is used to construct a simulation environment based on real teaching data, and optimize the pose of the target object when performing task simulation in the simulation environment by using the action sequence in the real teaching data as the driving force, so as to obtain the dynamic model of the simulation environment. The reward learning module is used to construct a hybrid reward model for the simulation environment based on the similarity between the image sequences collected by the simulated robot during task execution simulation and the image sequences in the real teaching data, as well as the judgment results of whether the task has been truly completed. The strategy training module is used to train the robot's operation strategy based on reinforcement learning algorithms, through a dynamic model of the simulation environment and a hybrid reward model. The strategy transfer module is used to deploy the trained operation strategies to the real robot, so as to transfer the robot's operation strategies from the simulation environment to the real scene.

9. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method for transferring robot operation strategies from simulation to reality as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for transferring robot operation strategies from simulation to reality as described in any one of claims 1 to 7.

Images