Robot reinforcement learning strategy migration deployment method, computing device and storage medium

Abstract

The invention discloses a robot reinforcement learning strategy migration deployment method, computing equipment and a storage medium, and the method comprises the steps: configuring the parameters of a controller of a robot in a target simulation environment according to the simulation parameter difference between a source simulation environment and the target simulation environment; on the basis of state data of a plurality of continuous time steps collected by the robot in the target simulation environment, an enhanced observation vector is constructed; inputting the enhanced observation vector into a reinforcement learning strategy which is trained in a source simulation environment to obtain an initial action instruction; performing adaptive adjustment on the initial action instruction based on a current task stage determined by a multi-stage task state machine, and generating a stage adaptive action instruction; the multi-stage task state machine triggers automatic switching between stages based on the detection result of the real-time physical state of the robot; and the configured controller is used for resolving the stage adaptive action instruction into a bottom-layer joint control instruction and executing the bottom-layer joint control instruction so as to drive the robot to complete the target task.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method for transferring and deploying reinforcement learning strategies for robots, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of reinforcement learning technology in robotics, training control policies in simulation environments and transferring them to real robots (Sim-to-Real, Sim2Real) has become the mainstream paradigm. However, directly deploying reinforcement learning policies trained in simulation environments to real robot systems often faces the problem of significant performance degradation. This "reality gap" severely restricts the practical application of reinforcement learning in the field of robotics.

[0003] Traditional policy transfer methods typically rely on the collection of large amounts of real-world data or tedious manual policy adjustments, resulting in low transfer efficiency and low task success rates, which limits the adaptability and robustness of reinforcement learning policies in real-world environments. Summary of the Invention

[0004] The purpose of this application is to provide a method for transferring and deploying robot reinforcement learning strategies, a computing device, and a computer-readable storage medium, which can significantly improve the robustness and adaptability of the transfer, as well as increase the task success rate and overall efficiency.

[0005] To achieve the above objectives: In a first aspect, embodiments of this application provide a method for transferring and deploying robot reinforcement learning policies, the method comprising: Based on the differences in simulation parameters between the source simulation environment and the target simulation environment, configure the parameters of the robot's controller in the target simulation environment; An enhanced observation vector is constructed based on the state data of the robot collected from multiple consecutive time steps in the target simulation environment. The enhanced observation vector is input into the reinforcement learning policy trained in the source simulation environment to obtain the initial action command; The initial action instructions are adaptively adjusted based on the current task stage determined by a multi-stage task state machine to generate stage-adaptive action instructions; the multi-stage task state machine triggers automatic transitions between stages based on the detection results of the robot's real-time physical state. Using the configured controller, the stage-adapted motion instructions are calculated into low-level joint control instructions and executed to drive the robot to complete the target task.

[0006] In some embodiments, the reinforcement learning strategy infers at a first frequency, the underlying joint control commands are executed at a second frequency higher than the first frequency, and the first frequency and the second frequency are coordinated by a frequency reduction synchronization mechanism.

[0007] In some embodiments, the first frequency is 50Hz, the second frequency is 200Hz, and the frequency reduction coefficient of the frequency reduction synchronization mechanism is 4.

[0008] In some embodiments, configuring the parameters of the robot's controller in the target simulation environment based on the differences in simulation parameters between the source simulation environment and the target simulation environment includes: The time step scaling factor is determined based on the ratio of the simulation step size of the source simulation environment to the simulation step size of the target simulation environment. Based on the time step scaling factor, the proportional gain and derivative gain of the robot's proportional-differential controller are dynamically scaled linearly in a proportional manner.

[0009] In some embodiments, constructing an enhanced observation vector based on state data collected from the robot at multiple consecutive time steps in the target simulation environment includes: In each control cycle corresponding to the first frequency, a current observation vector of fixed dimensions is generated based on the collected state data and historical information. The current observation vector is added to the tail of the preset queue, and the historical observation vector at the head of the preset queue is removed; the preset queue is a first-in-first-out queue of length N, used to store the observation vectors corresponding to N consecutive control cycles, where N is an integer greater than 1; The enhanced observation vector is formed by concatenating the N observation vectors stored in the preset queue in chronological order.

[0010] In some embodiments, generating a fixed-dimensional current observation vector based on the collected state data and historical information includes: According to the predetermined dimension mapping rules, the joint positions, end effector poses, target object positions, and action commands from the previous control cycle in the state data are fused into a fixed-dimensional basic observation vector. The basic observation vector is standardized, and the value range of the standardized basic observation vector is clipped to obtain the current observation vector.

[0011] In some embodiments, the target task includes a grasping and placing task, and the multi-stage task state machine divides the grasping and placing task into at least four stages to be executed sequentially: an approach stage, a grasping stage, a transport stage, and a placing stage; the method further includes: When the distance between the robot's end effector and the target grasping point is detected to be less than or equal to a first distance threshold, the transition from the approach phase to the grasping phase is triggered. When the distance between the gripper and the target object is detected to be less than or equal to a second distance threshold within M consecutive control cycles, the transition from the gripping phase to the transport phase is triggered; M is an integer greater than 1. When the distance between the end effector and the target placement point is detected to be less than or equal to a third distance threshold, the transition from the transportation phase to the placement phase is triggered.

[0012] In some embodiments, the method further includes: During the gripping phase and / or the transport phase, a constant holding torque or holding force is superimposed on the gripper control command in the initial motion command.

[0013] In a second aspect, embodiments of this application provide a computing device, including: a storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the robot reinforcement learning policy transfer and deployment method as described in the first aspect.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot reinforcement learning policy transfer and deployment method as described in the first aspect.

[0015] The robot reinforcement learning policy migration and deployment method, computing device, and computer-readable storage medium provided in this application embodiment include: configuring the parameters of the robot's controller in the target simulation environment according to the simulation parameter differences between the source simulation environment and the target simulation environment; constructing an enhanced observation vector based on the state data of the robot collected from multiple consecutive time steps in the target simulation environment; inputting the enhanced observation vector into the reinforcement learning policy trained in the source simulation environment to obtain initial action instructions; adaptively adjusting the initial action instructions based on the current task stage determined by a multi-stage task state machine to generate stage-adapted action instructions; triggering automatic transitions between stages based on the detection results of the robot's real-time physical state by the multi-stage task state machine; and using the configured controller to solve the stage-adapted action instructions into low-level joint control instructions and execute them to drive the robot to complete the target task. Thus, through dynamic parameter configuration, temporal enhanced perception, multi-stage adaptive decision-making, and collaborative execution, the method systematically solves the problems of overall performance degradation and poor adaptability caused by environmental differences, incomplete perception information, and rigid task logic when migrating reinforcement learning policies across simulation environments. This method enables high-fidelity, closed-loop migration verification of strategies from the source simulation environment to the target simulation environment, providing a reliable bridge for subsequent simulation to real-world deployment, significantly improving the robustness and adaptability of the migration, as well as increasing the task success rate and overall efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the robot reinforcement learning strategy transfer and deployment method provided in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the overall process of the robot reinforcement learning strategy transfer and deployment method provided in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the simulation environment and robot system initialization process in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the multimodal perception and feature construction process in an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram of the control logic and multi-stage task state machine adaptive control process in an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0023] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0024] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0025] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0026] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0027] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0028] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0029] See Figure 1 This application provides a robot reinforcement learning policy transfer and deployment method, which can be executed by a robot reinforcement learning policy transfer and deployment device provided in this application. This device can be implemented using software and / or hardware, such as computing devices like computers, high-performance workstations, or servers. Specifically, the method provided in this embodiment can be executed in a simulation environment hosted by a computing device. The method provided in this embodiment includes: S101, Configure the parameters of the robot's controller in the target simulation environment based on the differences in simulation parameters between the source simulation environment and the target simulation environment.

[0030] In this context, the source simulation environment refers to the simulation platform initially used to train the reinforcement learning policy, such as the IsaacLab platform; the target simulation environment refers to the simulation platform to which the policy needs to be migrated and validated or deployed, such as the MuJoCo platform. Differences in simulation parameters may include variations between the two simulation environments in at least one of the following physical modeling parameters: simulation step size, gravitational acceleration, joint damping characteristics, and friction coefficient. The controller generally refers to the underlying control module capable of converting target motion commands into specific joint force signals or joint torque signals; in a preferred embodiment of this application, it specifically refers to a proportional-derivative controller.

[0031] To eliminate or reduce the differences in basic physics and control system responses between the source and target simulation environments, and achieve seamless alignment across simulation platforms, the robot's controller parameters in the target simulation environment can be configured based on the differences in simulation parameters between the two environments. In this embodiment, dynamic parameter calibration technology can be used to achieve seamless alignment across simulation platforms. For example, a factor can be determined based on the difference between the simulation step size of the source and target simulation environments, and the parameters of the underlying controller can be dynamically scaled based on this factor. Simultaneously, compensatory configuration or online estimation optimization can be performed on physics engine parameters such as friction coefficient and mass properties to achieve more comprehensive physical consistency.

[0032] In one embodiment, the parameters of the robot's controller in the target simulation environment are configured based on the differences in simulation parameters between the source simulation environment and the target simulation environment, including: The time step scaling factor is determined based on the ratio of the simulation step size of the source simulation environment to the simulation step size of the target simulation environment. Based on the time step scaling factor, the proportional gain and derivative gain of the robot's proportional-derivative controller (PD controller) are dynamically scaled linearly in a proportional manner.

[0033] Among these parameters, the simulation step size is one of the most critical differences between the source and target simulation environments. Since the simulation step sizes differ across environments, directly using the same control parameters can lead to inconsistent system responses. In this embodiment, to map the controller behavior in the source simulation environment to the target simulation environment to produce the same dynamic effects, a time step scaling factor α can be calculated using the formula α = dt_target / dt_source. This factor is then used to dynamically scale the parameters of the underlying controller, where dt_source is the simulation step size of the source simulation environment and dt_target is the simulation step size of the target simulation environment. For example, for a proportional-derivative controller, the actual gain parameters can be adjusted using the following formula: Kp_actual = Kp_nom × α, Kd_actual = Kd_nom × α, where Kp_nom and Kd_nom are the nominal gains calibrated in the source simulation environment, Kp represents the proportional gain, and Kd represents the derivative gain. This scaling operation effectively compensates for system response differences caused by variations in the simulation frequency, ensuring control stability across simulation platforms.

[0034] Furthermore, the robot system can be initialized, including: configuring hierarchical PD parameters (high gain for large joints, moderate gain for small joints) according to the robot model (e.g., a 7-DOF robotic arm + gripper); setting joint torque limits to prevent overshoot and damage to the model; and constructing the observation space (25-dimensional: joint state, target pose, historical actions) and action space (8-dimensional: continuous actions of 7 joints + discrete actions of 1 gripper) for the reinforcement learning strategy. For example, high gains (Kp=400, Kd=15) are set for large joints (joints 1-4), moderate gains (Kp=300, Kd=10) are used for small joints (joints 5-7), and the gripper joint is configured separately (gripper_kp=100.0). Torque limits (τ_limit=50-30 N·m) are set for all joints to prevent overshoot. Additionally, the gain parameters corresponding to the joints can be dynamically adjusted using the formula Kp_actual=Kp_nom×(dt_target / dt_source) to achieve adaptive scaling based on the simulation step size, ensuring control stability at different simulation frequencies.

[0035] In this way, by linearly and dynamically scaling the gain of the proportional-derivative controller based on the simulation step size scaling factor, the stability and response characteristics of the control loop are kept consistent at different simulation frequencies, which fundamentally alleviates the strategy failure caused by the difference in simulation step size and ensures that the control performance does not degrade after migration.

[0036] S102, based on the state data of the robot collected from multiple consecutive time steps in the target simulation environment, construct an enhanced observation vector.

[0037] The enhanced observation vector refers to a state representation with higher dimensions and temporal context that integrates observation information from multiple current and historical moments. Addressing the limitation of traditional single-frame observations in providing dynamic information such as velocity and acceleration, this embodiment overcomes this deficiency by constructing an enhanced observation vector containing temporal information based on state data collected from multiple consecutive time steps of the robot.

[0038] Among them, at a high frequency of 200Hz, three types of data are synchronously collected through a dedicated interface function: joint position (7-dimensional, offset to reduce dimensionality), end effector pose (7-dimensional: position + quaternion pose) and target object position (3-dimensional).

[0039] In one embodiment, an enhanced observation vector is constructed based on state data collected from the robot at multiple consecutive time steps in the target simulation environment, including: In each control cycle corresponding to the first frequency, a fixed-dimensional current observation vector is generated based on the collected state data and historical information. The current observation vector is added to the tail of the preset queue, and the historical observation vector at the head of the preset queue is removed. The preset queue is a first-in-first-out queue of length N, used to store the observation vectors corresponding to N consecutive control cycles, where N is an integer greater than 1. The N observation vectors stored in the preset queue are concatenated in chronological order to form an enhanced observation vector.

[0040] Specifically, raw state data, including joint positions, end effector poses (position and attitude), and target object positions, are acquired at high frequencies (e.g., 200Hz) in the target simulation environment. During each high-level decision control cycle (e.g., 50Hz), a current observation vector is constructed based on the current and recently acquired data. Furthermore, a first-in-first-out queue is maintained as a sliding time window, continuously storing the observation vectors corresponding to the most recent N control cycles (e.g., N=3). When a decision is needed, these N observation vectors in the queue are concatenated in chronological order to form an enhanced observation vector. This enhanced observation vector implies the trend of state changes over time, providing implicit short-term memory capabilities for reinforcement learning strategies.

[0041] In each control cycle, the newly generated current observation vector is added to the tail of the queue, while the oldest observation vector at the head of the queue is removed, thus achieving a sliding window update. Furthermore, concatenating the N observation vectors in the queue yields an enhanced observation vector containing temporal information. In this way, by using a first-in, first-out queue to construct the enhanced observation vector through a sliding time window, the policy is provided with temporal awareness capabilities containing continuous historical information. This allows the policy to perceive trends in state changes (such as speed and direction of movement), overcoming the limitation of traditional single-frame observations in understanding dynamic processes. This significantly improves the prediction and decision-making accuracy of the policy in tasks requiring continuous interaction (such as tracking and grasping).

[0042] In one embodiment, a fixed-dimensional current observation vector is generated based on the collected state data and historical information, including: According to the predetermined dimension mapping rules, the joint positions, end effector poses, target object positions, and motion commands from the previous control cycle in the state data are fused into a fixed-dimensional basic observation vector. The basic observation vector is standardized, and the range of the standardized basic observation vector is clipped to obtain the current observation vector.

[0043] Specifically, firstly, according to a predetermined dimensionality mapping rule, the collected multimodal state data, such as joint positions, end-effector poses, and target object positions, along with the action commands from the previous control cycle (as historical information), are fused into a fixed-dimensional basic observation vector (e.g., 25 dimensions). This dimensionality mapping rule defines the position and representation of each data type within the vector. Then, to match the input distribution desired by the reinforcement learning strategy during training, this basic observation vector needs to be processed using a normalizer built into the reinforcement learning strategy (e.g., Z-score standardization). Finally, to prevent interference from simulation outliers, the standardized basic observation vector needs to be cropped (e.g., limiting the values ​​of each dimension to between [-1000, 1000]) to obtain the final observation vector sent to the queue.

[0044] In this way, the dimensional differences between data from different sources are eliminated, and the input distribution is aligned to the statistical characteristics desired during policy training. Meanwhile, outlier interference is prevented through pruning. This effectively avoids the problem of drastic performance degradation or divergence caused by input data distribution shifts in unfamiliar simulation environments, ensuring the stability of policy inference.

[0045] S103 enhances the reinforcement learning policy trained in the source simulation environment by inputting the observation vector and obtains the initial action command.

[0046] The constructed enhanced observation vectors are input into a reinforcement learning policy, such as a policy neural network, pre-trained in a source simulation environment (such as IsaacLab) using reinforcement learning algorithms (such as PPO, SAC, etc.). This reinforcement learning policy infers and outputs a preliminary, unadapted initial action command based on the combined current and historical state information. This initial action command typically includes the target position or velocity of each joint of the robot and the opening and closing commands of the grippers.

[0047] S104, based on the current task stage determined by a multi-stage task state machine, adaptively adjusts the initial action instructions and generates stage-adaptive action instructions; the multi-stage task state machine triggers automatic transitions between stages based on the detection results of the robot's real-time physical state.

[0048] A multi-stage task state machine is a logical control model that decomposes a complex, continuous task into several sub-stages and manages the transitions between stages based on real-time environmental feedback. A multi-stage task state machine can decompose a grasping and placing task into a logically clear sequence of sub-stages, such as approach, grasping, transport, and placement. Simultaneously, the multi-stage task state machine maintains a variable indicating the current task stage. The initial action commands output by the reinforcement learning policy are adjusted specifically based on the current task stage. For example, in the grasping stage, it might be necessary to strengthen the gripper's closing command. More importantly, the transitions between stages are not based on preset time but are entirely triggered by real-time physical states. For example, the condition for transitioning from the approach stage to the grasping stage is that the end effector is very close to the target object; the condition for transitioning from the grasping stage to the transport stage is that the object must be stably grasped for several consecutive control cycles. This intelligent switching based on physical feedback greatly improves the robustness of the task.

[0049] S105 uses the configured controller to calculate the stage-adapted motion instructions into low-level joint control instructions and execute them to drive the robot to complete the target task.

[0050] In one embodiment, the reinforcement learning strategy infers at a first frequency, the underlying joint control commands are executed at a second frequency higher than the first frequency, and the first frequency and the second frequency are coordinated by a frequency reduction synchronization mechanism.

[0051] In this embodiment, a high- and low-frequency layered control architecture is adopted. The high-frequency layer (e.g., 200Hz) runs the PD controller, responsible for real-time torque calculation and output of the joints, ensuring the accuracy and stability of the underlying control. The low-frequency layer (e.g., 50Hz) runs the reinforcement learning strategy, performing observation processing and action decision-making. That is, the inference of the reinforcement learning strategy (including observation construction, network forward propagation, and state machine decision-making) is performed periodically at a lower first frequency (e.g., 50Hz), and this period is called the control cycle (e.g., 20ms). The calculation and execution of the underlying joint control commands, i.e., the operation of the underlying controller and the updating of the physical simulation, are performed at a higher second frequency (e.g., 200Hz), and this high-frequency update cycle is the simulation time step (e.g., 5ms). The first and second frequencies are coordinated through a frequency reduction synchronization mechanism to ensure that the high and low frequency collaborative operation does not cause confusion.

[0052] Thus, by limiting strategic reasoning (decision-making) and underlying control execution to operate at different frequencies and coordinating them through a frequency reduction synchronization mechanism, the contradiction between real-time decision-making and control precision under a single-frequency architecture is resolved. This heterogeneous frequency control architecture ensures that high-level intelligent decision-making has sufficient computation time, while the underlying physical control can maintain a high-frequency, precise response, thereby balancing the system's intelligence and stability in complex tasks and ensuring reliable migration and execution.

[0053] In one embodiment, the first frequency is 50Hz, the second frequency is 200Hz, and the frequency reduction coefficient of the frequency reduction synchronization mechanism is 4.

[0054] Specifically, when the first frequency is 50Hz and the second frequency is 200Hz, the frequency reduction factor is 4. This means that for each policy inference (corresponding to one control cycle), the underlying layer will execute four complete control and physical updates (corresponding to four time steps). This ensures strict timing alignment between high-level decision-making and low-level execution, guaranteeing both real-time control and sufficient computation time for policy inference. It achieves an optimal balance between ensuring real-time control (5ms time step) and decision sufficiency (20ms control cycle), making it suitable for robot operation tasks requiring precise force control and effectively avoiding control jitter or decision delays caused by timing mismatches.

[0055] In one embodiment, the target task includes a grasping and placing task, and a multi-stage task state machine divides the grasping and placing task into at least four stages to be executed sequentially: an approach stage, a grasping stage, a transport stage, and a placing stage; the method further includes: When the distance between the robot's end effector and the target grasping point is detected to be less than or equal to a first distance threshold, the transition from the approach phase to the grasping phase is triggered. When the distance between the gripper and the target object is detected to be less than or equal to the second distance threshold within M consecutive control cycles, the transition from the gripping phase to the transport phase is triggered; M is an integer greater than 1. When the distance between the end effector and the target placement point is detected to be less than or equal to the third distance threshold, the transition from the transport phase to the placement phase is triggered.

[0056] The grasping and placement task can include information such as the target grasping point and target placement point for the target object to be grasped. When the distance between the robot's end effector and the target grasping point is less than or equal to a first distance threshold (e.g., 0.05m), it indicates that the robot's end effector has moved to the target grasping point where the target object is located. At this time, the transition from the approach phase to the grasping phase is triggered, i.e., the operation of grasping the target object is initiated. When the distance between the gripper and the target object is less than or equal to a second distance threshold (e.g., 0.03m) within M consecutive control cycles (M>1, e.g., 20), it indicates that the gripper has grasped the target object. At this time, the transition from the grasping phase to the transport phase is triggered, i.e., the operation of transporting the target object is initiated. This "counting-based de-jittering" mechanism avoids misjudgments caused by sensor noise, ensuring a firm grasp. When the distance between the end effector and the target placement point is less than or equal to a third distance threshold (e.g., 0.07m), it indicates that the target object has been transported to the target placement point. At this time, the transition from the transport phase to the placement phase is triggered, i.e., the operation of placing the target object is initiated. In this way, intelligent and modular management of complex operation tasks is achieved, which greatly enhances the system's ability to cope with uncertainties (such as slight slippage of objects), thereby significantly improving the success rate and robustness of complex continuous tasks (such as grasping and placing).

[0057] In one embodiment, the method further includes: during the gripping phase and / or transport phase, superimposing a constant holding torque or holding force on the gripper control command in the initial motion command.

[0058] Specifically, during the grasping and / or transport phases, a constant holding force or holding torque is automatically superimposed on the gripper control commands output by the reinforcement learning strategy. This force differs from the command issued by the reinforcement learning strategy to close the gripper; it is an additional static force designed to resist slippage of the object due to inertia or vibration during movement, significantly improving the reliability of grasping and handling. Thus, by superimposing a constant holding force on the gripper commands during the grasping and transport phases, the common problem of slippage due to inertia, vibration, or external disturbances during object handling is specifically addressed, significantly improving the physical reliability of grasping.

[0059] In summary, the above methods achieve high-fidelity, closed-loop migration verification of strategies from the source simulation environment to the target simulation environment, providing a reliable bridge for subsequent simulation to real-world deployment, significantly improving the robustness and adaptability of the migration, as well as increasing the task success rate and overall efficiency.

[0060] The method provided in this embodiment will be illustrated below with a specific example, taking the source simulation environment as the robot simulation software platform IsaacLab and the target simulation environment as the policy migration platform Mujoco as an example.

[0061] The devices involved in this embodiment include, but are not limited to, the following: 1) IsaacLab, a robot simulation software platform, mainly used for training reinforcement learning policies (hereinafter referred to as policies). It provides a high-fidelity physical simulation environment, a rich library of robot models and sensor simulations, and supports the training and evaluation of various reinforcement learning algorithms (such as PPO, SAC, DDPG, etc.). This platform needs to be able to support large-scale parallel training, automatic hyperparameter tuning, and real-time monitoring and analysis of policies; 2) Mujoco, a policy transfer platform, serving as the target simulation environment, providing accurate physical simulation and efficient computational performance for verifying and testing reinforcement learning policies transferred from IsaacLab. This platform needs to support common robot models and task scenarios and be compatible with mainstream deep learning frameworks.

[0062] See Figure 2 This is a schematic diagram of the overall process of the robot reinforcement learning policy transfer and deployment method provided in this embodiment. It mainly includes two parts: training environment and deployment environment. After the policy is processed by parameter configuration, observation processing and control logic and multi-stage state machine adaptive control, the Mujoco simulation interface will be brought up for visualization. The consistency of the actions of the two simulation platforms will be compared to complete the Sim2Sim verification. The specific details of these three parts are as follows.

[0063] 1. Parameter Configuration like Figure 3 As shown, the initialization of the simulation environment and robot system through parameter configuration includes the following operations: 1.1 Multi-frequency heterogeneous control architecture design A high-frequency and low-frequency layered control architecture was constructed, achieving decoupling between physical control and decision-making. The high-frequency layer (200Hz) focuses on the real-time joint torque output of the PD controller, while the low-frequency layer (50Hz) handles reinforcement learning policy inference. A frequency reduction mechanism with Decimation=4 ensures timing synchronization. Specifically, the architecture manages the control cycle through the Decimation Module class, enabling each policy inference to trigger four PD control cycles, effectively avoiding the timing conflict problem of traditional single-frequency control. This heterogeneous design ensures the real-time performance of the lower-level control (5ms step size) while providing sufficient computation time for higher-level decision-making (20ms cycle size), making it particularly suitable for grasping tasks requiring precise force control.

[0064] 1.2 Dynamic Consistency Parameter Configuration Seamless alignment across simulation platforms is achieved through dynamic parameter calibration technology. The core of this technology is the introduction of a time step scaling factor α = dt_target / dt_source, which dynamically scales the controller parameters (e.g., Kp' = α × Kp) based on the time step difference between the source simulation environment (IsaacLab) and the target simulation environment (MuJoCo). Simultaneously, physics engine parameters (friction coefficient, mass properties) are optimized based on KL divergence, and online estimation replaces traditional domain randomization to correct for environmental differences (e.g., gravity, inertia) in real time. This dynamic consistency maintenance technique significantly reduces the accumulated error in Sim2Sim migration, enabling the strategy to have better adaptability across heterogeneous simulation platforms.

[0065] 1.3 High-frequency PD controller parameter configuration A hierarchical PD control system was established by configuring refined parameters for the robot's seven joints. Large joints (joints 1-4) were given higher gains (Kp=400, Kd=15), while small joints (joints 5-7) used moderate gains (Kp=300, Kd=10). The gripper joint was configured separately (gripper_kp=100.0). Torque limits (τ_limit=50-30 N·m) were set for all joints to prevent overshoot. The key innovation lies in the adaptive scaling mechanism based on the simulation step size. The gain parameters are dynamically adjusted using the formula Kp_actual=Kp_nom×(dt_target / dt_nom) to ensure control stability at different simulation frequencies.

[0066] 1.4 Low-frequency strategy output and resource scheduling The strategy inference process is optimized by constructing a 25-dimensional observation space (joint state, target pose, historical actions) and an 8-dimensional action space (7-dimensional continuous joint control + 1-dimensional discrete gripper actions). A normalizer unifies the dimensions, and clip_actions limits the output range. An FPGA-inspired time multiplexing technique is employed to pipeline the matrix operations in PD control, optimizing the number of multipliers from 960 to 320, significantly reducing computational resource consumption.

[0067] 2. Observation and Processing like Figure 4 As shown, multimodal perception and feature construction through observation processing includes the following operations: 2.1 Multimodal Raw Data Acquisition and Real-time Extraction High-frequency data acquisition at 200Hz is achieved through three dedicated interface functions. Specifically, the `get_arm_obs` function is first used to read the real-time position information of the seven joints, retaining only the offset relative to the initial position while ignoring velocity information, effectively reducing the dimensionality of observation. Secondly, the `get_end_effector_pose` function obtains the 7-dimensional pose (position and quaternion) of the end effector, and rotation matrix transformation is used to ensure pose accuracy. Finally, the `get_object_pose` function is called to monitor the 3-dimensional spatial position of the target object. All data acquisition is strictly synchronized with the simulation physics engine to ensure timestamp consistency, providing accurate multimodal input for subsequent feature fusion.

[0068] 2.2 Temporal memory enhancement mechanism and multi-frame state stacking This embodiment innovatively introduces a sliding time window mechanism based on a deque double-ended queue, endowing the policy with short-term memory capabilities. The system initializes a deque structure of length frame_stack (configurable) to store observation data for consecutive time steps. Each control cycle (50Hz) pushes the current 25-dimensional observation vector to the tail of the queue while removing the oldest observation data, achieving rolling updates. When the time sequence depth is set to 3, the policy can obtain 75-dimensional joint observations including the current and the previous two time steps, enabling the policy to perceive state change trends (such as object trajectory and continuous joint movements). This design overcomes the problem of missing dynamic information in traditional single-frame observations, providing crucial temporal context for task phase transitions (such as approach → grasp → place).

[0069] 2.3 Multimodal Feature Fusion and Observation Vector Construction The design employs a dimensional mapping rule to fuse heterogeneous data into a unified 25-dimensional policy input vector. The specific structure includes: dimensions 0-6 storing joint position offsets; dimensions 7-9 representing the target grasping position; dimensions 10-12 representing the target placement position; dimensions 13-16 recording the quaternions of the target placement pose; and dimensions 17-24 retaining the 8-dimensional motion (7-dimensional arm motion + 1-dimensional gripper motion) from the previous time step. This multimodal alignment mechanism eliminates the dimensional differences between position, pose, and motion data.

[0070] 2.4 Normalized Pruning and Strategy Input Optimization The observation vectors are Z-score normalized using the policy's built-in normalizer (policy.normalizer), mapping each dimension of the data to a zero-mean, unit-variance distribution to accelerate policy convergence. A threshold constraint of clip_observations=1000.0 is then applied to prevent policy divergence caused by simulation outliers. Finally, the processed observation vectors are converted to PyTorch tensor format, supporting dimensional reconstruction (e.g., 1×75-dimensional input) when frame_stack>1, ensuring compatibility with pre-trained policy networks.

[0071] 3. Control Logic and Multi-Stage Task State Machine Adaptive Control like Figure 5 As shown, the control logic and multi-stage task state machine adaptive control include the following operations: 3.1 Dynamic Task Phase Division and State Machine Initialization The grasping task is finely divided into four consecutive phases: Approach, Grasp, Transport, and Place, with phase management implemented through state variables (task_phase). During simulation initialization, the system defaults to the "Approach" phase and dynamically generates control targets for each phase based on task objectives (such as the grasp position `grasp_pos` and the placement position `place_pos`). The target position for the Approach phase is the pre-grasp height offset above the grasp point. The multi-task state machine maintains the phase context through a unified interface, ensuring seamless transitions between phases and avoiding the policy rigidity problem caused by fixed target settings in traditional methods.

[0072] 3.2 Adaptive Observation Space Construction and Stage Dependency Adjustment The observation space is dynamically reconstructed during the current task phase to achieve intelligent adaptation of perceived content. During the approach phase, the observation vector prioritizes the grasping target position (grasp_pos) and the current end effector pose to guide the robot safely towards the object. Upon entering the grasping phase, the focus shifts to the object's position (obtained via get_object_pose) and the gripper state for precise grasping judgment. During the transport and placement phases, the observation of the placement target (place_pos) and the relative position of the object is strengthened. This dynamic observation mechanism adjusts the weight allocation of the observation vectors in real time through conditional logic.

[0073] 3.3 Intelligent State Transition Triggering and Condition Detection Mechanism This embodiment designs an automatic stage transition logic based on multiple threshold conditions to achieve intelligent switching between states. The transition trigger relies on real-time calculated geometric distance and stability metrics: the switch from the approach to the grasping stage is triggered when the distance between the end point and the grasping point (ee_to_grasp_dist) is less than a threshold (e.g., 0.05m); the transition from grasping to transportation requires continuous detection of successful grasping (via the check_grasp_success function, distance threshold 0.03m) for at least 20 steps to ensure stability; the transition from transportation to placement is controlled by the threshold distance between the end point and the placement point (ee_to_place_dist). High-frequency monitoring (50Hz) and a counting de-jitter mechanism (e.g., grab_success_count) are used to avoid false triggers and improve robustness.

[0074] 3.4 Enhanced and optimized control execution and gripping retention The system integrates PD control and gripping / holding mechanisms to achieve phase-adaptive execution. The arm joints track the target position dependent on the phase through the PD controller (pd_control function), and the torque is applied after being tau_limit trimming. The gripper control switches according to phase requirements—closing during the gripping phase (gripper_close_command) and opening during the placement phase. Furthermore, this embodiment employs an enhanced gripping and holding force method; that is, when the multi-stage task state machine enters the gripping or transport phase (is_grasping=True), the system automatically adds a holding force (grasp_force_maintain=5.0) to the gripper torque to prevent the object from slipping.

[0075] In summary, the methods provided in the above embodiments have the following innovations: 1) They possess a multi-frequency heterogeneous control architecture, designing a heterogeneous frequency control architecture with a high-frequency underlying PD controller and low-frequency strategy output, achieving perfect synchronization between high-frequency physical control and low-frequency decision output through a Decimation mechanism. 2) They possess a temporal memory-enhanced state observation mechanism, constructing a multi-frame state observation mechanism based on historical observation stacking, and achieving state information fusion within a time window through deque, providing implicit short-term memory capabilities for the strategy. 3) They construct an adaptive control method for robot grasping tasks based on a multi-stage state machine, solving the problems of poor adaptability and low task success rate caused by fixed target settings and a single observation space in traditional grasping strategies through dynamic task stage division, adaptive observation space construction, and intelligent state transition triggering conditions. This method decomposes the grasping task into four consecutive stages: approach, grasping, transport, and placement. It dynamically adjusts the observation content and control objectives based on the current stage and achieves automatic switching between stages through distance thresholds and grasping stability detection. Combined with a grasping and holding force enhancement mechanism, it ensures stable object handling, enabling reliable execution of complex desktop operation tasks and significantly improving the deployment success rate and robustness of reinforcement learning strategies in real robot systems. Based on these innovations, the method provided in this embodiment significantly improves the flexibility, safety, and generalization ability of robot reinforcement learning training. Specific technical effects include: 1) Highly efficient simulation environment transfer capability: Breaking through the dependence of traditional real-machine deployment on the physical environment, it achieves policy transfer between different simulation environments through Sim2Sim transfer technology, enabling rapid adaptation to multiple simulation platforms and greatly improving training flexibility and environmental adaptability. 2) Reduced training costs and resource consumption: Through simulation verification in a virtual simulation environment, this solution avoids the need for a large number of physical robots and physical sites, significantly reducing deployment costs from the high investment of traditional real-machine deployment and reducing hardware wear and maintenance costs. 3) Enhanced training safety and trial-and-error space: Allows for extensive trial-and-error training in a virtual environment without concerns about equipment damage or safety incidents. It enables the exploration of more high-risk operational strategies, improving the integrity and safety of strategy learning. 4) Enhanced policy generalization ability: Through alternating training and transfer across multiple simulation environments, more generalizable reinforcement learning policies can be generated, enabling robots to adapt to more diverse real-world scenarios and effectively broadening the application scope of the strategies.

[0076] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a computing device, such as... Figure 6 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 6The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 6 The memory 311 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, there can be one or more memories 311. When the processor 310 runs the computer program, the robot reinforcement learning policy transfer deployment method applied to the above-mentioned computing device is implemented.

[0077] The device may also include at least one network interface 312. The various components of the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to implement communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 313.

[0078] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0079] The memory 311 in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0080] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the robot reinforcement learning policy transfer and deployment method as described above.

[0081] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for transferring and deploying reinforcement learning policies in robots, characterized in that, The method includes: Based on the differences in simulation parameters between the source simulation environment and the target simulation environment, configure the parameters of the robot's controller in the target simulation environment; An enhanced observation vector is constructed based on the state data of the robot collected from multiple consecutive time steps in the target simulation environment. The enhanced observation vector is input into the reinforcement learning policy trained in the source simulation environment to obtain the initial action command; The initial action instructions are adaptively adjusted based on the current task stage determined by a multi-stage task state machine to generate stage-adaptive action instructions; the multi-stage task state machine triggers automatic transitions between stages based on the detection results of the robot's real-time physical state. Using the configured controller, the stage-adapted motion instructions are calculated into low-level joint control instructions and executed to drive the robot to complete the target task.

2. The method according to claim 1, characterized in that, The reinforcement learning strategy infers at a first frequency, the underlying joint control commands are executed at a second frequency higher than the first frequency, and the first frequency and the second frequency are coordinated through a frequency reduction synchronization mechanism.

3. The method according to claim 2, characterized in that, The first frequency is 50Hz, the second frequency is 200Hz, and the frequency reduction coefficient of the frequency reduction synchronization mechanism is 4.

4. The method according to claim 1 or 2, characterized in that, The step of configuring the parameters of the robot's controller in the target simulation environment based on the differences in simulation parameters between the source simulation environment and the target simulation environment includes: The time step scaling factor is determined based on the ratio of the simulation step size of the source simulation environment to the simulation step size of the target simulation environment. Based on the time step scaling factor, the proportional gain and derivative gain of the robot's proportional-differential controller are dynamically scaled linearly in a proportional manner.

5. The method according to claim 2 or 3, characterized in that, The process of constructing an enhanced observation vector based on state data collected from the robot at multiple consecutive time steps in the target simulation environment includes: In each control cycle corresponding to the first frequency, a current observation vector of fixed dimensions is generated based on the collected state data and historical information. The current observation vector is added to the tail of the preset queue, and the historical observation vector at the head of the preset queue is removed; the preset queue is a first-in-first-out queue of length N, used to store the observation vectors corresponding to N consecutive control cycles, where N is an integer greater than 1; The enhanced observation vector is formed by concatenating the N observation vectors stored in the preset queue in chronological order.

6. The method according to claim 5, characterized in that, The process of generating a fixed-dimensional current observation vector based on the collected state data and historical information includes: According to the predetermined dimension mapping rules, the joint positions, end effector poses, target object positions, and action commands from the previous control cycle in the state data are fused into a fixed-dimensional basic observation vector. The basic observation vector is standardized, and the value range of the standardized basic observation vector is clipped to obtain the current observation vector.

7. The method according to claim 1 or 2, characterized in that, The target task includes a grasping and placing task, and the multi-stage task state machine divides the grasping and placing task into at least four stages that are executed sequentially: an approach stage, a grasping stage, a transport stage, and a placing stage; the method further includes: When the distance between the robot's end effector and the target grasping point is detected to be less than or equal to a first distance threshold, the transition from the approach phase to the grasping phase is triggered. When the distance between the gripper and the target object is detected to be less than or equal to a second distance threshold within M consecutive control cycles, the transition from the gripping phase to the transport phase is triggered; M is an integer greater than 1. When the distance between the end effector and the target placement point is detected to be less than or equal to a third distance threshold, the transition from the transportation phase to the placement phase is triggered.

8. The method according to claim 7, characterized in that, The method further includes: During the gripping phase and / or the transport phase, a constant holding torque or holding force is superimposed on the gripper control command in the initial motion command.

9. A computing device, characterized in that, include: A storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the robot reinforcement learning policy transfer and deployment method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the robot reinforcement learning policy transfer and deployment method as described in any one of claims 1 to 8.

Images