A Robot Online Reinforcement Learning Control Method and System Based on Human-Machine Collaborative Intervention
By employing a distributed architecture with human-machine collaborative intervention and hybrid experience pool sampling, the real-time performance and stability issues in online reinforcement learning for real robots are resolved, improving learning efficiency and the success rate of complex operation tasks, and meeting the long-term online training needs in industrial and scientific research scenarios.
Patent Information
- Application Number
- CN202611381231.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-17
AI Technical Summary
Existing online reinforcement learning methods for real robots suffer from insufficient real-time performance and recoverability in training loops. Human intervention behaviors are difficult to effectively convert into learning signals, and high-value error correction samples are easily diluted in ordinary data, resulting in poor learning stability.
We adopt an online reinforcement learning control method for robots based on human-machine collaborative intervention. By decoupling the execution end, communication server end and learning end through a distributed architecture, we can detect and record human intervention signals in real time to form high-value samples. Combined with online and offline experience pools for mixed sampling, we use the soft actor-critic SAC algorithm for joint optimization to achieve online closed-loop training.
It significantly improves the real-time performance and stability of online training for real robots, enhances the learning efficiency of sparse reward tasks, increases the success rate of gripper-related complex operation tasks, and meets the long-term online training needs in industrial and scientific research scenarios.
Smart Images

Figure CN123103653A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control and artificial intelligence, specifically to a robot online reinforcement learning control method and system based on human-machine collaborative intervention. Background Technology
[0002] Reinforcement learning has made significant progress in the field of robot operation, gradually migrating from simulation environments to real robot systems. Industrial and service scenarios place higher demands on robots' ability to continuously perform long-sequential tasks such as grasping, handling, positioning, and assembly in dynamic, unstructured environments. Traditional control paradigms based on manual rules and precise modeling are costly to maintain and have poor transferability in complex contact tasks, making it difficult to cover diverse working conditions. Reinforcement learning, with its core idea of "automatically learning control strategies through interaction with the environment," provides a feasible technical path for solving these problems.
[0003] However, the application of existing reinforcement learning methods in real robot systems still faces the following problems: (1) Simulation environments can sample at high speed in parallel, while the sampling speed of real robots is constrained by mechanical execution cycles, communication delays, perception refresh rates and safety limits, resulting in a much smaller number of effective samples per unit time compared to simulation environments. Non-ideal factors such as sensor noise, calibration deviation, friction and hysteresis, gripper lag, visual occlusion, and illumination changes that are common in real systems make the strategy prone to instability due to distribution drift during training. At the same time, the reward signals for robot operation tasks are usually sparse, and successful feedback only appears at the end of the round, making it difficult for random exploration strategies to converge effectively within a limited interaction budget. (2) Real devices have clear safety boundaries, and exploration actions must meet speed, torque and pose limits. Overly aggressive trial and error not only affects training efficiency but may also lead to hardware damage. (3) Many robot operation tasks have mixed action characteristics: the end pose or joint increment is a continuous control quantity, while gripper opening and closing, mode switching, etc. are discrete decisions. When using a single continuous strategy to model all control variables, failure modes such as gripper opening and closing jitter, unstable action switching timing, and "in position but not gripped firmly" are prone to occur. If continuous control and discrete control are completely separated into multiple independent controllers, the system coupling complexity and parameter tuning difficulty will increase. (4) In the actual training process, human operators often take over briefly when the strategy is about to fail. These intervention segments contain high-value error correction information. However, in the existing schemes, such information is either diluted into the massive data as ordinary samples or used only for strategy initialization in the offline stage, failing to form a continuous gain for the online learning closed loop.
[0004] In addition, the engineering deployment of real robot reinforcement learning systems also faces engineering problems such as parameter synchronization, anomaly isolation, breakpoint recovery, and data version management, which need to be addressed in a comprehensive manner at the system level.
[0005] Chinese patent publication number CN122047374A proposes a method and system for training embodied intelligent agent policies based on preference reinforcement learning. This method first constructs a multi-source non-stationary environment preference dataset, then trains an environment dynamic predictor, and combines conditional mutual information and regret models to achieve decoupled joint optimization, extracting a perturbation-resistant pure task reward decoder. It then uses expectation regression to distill safe action policies, generating device control policies. After network convergence, it compiles instructions and deploys them to physical devices. However, this method completely couples the reinforcement learning training process into a single process, i.e., completing environment interaction, experience caching, and network parameter updates within the same program. Such schemes are prone to unstable control frequencies in real-world robotic tasks—the competition for computational power between policy inference and backpropagation reduces the real-time performance of the execution-side control; simultaneously, the system has weak fault tolerance, and once an anomaly occurs on the training side, the execution side is interrupted, affecting device safety and experimental continuity.
[0006] Chinese patent publication number CN122033996A proposes a training method for embodied intelligent robots based on a self-training architecture and an end-device chip. This method collects experience data through multimodal sensors, which is then converted into a unified semantic representation and generates an initial behavioral strategy by a perception-planning processing unit. A central coordination unit schedules the actuators to execute the strategy while simultaneously collecting environmental feedback. An optimization-learning processing unit calculates gradients and generates parameter adjustment instructions based on the strategy and feedback using online reinforcement learning. The central coordination unit synchronizes the instructions to the perception-planning processing unit, enabling online updates to the behavioral model. Although this method employs a distributed computing architecture, it primarily targets purely automated exploration scenarios and lacks a mechanism for human experts to intervene and correct errors in critical situations. Consequently, its learning efficiency is low for complex tasks such as gripper coordination, spatial obstacle avoidance, and precise end-device alignment.
[0007] In summary, existing technologies still have the aforementioned shortcomings in terms of sample efficiency, training security, hybrid motion control, and utilization of human-machine intervention information in real robot reinforcement learning. Summary of the Invention
[0008] The technical problem to be solved by this invention is how to address the shortcomings of existing online reinforcement learning methods for real robots, such as insufficient real-time performance and recoverability of training loops, difficulty in effectively converting human intervention into learning signals, and poor learning stability caused by the dilution of high-value error correction samples in ordinary data.
[0009] This invention solves the above-mentioned technical problems through the following technical means: an online reinforcement learning control method for robots based on human-machine collaborative intervention, applied in a distributed architecture including an execution end, a communication server end, and a learning end, comprising: The execution end loads the current strategy parameters, performs actions in a real robot environment and receives observations; it detects human intervention trigger signals in real time, and when triggered, switches the strategy action to a human action and records the intervention mark; each time step generates transition data including state, action, reward and the next state, and uploads it to the communication server. The communication server receives the transfer data and forwards it to the learning terminal. The learning end maintains an online experience pool and an offline experience pool, receives the transfer data, stores real-time interactive transfers in the online experience pool, and stores transfer data with the intervention tag as high-value samples in the offline experience pool; it extracts samples from the online and offline experience pools proportionally using a hybrid sampler, and splices them to form a training batch; based on the training batch, it uses a reinforcement learning algorithm for joint optimization, updates the policy network parameters, and sends the updated policy parameters to the communication server; The communication server receives the updated policy parameters and forwards them to the execution server; At the end of a round or a predefined safe point, the execution end reads and loads the updated policy parameters in a non-blocking manner to replace the current policy parameters and continues the interaction in the next round, forming an online closed-loop training.
[0010] Furthermore, the reinforcement learning algorithm is the soft actor-critic SAC algorithm, and the SAC algorithm includes a discrete Critic branch.
[0011] Furthermore, after acquiring the initial observations, the execution end preprocesses the observation data, including: cropping, scaling, format conversion, and normalization of the image data; key-value mapping and normalization of the state variables; adding batch dimensions and migrating them to the inference device; The transferred data also includes supplementary information, which contains a human intervention flag.
[0012] Furthermore, the data types stored in the offline experience pool include: historical teaching samples, high-value samples generated by manual intervention, or task prior data; The sampling ratio of the online experience pool to the offline experience pool in the hybrid sampler is dynamically adjusted according to the current training progress.
[0013] Furthermore, the joint optimization employs a joint loss function, the formula for which the joint loss function is calculated is:
[0014] in, For continuous Critic loss, For discrete Critic loss, For Actor loss, For temperature parameter loss, These are the discrete loss weighting coefficients.
[0015] Furthermore, the formula for calculating the continuous Critic loss in the joint loss function is as follows:
[0016] in, For the number of Critic integrations, Mean square error, For the first Parameters of a Critic network Indicates the current state. For action, As a reward, As a discount factor, The state is terminated. The parameters of the target Critic network are... For the next state, For the action to be taken by the Actor network in the next state, Temperature coefficient; The formula for calculating discrete Critic loss is:
[0017] in, For discrete actions, The optimal discrete action for the next state; The formula for calculating Actor loss is:
[0018] The formula for calculating temperature parameter loss is:
[0019] in, Let be the target entropy.
[0020] Furthermore, the non-blocking read and load specifically includes: The execution end reads the updated strategy parameters and determines whether the reading was successful; If the read is successful, then load the updated strategy parameters; If the read fails, the current strategy parameters will be used.
[0021] Furthermore, the manual intervention trigger signal is detected from the explicit switching mode input of the handle, keyboard, or master-slave operator; The reward value is calculated using an environmental reward wrapper chain calculation method. The reward function used in the chain calculation method is a sparse reward function, which includes: the basic step reward is set to a preset constant value, and when the operator inputs a mark success command, the final success reward is set to a preset positive constant value.
[0022] Furthermore, the learning terminal only triggers the current strategy parameter distribution operation when the time interval between the last distribution action and the current distribution action is not less than a preset time threshold.
[0023] This invention also provides an online reinforcement learning control system for robots based on human-machine collaborative intervention, comprising: The execution end, deployed on the real robot side, is used to load the current strategy parameters and execute actions in the environment; the execution end is configured with an intervention detection interface to detect human intervention trigger signals in real time, and when triggered, switch the strategy action to the human action and record the intervention mark; the execution end is also used to generate and upload transition data containing state, action, reward and next state at each time step. The communication server is connected to the execution terminal and is used to receive and forward the transfer data uploaded by the execution terminal, as well as to receive the updated policy parameters and forward them to the execution terminal. The learning end, connected to the communication server, is used to maintain an online experience pool and an offline experience pool. It receives the transfer data forwarded by the communication server, stores real-time interactive transfers in the online experience pool, and stores transfer data with the intervention tag in the offline experience pool. The learning end is also used to extract samples proportionally from the online and offline experience pools using a hybrid sampler, splice them to form a training batch, and perform joint optimization using a reinforcement learning algorithm based on the training batch to update the policy network parameters. The updated policy parameters are then sent to the communication server. The execution terminal is also used to read and load the updated policy parameters in a non-blocking manner at the end of a round or at a predefined safe point to replace the current policy parameters and continue the interaction in the next round, forming an online closed-loop training.
[0024] The advantages of this invention are: (1) This invention significantly improves the real-time performance and stability of online training of real robots by decoupling the Actor execution end and the Learner learning end in a distributed manner, and reduces the risk of blocking on the execution side.
[0025] (2) This invention transforms human error correction behavior into learnable signals by intervening in reward shaping and intervention in annotation mechanisms, thereby effectively improving the learning efficiency of sparse reward tasks.
[0026] (3) This invention improves the utilization rate of key samples and reduces training oscillations by using online / offline dual experience pools for mixed sampling.
[0027] (4) This invention improves the success rate of gripper-related composite operation tasks by jointly optimizing continuous and discrete actions.
[0028] (5) This invention meets the long-term online training needs in industrial and scientific research scenarios through parameter streaming synchronization and breakpoint recovery mechanism. Attached Figure Description
[0029] Figure 1 This is a flowchart of the robot online reinforcement learning control method based on human-machine collaborative intervention according to Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the interaction, intervention, takeover, and trajectory feedback of the Actor in Embodiment 1 of the present invention in a real robot environment. Figure 3 This is a flowchart illustrating the construction and hybrid sampling process of the online and offline experience pools for the Learner in Embodiment 1 of the present invention. Figure 4 This is a timing diagram of the strategy parameter streaming, execution-end hot update, and training closed loop in Embodiment 1 of the present invention. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] To facilitate understanding of the technical solution of the present invention by those skilled in the art, the technical solution of the present invention will now be further described in conjunction with the accompanying drawings.
[0032] Example 1 like Figure 1 As shown, an online reinforcement learning control method for robots based on human-robot collaborative intervention is applied to a distributed architecture comprising an execution end, a communication server, and a learning end. The method includes the following steps: S1. The execution end loads the current strategy parameters, executes actions in a real robot environment and receives observations; it detects human intervention trigger signals in real time, and when triggered, it switches the strategy action to a human action and records the intervention mark; each time step generates transition data including state, action, reward and the next state, and uploads it to the communication server.
[0033] Specifically, the online interaction and human-machine intervention process at the execution end is as follows: Figure 2The execution process adopts a closed loop of "sampling-judgment-execution-feedback," and each round can independently complete strategy execution and parameter hot updates. The specific process is as follows: S1.1 Environment Initialization and Observation Acquisition At the start of the round, the executor calls the environment's reset interface to obtain initial observations (obs) containing multimodal data. These observations include: robot joint states, end-effector states, multi-channel image information, and other additional state variables.
[0034] The execution end converts observations into a unified policy inference input format.
[0035] Image preprocessing: For the original uint8 format HWC (height × width × channel) image, it is first cropped and scaled using ImageCropResizeWrapper (e.g., to unify the resolution to 128 × 128 pixels); then the data type is converted to float32 and normalized to the [0,1] range, and the dimension order is adjusted to CHW format; finally, according to the key name mapping rules, the processed image data is mounted to the observation.images.{camera_name} field.
[0036] State variable processing: Map state variables such as joint positions to observation.state floating-point tensors.
[0037] Batch dimension compatibility: Add batch dimensions to single-environment inference data using BatchCompatibleWrapper (i.e., perform the unsqueeze(0) operation).
[0038] Device migration: Migrate the processed tensors to policy inference devices (such as CUDA GPUs).
[0039] Policy-side normalization: Before inference, based on the pre-loaded dataset_stats statistics, MEAN_STD normalization (i.e., subtracting the mean and dividing by the standard deviation) is used for image data; MIN_MAX normalization (i.e., linear scaling to the [-1, 1] interval) is used for state variables and motion variables. This ensures that the final input is consistent with the input_features definition of the SAC policy. The dataset_stats statistics are obtained as follows: Before system deployment, observation data covering the entire state space of the task (including image sequences from each camera viewpoint, robot joint positions, and end-effector poses) is collected offline. The mean, standard deviation (images), minimum, and maximum values (state variables) of each observation modality are calculated, and the calculation results are serialized and stored as a configuration file loaded along with the policy model. This statistical information remains fixed during online training and is not updated online.
[0040] S1.2, Strategy Reasoning and Action Generation Once the number of online interaction steps reaches a preset learning threshold, the policy network outputs actions based on preprocessed observations (obs). During the warm-up phase, random actions can be used to increase exploration coverage. For tasks involving a mixture of continuous and discrete action dimensions, the policy network adopts a multi-head output architecture: the backbone feature extraction layer of the Actor network is shared, connecting the continuous action head and the discrete action head respectively; the continuous action head outputs torque commands in the dimension of the robot joint degrees of freedom, which are mapped to the [-1,1] interval after tanh activation; the discrete action head outputs discrete commands (such as grasp / release), which are obtained by taking argmax after softmax activation. The policy network outputs a mixed action vector containing continuous torque and discrete commands (such as grasp / release) according to a predefined unified action format.
[0041] S1.3, Intervention Determination and Action Takeover The execution end monitors input signals from peripherals such as gamepads, keyboards, or master-slave controllers in real time. In this embodiment, manual intervention adopts an explicit switching mode, which requires the operator to actively press a specific key on the keyboard (such as the "Space" key) to trigger intervention.
[0042] The system monitors the human intervention flag in real time. When the human intervention flag is detected as true, the system immediately overrides the action output by the policy network with the human action and writes the human intervention field (is_intervention=True) into the metadata of this step, thus transforming human error correction from a temporary operation into a structured signal that can be continuously incorporated into RL training. If no intervention signal is detected, the policy action continues to be executed. This process is similar to... Figure 2 The decision branches are consistent.
[0043] Specifically, if manual intervention is triggered during the warm-up phase, the system will also overwrite the random action with the manual action and mark the step as an intervention sample. For intervention samples generated during the warm-up phase, an additional "warmup_intervention=True" mark will be added when uploading them to the execution end, so that the learning end can decide whether to include them in the high-value sample set according to the configuration when storing them in the offline experience pool, thus avoiding the pollution of the offline experience pool by the takeover data from the low-quality random exploration phase.
[0044] S1.4, Environment Stepping and State Transition Construction The execution end calls the environment's step interface to obtain the next observation, reward value, end flag, and additional information.
[0045] Specifically, reward value Chained calculations are performed using an environmental reward wrapper: the base step reward is set to 0.0 per step; when the operator presses a preset success marker key (such as the "s" key), a final success reward is triggered, with r=1.0. Based on this, an intervention mini-reward mechanism is introduced: when is_intervention=True for that step, a small positive reward (such as +0.02) is added. That is, the actual reward calculation formula is:
[0046] in, As a reward for completing the task, when the task is successful. ,otherwise ; To intervene in the reward, when is_intervention=True for this step. ,otherwise .
[0047] This design decouples the sparse task reward (0 / 1) from the intervention reward (+0.02): the sparse task reward tells the policy whether the task was ultimately successful; the intervention reward only tells the policy that "the human error correction action was better locally." With this decoupling, the policy learns error correction behavior, rather than waiting for human intervention. Unlike existing technologies that score the entire trajectory or convert clicks into auxiliary rewards, this scheme applies intervention rewards gradually and conditionally, without polluting the final signal.
[0048] Furthermore, during online learning, the value of completing a task through the strategy itself differs from the reward of completing a task through extensive intervention. Preferably, at the end of a round, if the task is successful but the overall intervention rate exceeds a preset threshold (e.g., 30%), the success reward is multiplied by a discount factor (e.g., 0.5). This distinguishes the value of "autonomous completion" from "reinforced completion," encouraging the strategy to gradually reduce its reliance on human intervention.
[0049] Subsequently, the execution end assembles a standard transition sample, which includes the current state, the action performed, the reward value, the next state, the round termination flag, and additional information fields recording intervention markers. If human intervention occurred in this round, the intervention marker must be stored in the supplementary information field and uploaded along with the sample.
[0050] S1.5, Statistical Feedback and Parameter Hot Update At the end of the round or at a specified synchronization time, the executor uploads statistical information such as the cumulative reward, human intervention rate, and strategy reasoning frequency of the current round to the communication server.
[0051] Subsequently, the execution end performs a non-blocking parameter hot update process: the system attempts to read the latest policy parameters from the parameter queue sent by the communication server in a non-blocking manner. If the read is successful (i.e., the queue is not empty and the verification passes), the new parameters are loaded at a predefined safe update point; if the read fails (e.g., the queue is empty or the network times out), the current old policy parameters are used. This design ensures that the high-frequency real-time performance of the robot control loop is not affected by network fluctuations, realizes online hot updates, and supports seamless execution in the next round.
[0052] S2. The communication server receives the transfer data and forwards the transfer data to the learning terminal.
[0053] S3. The learning end maintains an online experience pool and an offline experience pool, receives the transfer data, stores real-time interactive transfers in the online experience pool, and stores transfer data with the intervention tag as high-value samples in the offline experience pool; it extracts samples from the online experience pool and the offline experience pool proportionally using a hybrid sampler, and splices them to form a training batch; based on the training batch, it uses a reinforcement learning algorithm for joint optimization, updates the policy network parameters, and sends the updated policy parameters to the communication server.
[0054] Specifically, to improve the utilization rate of key samples, this method adopts a parallel management approach of online and offline experience pools, and the training batch is formed by a mixture of the two types of experience.
[0055] S3.1 Construction of the online experience pool ReplayBuffer_online The real-time interactive transfer uploaded by the execution end first enters the online experience pool. The online experience pool reflects the current policy distribution and can provide training samples of the "latest error pattern" and the "latest success pattern" in a timely manner.
[0056] S3.2 Construction of the offline experience pool ReplayBuffer_offline The online and offline experience pools are managed separately throughout the training process, and offline data is not merged into the online pool. The offline experience pool contains historical teaching samples, high-value samples from manual intervention, or task prior data. Historical teaching samples are task demonstration data collected before system deployment via remote operation or script control, and are written offline to the offline experience pool in a standard transfer format. High-value samples from manual intervention are transfer data where the intervention flag is true during online interaction. The criterion for determining whether a transfer data point is a high-value sample is the intervention flag in the transfer data. The determination logic is as follows: when the intervention flag is true and not marked during the warm-up phase, it indicates that the step was manually taken over, including expert teaching or error correction actions, and has higher learning value than the policy autonomous exploration step. Therefore, it should be persistently retained in the offline experience pool for repeated use by the hybrid sampler. Offline samples can provide stable learning leads in the early stages of training and continuously provide error correction references in the later stages of training.
[0057] S3.3, Hybrid Sampler Construction of Training Batches like Figure 3 As shown, the hybrid sampler samples from both the online and offline experience pools, concatenating them to form a training batch B = B_online∪B_offline. By adjusting the sampling ratio, a balance can be achieved between "rapidly adapting to the current distribution" and "maintaining high-quality priors." Unlike existing technologies that merge teaching and intervention data into a single training set for incremental fine-tuning during the imitation learning phase, this method's "union" applies to the replay hybrid sampling of SAC during online reinforcement learning training. The two pools remain independent, and intervention samples are dynamically written into the offline pool to prevent them from being overwhelmed by ordinary online data. In this embodiment, the default sampling ratio is 1:1. As a preferred implementation, the sampling ratio can be dynamically adjusted according to the current training steps. In the early stages of training (e.g., when the number of training steps is less than 5 x 10), 5 (Step 1) Set the sampling ratio of offline samples to online samples to 7:3 to utilize expert data in the offline pool to accelerate convergence; During the middle of training (e.g., when the number of training steps is between 5 x 10), 5 Up to 2×10 6 (Step), linearly transitioning to a 1:1 ratio; In the later stages of training (e.g., when the number of training steps is greater than 2 × 10), 6 (Step), setting the sampling ratio to 3:7, to enable the strategy to finely adapt to the current environmental distribution.
[0058] S3.4 Data Preprocessing and Input Specifications Before input, mixed batches undergo NaN filtering, tensor device migration, dimension alignment, and field integrity checks. The preprocessed training batches are then input into the SAC optimization module to prevent anomalous data from contaminating the training process.
[0059] S3.5, Multi-branch Joint Optimization Based on SAC In this embodiment, the Soft Actor-Critic (SAC) algorithm framework is used as the main optimizer. This method adopts a single-policy architecture, eliminating the interaction between offline and online policies. During each optimization round, multiple Critic network updates are first performed based on the UTD (Update-To-Data) ratio: the temporal difference (TD) target value is calculated using the target network, and the parameters of the online Critic network are updated by minimizing the mean squared error (MSE) loss; subsequently, Actor network updates and temperature parameter updates are performed according to a preset policy update frequency to achieve a balance between maximizing expected reward and maintaining policy entropy (i.e., exploratory nature).
[0060] For discrete motion dimensions such as the opening and closing of robot grippers, this invention improves the standard SAC framework by extending it with a discrete Critic branch. This branch decouples the value estimation of discrete motions from the continuous motion flow and achieves continuous-discrete hybrid motion space optimization through joint loss training, reducing gripper motion jitter and false trigger probability.
[0061] The continuous Critic branch adopts a standard SAC dual-Q network structure (integration number is...). Input state and combined action (Covering the dimension of continuous actions), outputting the value of continuous actions. ,in For the first Parameters of a Critic network For network indexing.
[0062] The network structure of the discrete Critic branch is as follows: Input state features and discrete actions The value estimate corresponding to the discrete action is output through an independent fully connected network. .
[0063] Calculate the joint loss function based on training batches:
[0064] in, For continuous Critic loss, For discrete Critic loss, For Actor loss, For temperature parameter loss, These are the discrete loss weighting coefficients.
[0065] The formula for calculating continuous Critic loss is:
[0066] in, For the number of Critic integrations, Mean square error, For the first Parameters of a Critic network Indicates the current state. For action, As a reward, As a discount factor, The state is terminated. The parameters of the target Critic network are... For the next state, For the action to be taken by the Actor network in the next state, This is the temperature coefficient.
[0067] The formula for calculating discrete Critic loss is:
[0068] in, Mean square error, For the current discrete Critic network, the current state and discrete actions Predicted Q value, For the target discrete Critic network, the next state All possible discrete actions The Q value of the evaluation, To select the target Q value corresponding to the optimal discrete action in the next state, For discrete actions, The optimal discrete action for the next state is determined by this formula. This formula trains a discrete Critic network to accurately evaluate the value of discrete actions (such as gripper gripping / releasing) by minimizing the mean square error between the predicted Q value and the temporal difference (TD) target value.
[0069] The formula for calculating Actor loss is:
[0070] in, The continuous and discrete joint action distribution output by the Actor network. This is the minimum Q-value for the joint evaluation of continuous and discrete Critic. This formula incorporates the discrete action head into the policy gradient optimization, achieving overall optimization of the continuous-discrete hybrid action space under a single policy, and ensuring that discrete action dimensions (such as gripper grab / release) can obtain updated signals through the policy gradient.
[0071] The formula for calculating temperature parameter loss is:
[0072] in, Let be the target entropy.
[0073] S4. The communication server receives the updated policy parameters and forwards them to the execution server.
[0074] S5. At the end of the round or at a predefined safe point, the execution end reads and loads the updated policy parameters in a non-blocking manner to replace the current policy parameters and continues the interaction of the next round, forming an online closed-loop training.
[0075] Specifically, this invention employs an "asynchronous sampling, quasi-synchronous update" mechanism, with the training closed-loop timing as follows: Figure 4 As shown.
[0076] S5.1, Asynchronous Data Upload The execution end uploads state transition samples to the communication server. These state transition samples include: the current observation, the actual action performed (or a manual action if human intervention occurs), the reward value, the next observation, a termination flag, and supplementary information (including intervention markers, etc.). Upon receiving the data, the communication server immediately forwards it to the learning end. The execution end can continue its subsequent control loop without waiting for the learning end to finish processing, thus achieving asynchronous decoupling between sampling and training.
[0077] S5.2 Parameter Distribution and Delay Control After completing gradient updates, the learning end checks the parameter push conditions. To prevent frequent updates from causing load fluctuations on the execution end, a preset time threshold of 4 seconds is set. This means that the latest policy parameters are only sent to the communication server if the time interval since the last parameter release is at least 4 seconds. If multiple new parameters are generated within 4 seconds, the learning end retains only the latest version of the policy parameters and sends them all at once the time threshold condition is met again. The communication server pushes data to the execution end via a streaming transmission channel.
[0078] S5.3, Safepoint Non-blocking Loading The executor loads new parameters at the end of a round or at a predefined safe point (such as when the robot is stationary). Loading is non-blocking: the executor asynchronously reads the parameter queue. If multiple parameters exist in the queue, only the latest version is read. If the new parameter is successfully retrieved, it is loaded and replaced at the safe point; otherwise, the current parameter is used. This mechanism ensures that policy updates are completed during non-critical control moments, thus avoiding motion jitter.
[0079] S5.4 Fault-Tolerant Recovery and Continuous Closed Loop The execution end continues the next round of environmental interaction, while the learning end continues training based on new data. Both sides form a continuous closed loop of "sampling → training → synchronization" through the communication server. The system supports a checkpoint-based recovery mechanism, where each checkpoint includes policy network weights, optimizer state, complete training configuration, and data snapshots of the online and offline experience pools. In the event of a disconnection or system crash, the system can load the most recent checkpoint to restore the training state, ensuring the reliability and scalability of long-term online training.
[0080] Through the above steps, this invention realizes a sustainable, recoverable, and scalable online reinforcement learning training process in real-world robotic scenarios. Human-robot intervention is transformed into structured, progressively labeled training signals (is_intervention). Decoupled sparse task rewards (0 / 1) and small intervention rewards (+0.02) guide the policy to learn error-correcting behavior. Simultaneously, a dual-pool independent parallel and hybrid replay mechanism under a single policy is employed to dynamically write intervention samples, preventing high-value error-correcting data from being overwhelmed by the online data stream, achieving a balance between adapting to the current distribution and preserving error-correcting priors.
[0081] Example 2 Based on Example 1, Example 2 also provides an online reinforcement learning control system for robots based on human-machine collaborative intervention, including: The execution end, deployed on the real robot side, is used to load the current strategy parameters and execute actions in the environment; the execution end is configured with an intervention detection interface, which is used to detect human intervention trigger signals in real time, and when triggered, switch the strategy action to the human action and record the intervention mark; the execution end is also used to generate and upload transition data containing state, action, reward and next state at each time step.
[0082] Specifically, it includes: a preprocessing unit, used to preprocess the observation data after acquiring the initial observations, the preprocessing including: cropping, scaling, format conversion and normalization of image data; key-value mapping and normalization of state variables; adding batch dimensions and migrating them to the inference device.
[0083] The metadata encapsulation unit is used when assembling transfer data, which also includes supplementary information. This unit is used to write human intervention flags into the supplementary information so that the learning end can identify high-value samples.
[0084] The communication server is connected to the execution terminal and is used to receive and forward the transfer data uploaded by the execution terminal, as well as to receive the updated policy parameters and forward them to the execution terminal.
[0085] Specifically, the explicit intervention detection unit detects the manual intervention trigger signal from the explicit switching mode input of the handle, keyboard, or master-slave operator.
[0086] The reward calculation unit is used to employ an environmental reward wrapper chain calculation method. The reward function used in the chain calculation method is a sparse reward function, which includes: setting the basic step reward to a preset constant value, and setting the final success reward to a preset positive constant value when the operator inputs a mark success command.
[0087] The learning end, connected to the communication server, is used to maintain an online experience pool and an offline experience pool. It receives the transfer data forwarded by the communication server, stores real-time interactive transfers in the online experience pool, and stores transfer data with the intervention tag in the offline experience pool. The learning end is also used to extract samples proportionally from the online and offline experience pools using a hybrid sampler, splice them to form a training batch, and perform joint optimization using a reinforcement learning algorithm based on the training batch to update the policy network parameters. The updated policy parameters are then sent to the communication server.
[0088] Specifically, the data types stored in the offline experience pool include: historical teaching samples, high-value samples generated by human intervention, and task prior data.
[0089] The hybrid sampling unit has a built-in hybrid sampler, and the sampling ratio of the online experience pool to the offline experience pool in the hybrid sampler is dynamically adjusted according to the current training progress.
[0090] The reinforcement learning algorithm is the soft actor-critic SAC algorithm, and the SAC algorithm contains discrete action branches.
[0091] The formula for calculating the joint loss function is:
[0092] in, For continuous Critic loss, For discrete Critic loss, For Actor loss, For temperature parameter loss, These are the discrete loss weighting coefficients.
[0093] The formula for calculating continuous Critic loss is:
[0094] in, For the number of Critic integrations, Mean square error, For the first Parameters of a Critic network Indicates the current state. For action, As a reward, As a discount factor, The state is terminated. The parameters of the target Critic network are... For the next state, For the action to be taken by the Actor network in the next state, Temperature coefficient; The formula for calculating discrete Critic loss is:
[0095] in, For discrete actions, The optimal discrete action for the next state; The formula for calculating Actor loss is:
[0096] The formula for calculating temperature parameter loss is:
[0097] in, Let be the target entropy.
[0098] The learning terminal triggers the current strategy parameter distribution operation only when the time interval since the last distribution action is not less than a preset time threshold.
[0099] The execution terminal is also used to read and load the updated policy parameters in a non-blocking manner at the end of a round or at a predefined safe point to replace the current policy parameters and continue the interaction in the next round, forming an online closed-loop training.
[0100] Specifically, the non-blocking parameter hot update unit, used for non-blocking reading and loading, includes: The execution end reads the updated strategy parameters and determines whether the reading was successful; If the read is successful, then load the updated strategy parameters; If the read fails, the current strategy parameters will be used.
[0101] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A robot online reinforcement learning control method based on human-robot collaborative intervention, characterized in that, It is applied in a distributed architecture that includes an execution end, a communication server end, and a learning end end, including: The execution end loads the current strategy parameters, performs actions in a real robot environment and receives observations; it detects human intervention trigger signals in real time, and when triggered, switches the strategy action to a human action and records the intervention mark; each time step generates transition data including state, action, reward and the next state, and uploads it to the communication server. The communication server receives the transfer data and forwards it to the learning terminal. The learning end maintains an online experience pool and an offline experience pool, receives the transfer data, stores real-time interactive transfers in the online experience pool, and stores transfer data with the intervention tag as high-value samples in the offline experience pool; it extracts samples from the online and offline experience pools proportionally using a hybrid sampler, and splices them to form a training batch; based on the training batch, it uses a reinforcement learning algorithm for joint optimization, updates the policy network parameters, and sends the updated policy parameters to the communication server; The communication server receives the updated policy parameters and forwards them to the execution server; At the end of a round or a predefined safe point, the execution end reads and loads the updated policy parameters in a non-blocking manner to replace the current policy parameters and continues the interaction in the next round, forming an online closed-loop training.
2. The human-machine collaboration intervention based robot online reinforcement learning control method according to claim 1, characterized in that, The reinforcement learning algorithm is the soft actor-critic SAC algorithm, and the SAC algorithm includes a discrete Critic branch.
3. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 1, characterized in that, After acquiring the initial observations, the execution terminal preprocesses the observation data, including: cropping, scaling, format conversion, and normalization of the image data; key-value mapping and normalization of the state variables; adding batch dimensions and migrating them to the inference device. The transferred data also includes supplementary information, which contains a human intervention flag.
4. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 1, characterized in that, The data types stored in the offline experience pool include: historical teaching samples, high-value samples generated by manual intervention, or task prior data; The sampling ratio of the online experience pool to the offline experience pool in the hybrid sampler is dynamically adjusted according to the current training progress.
5. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 1, characterized in that, The joint optimization employs a joint loss function, the formula for which is: in, For continuous Critic loss, For discrete Critic loss, For Actor loss, For temperature parameter loss, These are the discrete loss weighting coefficients.
6. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 5, characterized in that, The formula for calculating the continuous Critic loss in the joint loss function is as follows: in, For the number of Critic integrations, Mean square error, For the first Parameters of a Critic network Indicates the current state. For action, As a reward, As a discount factor, The state is terminated. The parameters of the target Critic network are... For the next state, For the action to be taken by the Actor network in the next state, Temperature coefficient; The formula for calculating discrete Critic loss is: in, For discrete actions, The optimal discrete action for the next state; The formula for calculating Actor loss is: The formula for calculating temperature parameter loss is: in, Let be the target entropy.
7. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 1, characterized in that, The non-blocking read and load specifically includes: The execution end reads the updated strategy parameters and determines whether the reading was successful; If the read is successful, then load the updated strategy parameters; If the read fails, the current strategy parameters will be used.
8. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 1, characterized in that, The manual intervention trigger signal is detected from the explicit switching mode input of the handle, keyboard, or master-slave operator; The reward value is calculated using an environmental reward wrapper chain calculation method. The reward function used in the chain calculation method is a sparse reward function, which includes: the basic step reward is set to a preset constant value, and when the operator inputs a mark success command, the final success reward is set to a preset positive constant value.
9. The robot online reinforcement learning control method based on human-machine collaborative intervention according to claim 1, characterized in that, The learning terminal triggers the current strategy parameter distribution operation only when the time interval since the last distribution action is not less than a preset time threshold.
10. A robot online reinforcement learning control system based on human-machine collaborative intervention, characterized in that, include: The execution end, deployed on the real robot side, is used to load the current strategy parameters and execute actions in the environment; the execution end is configured with an intervention detection interface to detect human intervention trigger signals in real time, and when triggered, switch the strategy action to the human action and record the intervention mark; the execution end is also used to generate and upload transition data containing state, action, reward and next state at each time step. The communication server is connected to the execution terminal and is used to receive and forward the transfer data uploaded by the execution terminal, as well as to receive the updated policy parameters and forward them to the execution terminal. The learning end is connected to the communication server and is used to maintain the online experience pool and the offline experience pool. It receives the transfer data forwarded by the communication server, stores the real-time interactive transfer into the online experience pool, and stores the transfer data with the intervention mark into the offline experience pool. The learning end is also used to extract samples from the online experience pool and the offline experience pool in proportion through a hybrid sampler, splice them to form a training batch, and perform joint optimization based on the training batch using a reinforcement learning algorithm to update the policy network parameters, and send the updated policy parameters to the communication server. The execution terminal is also used to read and load the updated policy parameters in a non-blocking manner at the end of a round or at a predefined safe point to replace the current policy parameters and continue the interaction in the next round, forming an online closed-loop training.
Citation Information
Patent Citations
Self-training architecture and end-side chip-based intelligent robot training method and system, terminal and storage medium
CN122033996A
Personal agent strategy training method and system based on preference reinforcement learning
CN122047374A
