Self-adaptive reward human-in-loop robot real machine reinforcement learning method

Abstract

The invention belongs to the technical field of robot reinforcement learning and man-machine interaction intelligent control, and particularly relates to a self-adaptive rewarding human-in-loop robot real machine reinforcement learning method. The method comprises the following specific steps: constructing a reward model used for supervising the execution of a robot internal strategy network, training by using a pre-training data set labeled by a stage progress label, and putting the trained data set into a robot; when a judgment result output by the reward model does not accord with reality, inputting a discrete feedback signal by an operator, and recording the discrete feedback signal and a timestamp corresponding to the discrete feedback signal as a reward correction data set; and according to the reward correction data set, carrying out backtracking annotation to expand sparse feedback into a dense track so as to update a reward model.

Description

Technical Field

[0001] This invention belongs to the field of robot reinforcement learning and human-computer interaction intelligent control technology, and specifically relates to a real-machine reinforcement learning method for human-in-the-loop robots with adaptive rewards. Background Technology

[0002] In existing technologies, researchers have attempted various methods to address issues such as sparse rewards, difficult reward design, and reward invalidation due to distribution shifts in real-world reinforcement learning environments. However, these methods still have significant limitations. The following are some typical existing solutions: (1) Direct reward inference method based on static visual model Many studies have attempted to use pre-trained visual language models (VLMs) or visual classifiers to directly infer task success or output staged scores from images or video clips, replacing manual reward design. For example, RoboCLIP learns reward signals through a small number of demonstrations and can generalize to some extent to new tasks; GCR uses contrastive video learning to infer rewards from target demonstrations; and Adapt2Reward further improves the semantic consistency of rewards through visual language models. However, these methods are all trained once offline and remain static after deployment, unable to adjust in real time according to changes in the robot's exploration in the real environment. When visual conditions change or unseen states appear, reward estimation is prone to shift, leading to unstable policy training.

[0003] (2) Reward model learning method based on demonstration videos or preference learning Another type of research utilizes video demonstrations, preference comparisons, or time progress annotations to construct reward models. For example, Open-X-Embodiment-based methods learn general semantic rewards through large-scale video demonstrations; ReWiND infers task stage progress through linguistic conditional video comparisons; and inverse reinforcement learning (IRL) and event-based inverse control methods attempt to extract reward structures from offline demonstrations. However, because these methods rely on large-scale offline collected demonstrations, they struggle to cover the diverse variations in real-world environments; furthermore, the models lack online update capabilities, leading to reward estimation drift when the robot explores unseen states.

[0004] (3) Human-in-the-loop correction method Some methods ensure the safety of reinforcement learning in real-world robots by introducing real-time human intervention. For example, HIL-SERL allows operators to take over control when the policy deviates from the correct trajectory, thereby improving policy stability and sample efficiency. With the aid of human-interventional data, policy learning can avoid dangerous behaviors and converge more effectively. However, these methods primarily focus on action alignment, i.e., adjusting the policy itself by providing "correct actions" by humans, rather than updating the reward model. Since the reward remains static and may contain misjudgments, even with continuous action correction, the direction of policy learning can still be affected by erroneous rewards.

[0005] (4) Strategy stabilization methods based on auxiliary rewards or structured rewards Some research focuses on improving training stability by designing latent reward functions, constructing structured reward modules, or utilizing model-aided signals. For example, latent function reward shaping can improve training efficiency in sparse reward environments; event-triggered reward structures can enhance task stage recognition; and auxiliary rewards can mitigate the effects of reward noise or latency. However, these rewards all rely on manual design or static training and lack dynamic adjustment capabilities. When the environment's distribution changes, these rewards are difficult to maintain effectiveness and are not reliable enough for long-term deployment of real robots.

[0006] (5) Reward evaluation methods based on value learning or preference ranking Other studies utilize value functions, preference ranking, or reinforcement learning to evaluate plans or behaviors. For example, preference-based learning systems use human annotations to determine which behavior is better, which is then used to train the reward model; while RLHF-related methods progressively calibrate policies or reward models through human preferences. However, these methods typically require a large number of environmental interactions or preference samples, making them difficult to deploy efficiently on real robots. Furthermore, these models lack deep binding with visual semantics, resulting in insufficient robustness to rewards under visual perturbations.

[0007] A systematic analysis of existing technologies reveals that, despite the various reward modeling, human-computer interaction learning, and visual semantic inference schemes proposed in the field of reinforcement learning for real robots, these technologies still have key shortcomings in practical deployment, limiting the stability, reliability, and adaptability of real robot policy learning.

[0008] First, existing reward models are generally static and cannot be adaptively calibrated online based on new data from real-world environments. Whether it's a visual language model based on image or video inference, or a reward learning method based on demonstration or preference videos, training is completed offline and is not updated after deployment. Once the robot explores an unseen state, or the environment changes in lighting, angle, or layout, the reward estimation will shift, leading to a deviation in the policy learning direction or even complete failure.

[0009] Second, existing human-in-the-loop (HIL) technology is mainly used for action-level correction, using human intervention to avoid erroneous behaviors, but it cannot correct the judgments of the reward model. When the reward itself is misjudged, even if humans continuously correct the actions, policy learning will still be misled by erroneous rewards in the long run, making it difficult to achieve stable and reliable convergence.

[0010] Third, existing technologies cannot effectively transform sparse human feedback into dense reward trajectories with temporal consistency. Human feedback in real-world environments is often extremely limited, typically manifesting as "stop / continue" or "success / failure" endpoint signals. Due to the lack of a dense supervision mechanism to extend these discrete signals into dense signals that can be used for learning the entire trajectory, reward signals struggle to maintain smoothness, continuity, and differentiability, making them insufficient to support efficient reinforcement learning processes in real robots.

[0011] Fourth, while large-scale video language models possess powerful semantic representation capabilities, current technologies have yet to provide a stable and efficient online update mechanism that allows the model to adapt to constantly changing task states in real-world environments without compromising pre-trained semantics due to parameter updates, thus preventing catastrophic forgetting. Current solutions either fail to update the model at all or involve large-scale fine-tuning, neither of which can achieve the "lightweight, robust, and rapidly adaptable" requirements of real-world applications.

[0012] Furthermore, existing solutions lack a closed-loop system that unifies the management of "value alignment" and "action alignment." Reward calibration and action correction are often handled separately, preventing the system from forming a consistent and stable learning process. When a reward is misjudged, there is insufficient action data to correct the policy; conversely, when an action is corrected, the erroneous reward continues to mislead the policy, resulting in a long-term lack of synergy between the two.

[0013] In the process of reinforcement learning for real robots, how to design a reward model that can automatically generate dense, time-consistent, and adaptively updated reward model based on human sparse feedback, so that it can maintain the semantic capabilities of the pre-trained visual language model, continuously calibrate the reward estimation, and work in conjunction with the action correction mechanism, thereby effectively improving the stability, sample efficiency, and adaptability of policy learning in real environments. Summary of the Invention

[0014] In view of this, the present invention aims to solve the core problems such as reward misjudgment, reward drift, lack of online adaptation, insufficient sparse feedback, and disconnect between reward and action calibration, and provides a real-machine reinforcement learning method for human-in-the-loop robots with adaptive rewards.

[0015] The technical solution for implementing the present invention is as follows: Firstly, the present invention provides a real-machine reinforcement learning method for human-in-the-loop robots with adaptive rewards, the specific process of which is as follows: Construct a reward model to supervise the execution of the robot's internal policy network, and utilize stage progress labels. The labeled pre-trained dataset is used to train the robot and then fed into it. The robot uses the current policy network to perform tasks in a real-world environment; When the robot's actions based on the current policy network are unsafe or deviate from the target trajectory, the operator inputs control to generate an intervention action to replace the actions of the current policy network. This intervention action is then written into the policy intervention dataset as an example of the correct behavior. Used for policy network updates; When reward model When the output judgment result does not match reality, the operator inputs a discrete feedback signal, and records the discrete feedback signal and its corresponding timestamp as a reward correction dataset. According to the reward correction dataset Backtracking annotations extend sparse feedback into dense trajectories for updating the reward model.

[0016] Optionally, the label form of the present invention is:

[0017] in: For the video number frame, This refers to the video length.

[0018] Optionally, the present invention is based on the reward correction dataset. The specific process of performing backtracking annotation to expand sparse feedback into dense reward is as follows: Positive reward: For successful trajectories without human intervention, linear progress interpolation is used.

[0019] Negative penalty: For trajectories with human intervention, when the human action intervention occurs at time... When it happens, the previous The stage labels within the range are subject to decay penalties: .

[0020] Optionally, the feedback signals input by the operator in this invention include: a continue signal and a stop signal, wherein the misjudgment signal indicates that the reward model misjudged success too early, and the stop signal indicates that the reward model did not detect success.

[0021] Optionally, when updating the reward model described in this invention, the loss function is designed as a weighted average of the stage regression loss and the success / failure calibration loss; wherein, The stage regression loss is: ,in, The expected value of the reward model output stage. Supervision values ​​generated for backtracking the labeled dataset; Success / failure calibration loss: ,in: and Indicates the set parameters;

[0022] When marked as truly successful When marked as unsuccessful =0, This represents the probability that the reward model predicts a success.

[0023] Optionally, the reward model described in this invention It adopts a structure of "frozen visual language model + lightweight value head".

[0024] Optionally, the pre-training dataset described in this invention is a video from the Internet or an open-source robot dataset.

[0025] Optionally, before the robot performs a task in a real environment using the current policy network, the present invention still includes the following process. : Data collection for robot action demonstration, utilizing the dataset Robot policy network Conduct training; : Obtain a reward-aligned sample dataset and utilize the dataset Train the reward model.

[0026] Optionally, the policy network update of this invention adopts a soft evaluation-execution architecture (SAC), which introduces a human behavior regularization term into the executor loss function. This regularization term is updated based on behavior clones (BC).

[0027] Optionally, the output of the reward model of this invention is rewritten as follows: ,in, This represents the output of the reward model; the calculated immediate reward. The signal will be stored in a unified playback buffer and used as a supervisory signal to update the evaluator network in the SAC architecture. The evaluator network learns to evaluate the current action by minimizing the Bellman error. To advance the progress of the task The long-term value of the network guides the strategy network. Evolving towards actions of higher value.

[0028] Beneficial effects: First, an adaptive reward construction method suitable for real-world scenarios is proposed: an online reward construction method that can combine sparse human feedback and generate trajectory-level dense rewards is proposed. The backtracking annotation mechanism effectively alleviates the problems of reward sparsity and misjudgment in real-world environments, and significantly improves the learning stability of robots in dynamic scenarios.

[0029] Second, a human-in-the-loop intelligent calibration system based on dual-loop interaction is proposed: a dual-loop human-machine collaborative calibration system that integrates action intervention and reward error correction is proposed, which enables the robot to receive dual supervision from humans while performing tasks, realizes the synchronous alignment of strategy behavior and task semantics, and enhances the safety and controllability of strategy optimization.

[0030] Third, a hybrid training mechanism that integrates multi-source supervision signals is proposed: a hybrid sampling training mechanism that integrates pre-training data, demonstration data, and human error correction data is proposed. Through multi-source supervision, the generalization ability and stability of the reward model are enhanced, so that it still has consistent reward judgment ability in long-term operation.

[0031] Fourth, a reward-policy collaborative adaptive learning architecture for real-world environments: A closed-loop adaptive learning architecture with asynchronous coupling of reward calibration and policy learning is constructed, enabling the robot to continuously optimize its reward interpretation ability and behavioral policies in the real world, and achieve long-term stable operation of reinforcement learning in real robots. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram of the overall system architecture. Figure 3 This is a data flow diagram illustrating human intervention in this invention; Figure 4 This is a schematic diagram illustrating the principle of backtracking reward labeling and penalty window construction in this invention. Detailed Implementation

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0036] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0037] The design concept of this invention is as follows: during the operation of a real robot, the reward model is dynamically calibrated through sparse feedback provided by humans to construct a dense and temporally consistent reward sequence. Based on a parameter-efficient model structure, the reward model is updated online, so that the reward estimation continuously aligns with the semantics of the real task, thereby improving the stability and learning efficiency of the overall reinforcement learning strategy in the real environment.

[0038] This application provides an adaptive reward-based human-in-the-loop reinforcement learning method for real robots. like Figure 1 As shown, the flowchart of this invention illustrates the complete operational process from the initiation of the robot task, environmental interaction, generation of human-in-the-loop feedback, to reward model updates and policy iterative optimization. The flowchart covers key steps such as pre-training data preparation, online trajectory acquisition, backtracking annotation of sparse feedback, efficient parameter updates of the reward model, and action alignment optimization of the policy. Each step has a clear temporal sequence and logical dependency, ensuring that reward evaluation, action correction, and policy training are collaboratively advanced within a unified closed-loop system.

[0039] like Figure 2As shown, the overall system architecture of this invention includes: a real environment and robot body, a sensor module, a visual language backbone network + value head module, a backtracking annotation and reward rewriting module, a unified playback buffer, a policy learning module, a policy network, a human-in-the-loop interface module, and a human operator. These modules are connected via data flow and control flow, forming a closed-loop human-machine collaborative adaptive learning system.

[0040] This application provides an adaptive reward-based human-in-the-loop real robot reinforcement learning method, the specific process of which is as follows: Step 1: Reward model construction, data collection, and model initialization 1.1 Reward Model Construction Reward Model Construction: Reward Model Employing a "frozen visual language model (backbone)" + Lightweight value head The reward model outputs a structure of ")". This indicates the current task completion progress category (TaskCompletion Progress, 0~100%).

[0041] Reward Model The update rules are as follows: The Backbone visual language backbone network remains frozen and will no longer be updated; only during the demonstration phase will the value-head network be updated. Train the model to enable it to have basic progress judgment capabilities.

[0042] Robot Policy Network The update rule is: use Behavior Cloning (BC) to update.

[0043] 1.2 Construction of the pre-training dataset and initial training of the reward model To enable the reward model to possess basic semantic capabilities, this invention uses massive amounts of video data from the internet or open-source robotics datasets (such as Open-X-Embodiment) as pre-training data for the reward model. .

[0044] Each video segment is assigned a heuristic stage progress label (0–10), in the following format:

[0045] in: For the video number frame, This represents the video length; a larger value indicates that the task is closer to completion.

[0046] Using the pre-trained data Implement reward model During training, the aforementioned labels are used to provide approximate but scalable time progress supervision, enabling the reward model to learn general temporal semantics on large-scale videos.

[0047] 1.3 Collection and Training of the Demonstration Dataset This phase involves several successful demonstrations generated by a robot operated remotely by humans. Action demonstration data acquisition, including state-action pair sequences extracted from successful demonstration trajectories. Using the dataset Robot policy network Conduct training; The reward-aligned demonstration dataset contains visual observation sequences (such as image frames) of successfully demonstrated trajectories and their corresponding task progress labels. This dataset is then used to... The reward model is trained, and the reward model obtained from training in section 1.2 is... The learning focuses on general temporal semantics, therefore this step further utilizes the dataset. Retrain the reward model to make the reward model Adjust the execution semantics (such as viewpoint and joint movement) to the specific robot platform.

[0048] Step Two: Human-in-the-loop data acquisition and dual-loop interaction This invention designs a dual-loop interaction mechanism: an action alignment loop and a value alignment loop, which run simultaneously to form a dynamic closed loop.

[0049] 2.1 Autonomous Execution Strategy The robot uses the current policy network. Perform tasks in a real-world environment and collect the following data: Observation sequence ,action and stage estimation of the reward model output These data form an online dataset. .

[0050] 2.2 Human movement intervention When the strategy's actions become unsafe or deviate from the target trajectory, humans take over control via manipulators (such as SpaceMouse), resulting in human intervention actions, denoted as... This action The action output by the alternative policy is executed and written into the policy intervention dataset as an example of correct behavior (Ground-truth Action). ,Right now Subsequently, the dataset directly guides the policy network updates through Behavior Cloning Regularization (BC Regularization).

[0051] 2.3 Human Reward Judgment When reward model When the model's judgment does not match reality (for example, the reward model outputs 100%, indicating that the robot has completed the task, but the robot has not actually completed it, meaning the model has misjudged a premature success), humans can press a button to provide discrete feedback signals: "Continue" indicates that the model misjudged a premature success (positive error signal); "Stop" indicates that the model did not detect success (negative error signal). These discrete feedback signals and their corresponding timestamps are recorded and stored as a reward correction dataset. The dataset will then proceed to step three for backtracking annotation, transforming sparse feedback into dense supervisory signals to train the reward model, thereby calibrating the reward model's value alignment.

[0052] Combined with appendix Figure 2 The system architecture shown uses a sensor module to collect data on the robot's current observations. And transmitted to the frozen visual language backbone network (parameters) Extract semantic features. Value Head Module (parameters) The output stage is estimated based on this feature. When human-computer interaction occurs, as shown in the attached diagram... Figure 3 As shown, the generated motion intervention data and reward feedback data Import into unified playback buffer The buffer connection policy learning module provides it with correction actions. and rewrite rewards A mixed data stream is used to update the policy network. Interact with the next round.

[0053] Step 3: Backtracking annotation extends sparse feedback to dense trajectory generation: This is the core technical point that distinguishes this invention from the prior art: expanding the sparse signals of humans into a dense supervised sequence of the entire trajectory.

[0054] Positive Rewards: Linear Phase Interpolation. For successful trajectories without human intervention, this invention employs linear progress interpolation.

[0055] Although this interpolation is not perfectly accurate, it has good temporal continuity and can automatically generate training signals on large-scale data.

[0056] Negative penalty: Failure window decay, for reward correction datasets obtained after adding human intervention. When human intervention occurs at any time When it happens, the previous The stage labels within the range are subject to decay penalties:

[0057] Its function is to emphasize the antecedent states that lead to failure, provide directional negative sample supervision, and clearly indicate to the reward model what situations should be considered "incomplete".

[0058] Figure 4 This is a schematic diagram illustrating the principle of backtracking reward labeling and penalty window construction in this invention. Geometric anchor points are determined by calculating the spatial correspondence between the robot's restored pose and its autonomous historical trajectory. And at this anchor point, the starting point of human action intervention. A linearly decaying penalty window is constructed between the two to compensate for the drift caused by the reaction delay, thereby transforming the discrete feedback signal provided by humans into a continuous and dense value curve on the time axis, thus achieving accurate calibration and alignment of the reward model for the robot's local dynamic deviations.

[0059] Step 4: Efficiently update the parameters of the reward model online. Constructing a hybrid training batch: This invention uses a hybrid sampling of pre-training data, reward-aligned demonstration datasets, and online human intervention data for updating the reward model, so that the update of the reward model will not suffer from catastrophic forgetting and can quickly adapt to the current scenario.

[0060] Among them, the reward model loss function design: the reward model loss function adopts the hybrid objective function proposed in this invention.

[0061] Phase regression loss:

[0062] in, The expected value of the reward model output stage. Supervision values ​​generated for the back-labeled dataset.

[0063] Success / failure calibration loss:

[0064] in: and Indicates the set parameters;

[0065] When marked as truly successful When marked as unsuccessful =0, This represents the probability that the reward model predicts a success.

[0066] Final reward model optimization goal:

[0067] in, and This indicates the set weight.

[0068] The update strategy is to update only the reward model value head parameters and freeze the background function to avoid semantic forgetting.

[0069] Step 5: Policy Learning: Optimizing Policy Action Alignment 1. Rewrite the evaluator's objective based on the updated reward model, and rewrite the reward as follows: ,in The discount factor is used to adjust the weight of future state values ​​in the current decision. This represents the predicted task progress value output by the reward model.

[0070] like Figure 2 The output of the "Value Head" module is shown in the image. Based on current visual observation The evaluation yields a discretization progress level from 0 to 100. The immediate reward calculated from this is... It will be stored in a unified playback buffer (such as Figure 2 Buffer in This signal is used as a supervisory signal to update the Critic network in the SAC architecture. The Critic network learns to evaluate the current action by minimizing the Bellman error. To advance the progress of the task The long-term value of the network guides the strategy network. Evolving towards actions of higher value.

[0071] 2. Action Alignment and Policy Update The strategy employs a soft actor-critic (SAC) architecture, which specifically introduces a human behavior regularization term into the actor loss function. This regularization term is updated based on behavior clones (BC).

[0072] This application has the following technical advantages: First, improve the stability of the reward model and avoid semantic degradation. In current technologies for reinforcement learning in real-world robots, reward models are typically either completely frozen and unable to adapt to scene changes, or they employ end-to-end fine-tuning to directly update the entire visual language model. However, freezing the model prevents reward judgments from adjusting to environmental changes, resulting in reward drift when lighting, pose, or background changes occur. While end-to-end fine-tuning is adaptive, its large update range and extremely high parameter count easily disrupt the model's original semantic structure, leading to catastrophic forgetting and causing the model to lose its ability to judge basic object relationships and action semantics. Therefore, both of these methods lack adaptability or stability, and neither can support the reward estimation requirements of continuous operation in real-world robots.

[0073] Improvements in this application: To address the aforementioned contradictions, this application constructs a parameter-efficient online reward adaptation method that "freezes the visual language backbone and updates only the value head," enabling the model to dynamically adjust reward judgments while maintaining pre-trained semantic capabilities. Specifically, by freezing the general visual encoding structure of the backbone network, it ensures that the model will not destroy existing semantics during online training, while the value head, as an independent lightweight module, can be quickly corrected based on real-time human feedback. This structurally isolates semantic features from reward judgments and allows the reward model to adapt to environmental changes with minimal update overhead, fundamentally avoiding the semantic degradation problem caused by end-to-end fine-tuning.

[0074] Technical effect: Through the above-mentioned structured separation and local update mode, this application achieves the adaptive ability of reward judgment while ensuring the semantic stability of the model, significantly reducing the risk of reward drift, and enabling the reward model to maintain consistent and reliable judgment ability during long-term operation, thereby solving the core problem that existing technologies cannot take into account both robustness and adaptability.

[0075] Second, improve online update efficiency to achieve real-time reward calibration in a real-world environment. In scenarios requiring real-time adjustment of the reward model, existing technologies often result in excessively high training time and computational costs due to full-model fine-tuning, especially for large visual language models with parameter scales typically reaching hundreds of millions, making online updates impossible to complete within the robot's interaction intervals. Furthermore, the high-load update process can cause training oscillations, leading to significant fluctuations in reward estimation over short periods and reducing the overall stability of the reinforcement learning process. Therefore, traditional methods are often impractical for deployment in embedded devices or continuous interaction scenarios.

[0076] The improvement of this application lies in limiting the online update scope to a lightweight module called the value head, enabling the reward model to achieve real-time parameter updates without affecting the large-scale network structure. Because the value head has a small number of parameters and low computational overhead, it can quickly absorb human feedback after each interaction and complete online fine-tuning. Simultaneously, this method has low requirements for the computing platform and can run directly on the robot's onboard computing device, thus achieving true real-time reward adaptation.

[0077] Technical Effects: By significantly reducing the computational burden of online updates, this application achieves real-time calibration of the reward model, enabling the robot to continuously update the reward judgment criteria during task execution, improving task adaptability and behavioral stability. It is particularly suitable for real-world robot systems requiring long-term continuous operation. Experimental data shows that compared to updating the entire backbone network or using LoRA fine-tuning, the lightweight head update strategy of this invention can increase the task success rate from approximately 60% to 100% within the same interaction time, and effectively avoids catastrophic forgetting caused by excessive parameter updates.

[0078] Third, a dynamic balance must be achieved between adaptability and stability to construct a reliable reward-based learning mechanism. If existing reward models rely solely on online human feedback for updates, they are prone to overfitting due to insufficient or biased feedback data, leading to unstable reward judgments. Conversely, if they depend entirely on offline demonstrations or pre-training data, they cannot cope with dynamic changes in real-world environments. This contradiction between "insufficient adaptability" and "insufficient stability" is prevalent in existing technologies, causing reward models to be unable to respond promptly to new task semantics and to easily deviate from the original semantic space due to inappropriate update strategies.

[0079] Improvements in this application: This application constructs a hybrid sampling mechanism combining pre-training data, demonstration data, and online error correction data, enabling the value head to simultaneously reference both a "stable general semantic distribution" and "real-time human feedback information" during training. This data fusion method ensures that the model does not drastically shift due to small-scale online data, while also allowing the model to absorb new feedback in a timely manner, keeping it sensitive to the current task. Furthermore, the dense supervision signal generated by backtracking annotations further improves feedback utilization efficiency, making the reward model update process smoother and more controllable.

[0080] Technical effect: By achieving coordinated control of stability and adaptability in training objectives and data distribution, this application enables the reward model to continuously and smoothly optimize its judgment ability in real environment, avoiding reward oscillations or distortions caused by online training, thereby significantly improving the overall convergence speed and performance of the reinforcement learning process.

[0081] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-machine reinforcement learning method for human-in-the-loop robots with adaptive rewards, characterized in that, The specific process is as follows: Construct a reward model to supervise the execution of the robot's internal policy network, and utilize stage progress labels. The labeled pre-trained dataset is used to train the robot and then fed into it. The robot uses the current policy network to perform tasks in a real-world environment; When the robot's actions based on the current policy network are unsafe or deviate from the target trajectory, the operator inputs control to generate an intervention action to replace the actions of the current policy network. This intervention action is then written into the policy intervention dataset as an example of the correct behavior. Used for policy network updates; When reward model When the output judgment result does not match reality, the operator inputs a discrete feedback signal, and records the discrete feedback signal and its corresponding timestamp as a reward correction dataset. According to the reward correction dataset Backtracking annotations extend sparse feedback into dense trajectories for updating the reward model.

2. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 1, characterized in that, The label format is as follows: in: For the video number frame, This refers to the video length.

3. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 2, characterized in that, According to the reward correction dataset The specific process of performing backtracking annotation to expand sparse feedback into dense reward is as follows: Positive reward: For successful trajectories without human intervention, linear progress interpolation is used. Negative penalty: For trajectories with human intervention, when the human action intervention occurs at time... When it happens, the previous The stage labels within the range are subject to decay penalties: 。 4. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 3, characterized in that, The feedback signals input by the operator include a continue signal and a stop signal. The misjudgment signal indicates that the reward model misjudged success too early, and the stop signal indicates that the reward model did not detect success.

5. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 3, characterized in that, When updating the reward model, the loss function is designed as a weighted average of the stage regression loss and the success / failure calibration loss; where, The stage regression loss is: ,in, The expected value of the reward model output stage. Supervision values ​​generated for backtracking the labeled dataset; Success / failure calibration loss: ,in: and Indicates the set parameters; When marked as truly successful When marked as unsuccessful =0, This represents the probability that the reward model predicts a success.

6. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 1, characterized in that, The reward model It adopts a structure of "frozen visual language model + lightweight value head".

7. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 1, characterized in that, The pre-training dataset consists of videos from the internet or open-source robot datasets.

8. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 1, characterized in that, Before the robot performs tasks in a real environment using the current policy network, the following processes are still included. : Data collection for robot action demonstration, utilizing the dataset Robot policy network Conduct training; : Obtain a reward-aligned sample dataset and utilize the dataset Train the reward model.

9. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 1, characterized in that, The policy network update adopts a soft evaluation-execution architecture (SAC), which introduces a human behavior regularization term into the executor loss function. This regularization term is updated based on behavior clones (BC).

10. The adaptive reward human-in-the-loop robot real-machine reinforcement learning method according to claim 9, characterized in that, Rewrite the reward model output as ,in, This represents the output of the reward model; the calculated immediate reward. The signal will be stored in a unified playback buffer and used as a supervision signal to update the evaluator network in the SAC architecture. The evaluator network learns to evaluate the current action by minimizing the Bellman error. To advance the progress of the task The long-term value of the network guides the strategy network. Evolving towards actions of higher value.

Images