Robot imitation learning and reinforcement learning framework based on human feedback

Abstract

The invention discloses a robot imitation learning and reinforcement learning framework based on human feedback, and the framework comprises the following steps: a first stage: imitation learning based on human feedback: S1, training an initial strategy through employing an initial expert teaching data set, and S2, carrying out deployment and manual intervention, and carrying out the deployment on a real or simulation robot for reasoning operation; according to the method, a security baseline is provided in an imitation learning stage, and strong and weak points are directly supplemented in a targeted manner through human intervention; the reinforcement learning stage starts from a high-performance starting point and is guided by human rewards, so that the training time is greatly shortened, destructive exploration is avoided, meanwhile, missing critical states and abnormal conditions in initial teaching data are effectively covered by human intervention data, the generalization ability of a strategy is qualitatively leap, and the training efficiency is improved. Therefore, the device has the advantages that the learning efficiency and safety are remarkably improved, and the generalization ability bottleneck is broken through at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot learning and artificial intelligence, and more particularly, to a robot imitation learning and reinforcement learning framework based on human feedback. BACKGROUND

[0002] Data-driven learning methods, mainly including imitation learning (IL) and reinforcement learning (RL), are the core means to train robot skills. However, both of them have significant limitations:

[0003] 1. Defects of pure imitation learning (IL): IL learns a policy by imitating the demonstration data provided by an expert. Its performance is heavily dependent on the quantity and quality of the demonstration data. When the demonstration data is limited or fails to cover all possible scenarios (such as extreme situations, abnormal states), the learned policy has poor generalization ability and is prone to failure when encountering unknown states. The learning process is essentially "open-loop", lacking evaluation and improvement mechanisms for good or bad behavior.

[0004] 2. Defects of pure reinforcement learning (RL): RL learns a policy by interacting with the environment and relying on reward signals. It can explore a strategy that surpasses the expert, but its learning process is often inefficient (requiring massive interaction data) and slow to converge. The completely random exploration behavior at the initial stage is destructive and unsafe for both the robot and the environment. Designing an effective reward function to guide learning is also a major challenge.

[0005] 3. Common defects: The learning processes of the above two methods lack necessary human intervention and guidance. The entire process is automatic but blind, unable to inject prior knowledge and advanced judgment of human experts at critical moments, resulting in a single and inefficient learning process.

[0006] Although existing work attempts to combine IL and RL (such as providing an initial strategy through pre-training), how to integrate humans as a real-time, intelligent feedback source into the learning cycle remains a challenge that has not been fully addressed. SUMMARY

[0007] In view of the problems in the prior art, the purpose of the present application is to provide a robot imitation learning and reinforcement learning framework based on human feedback.

[0008] To solve the above problems, the present application adopts the following technical solutions.

[0009] A robot imitation learning and reinforcement learning framework based on human feedback, comprising the following steps:

[0010] First stage: imitation learning based on human feedback;

[0011] S1, using initial expert demonstration dataset Train an initial policy ;

[0012] S2, deploy with human intervention, and Deploy to real or simulated robots for inference running;

[0013] S3, when the robot is about to fail due to insufficient generalization, human takes over control through remote control mode;

[0014] S4, record the state and human action as a new demonstration data Record in real time and store in incremental dataset ;

[0015] S5, for merging , incremental fine-tuning is performed on to strengthen weak links;

[0016] Second stage: reinforcement learning based on human feedback;

[0017] S6, use the final policy of the first stage As the initial network of the reinforcement learning strategy Instead of random initialization;

[0018] S7, interact with the environment Collect experience data, while receiving real-time evaluation scores and point data from humans;

[0019] S8, human feedback is converted into auxiliary rewards Combined with task rewards ;

[0020] S9, apply a large negative reward to human-labeled dangerous behavior to limit exploration range and ensure safety;

[0021] S10, use data that integrates human feedback to update Through the RL algorithm, a safe and efficient policy is improved.

[0022] As a further description of the above technical solutions: the calculation formula of the expert demonstration dataset in the S1 step is: = , where the state and the corresponding human expert action , represents the total number of samples in the expert demonstration dataset .

[0023] As a further description of the above technical solution: the S1 step input = , the action prediction error of the strategy is minimized by supervised learning: ;

[0024] As a further description of the above technical solution: each sample in the S1 step contains the state of the environment perceived by the robot and the action performed by the expert in that state .

[0025] As a further description of the above technical solution: the incremental data set in S3 records the state-action pair when human intervention is formulaed as: ;

[0026] As a further description of the above technical solution: the policy iteration optimization formula in S6 to S7 steps: , the merged training set contains the initial teaching data and human remediation data.

[0027] As a further description of the above technical solution: the S7 is followed by intervention data to supplement the deficiencies of the initial data and improve the performance of the strategy in unknown states.

[0028] As a further description of the above technical solution: the S8 human feedback reward formula is: ,

[0029] where is the human feedback weight coefficient;

[0030] where is generated by expert evaluation.

[0031] Compared with the prior art, the advantages of the present application are:

[0032] (1) This scheme significantly improves the learning efficiency and safety: the imitation learning phase provides a safe baseline, and human intervention directly reinforces weaknesses; the reinforcement learning phase starts from a high-performance starting point and is guided by human rewards, greatly shortening the training time and avoiding destructive exploration:

[0033] Breakthrough the generalization ability bottleneck: human intervention data effectively covers the critical states and abnormal situations missing in the initial teaching data, making the generalization ability of the strategy leap forward.

[0034] (3) This scheme alleviates the difficulty of designing reward functions: human feedback provides an intuitive and efficient reward signal for reinforcement learning, reducing the dependence on precise design of environment reward functions:

[0035] Constructing a closed-loop learning system for human-machine collaboration: Integrating the advanced cognitive judgments of human experts as the core feedback source into the entire learning process forms a continuously improving optimization loop, making the robot more and more intelligent the more it is used. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the operation process of the present invention. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention;

[0038] To address the shortcomings of current robot learning methods that rely solely on imitation, Example 1 is proposed:

[0039] Please see Figure 1 In this invention, a robot imitation learning and reinforcement learning framework based on human feedback includes the following steps:

[0040] Phase 1: Imitation learning based on human feedback;

[0041] S1. Using the initial expert teaching dataset Train an initial policy ;

[0042] S2, deployment and manual intervention, will Deployed onto real or simulated robots for inference operations;

[0043] S3. When the robot is about to fail due to insufficient generalization, humans take over control through remote control mode.

[0044] S4. Record the state during intervention. and human actions As a new teaching data point Record and store in incremental dataset in real time ;

[0045] S5, Regarding the merger ,right Make incremental adjustments to address weaknesses in a targeted manner;

[0046] Phase Two: Reinforcement learning based on human feedback;

[0047] S6, will adopt the final strategy of the first phase. As a reinforcement learning strategy The initial network, rather than random initialization;

[0048] S7, will It interacts with the environment, collects experience data, and receives real-time human evaluations, ratings, and dislikes.

[0049] S8, Human Feedback Converted into Assistive Rewards and task rewards Combine;

[0050] S9. Impose extremely negative rewards on dangerous behaviors marked by humans, limit the scope of exploration, and ensure safety;

[0051] S10. Update using RL algorithm with data incorporating human feedback. This will enable a safer and more efficient strategy.

[0052] The formula for calculating the expert teaching dataset in step S1 is: = , where the state and corresponding human expert actions , Data presented by expert representatives The total number of samples in the set;

[0053] S1 Step Input = Minimize the action prediction error of the policy through supervised learning: ;

[0054] Each sample in step S1 Including the environmental state perceived by the robot And the actions performed by experts in this state ;

[0055] Incremental dataset in S3 The formula for recording state-action pairs during human intervention is: ;

[0056] In this embodiment, by providing a safe baseline during the imitation learning phase, human intervention directly and specifically addresses weaknesses; the reinforcement learning phase starts from a high-performance starting point and is guided by human rewards, which greatly shortens training time, avoids destructive exploration, and ensures that human intervention data effectively covers the critical states and abnormal situations missing in the initial teaching data, resulting in a qualitative leap in the generalization ability of the strategy. Thus, the device achieves the advantages of significantly improving learning efficiency and safety while breaking through the bottleneck of generalization ability.

[0057] Furthermore, to address the shortcomings of current robot learning methods that rely solely on reinforcement learning, a second implementation example is proposed:

[0058] Please see Figure 1 Wherein: the strategy iteration optimization formula in steps S6 to S7 is: The merged training set includes the initial teaching data and human remedial data;

[0059] S7 subsequently intervened in the data. Supplementing the deficiencies in the initial data and improving the performance of the strategy in unknown situations;

[0060] The S8 human feedback reward formula is: ,

[0061] in For human feedback weighting coefficients;

[0062] in Generated by expert evaluation.

[0063] In this embodiment, human feedback provides an intuitive and efficient reward signal for reinforcement learning, reducing the reliance on precisely designed environmental reward functions. By integrating the advanced cognitive judgment of human experts as the core feedback source into the entire learning process, a continuously improving optimization loop is formed, making the robot more and more intelligent with use. This achieves the advantages of the device in solving the design challenges of environmental reward functions and constructing a human-machine collaborative closed-loop learning system.

[0064] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. A human feedback based robot imitation learning and reinforcement learning framework, characterized in that: Comprising the steps of: Phase 1: imitation learning based on human feedback; S1, utilize an initial expert demonstration dataset train an initial policy ; S2, deploy with human intervention, to deploy to real or simulated robots for inference runs; S3, when the robot is about to fail due to insufficient generalization, the human takes over control through remote control mode; S4, record state at intervention and human actions , as a new teaching data pair recorded in real time and stored in the incremental dataset ; S5, for merging , for incremental fine-tuning, targeted reinforcement of weak links Phase 2: reinforcement learning based on human feedback; S6, the final policy of the first phase is used as the initial network for the reinforcement learning policy instead of random initialization; S7, to interact with the environment, collect empirical data, while receiving human real-time evaluation scores and click-through data; S8, human feedback is translated into auxiliary rewards in combination with task rewards in combination; S9, impose a large negative reward on human-labeled dangerous behavior, limit the exploration range, and ensure safety; S10, using the data fused with human feedback, updating by RL algorithm , realizing safe and efficient policy promotion.

2. The robot imitation and reinforcement learning framework based on human feedback according to claim 1, wherein: The formula for calculating the expert teaching dataset in step S1 is as follows: = , where the state and corresponding human expert actions , Data presented by expert representatives The total number of samples in the set.

3. The robot imitation and reinforcement learning framework based on human feedback according to claim 2, wherein: The S1 step inputs = , minimizing the policy's action prediction error through supervised learning: .

4. The robot imitation and reinforcement learning framework based on human feedback according to claim 3, wherein: Each sample in the S1 step An environment state comprising robot perception And actions performed by experts in that state .

5. The robot imitation and reinforcement learning framework based on human feedback according to claim 1, wherein: The S3 incremental data set The state-action pair formula at the time of human intervention is recorded as: .

6. The robot imitation and reinforcement learning framework based on human feedback according to claim 1, wherein: The policy iteration optimization formula in the S6 to S7 steps: The merged training set contains the initial teaching data and human remediation data.

7. The robot imitation and reinforcement learning framework based on human feedback according to claim 5, wherein: The S7 is followed by intervention data Supplement the deficiency of the initial data, and improve the performance of the strategy in unknown states. 8.The human feedback based robot imitation learning and reinforcement learning framework of claim 1, wherein: The S8 human feedback reward formula is: , wherein is the human anti- weight coefficient; wherein Generated by expert evaluation.

Images