Human-in-loop reinforcement learning data enhancement method based on preference transitivity
By constructing a human preference dependency graph, equal preferences are transformed into partial order preferences, which solves the problem of low feedback efficiency in human-in-the-loop reinforcement learning, achieves faster convergence speed and lower development cost, and is suitable for human-in-the-loop reinforcement learning frameworks.
Patent Information
- Application Number
- CN202410971397.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
AI Technical Summary
Existing human-in-the-loop reinforcement learning methods are inefficient when dealing with human feedback, especially when it comes to feedback with equal preferences, and fail to effectively utilize preference information across rounds, resulting in slow algorithm convergence and high development and deployment costs.
By constructing a human preference dependency graph, the equal preference relationship in human feedback is transformed into a partial-order preference relationship. The reward function is trained using the partial-order preference relationship to guide the policy learning of the reinforcement learning agent. Combined with the soft actor critic algorithm, convergence is accelerated.
It improves the efficiency of human feedback, increases the convergence speed of the algorithm, reduces development and deployment costs, and has high versatility and flexibility.
Smart Images

Figure CN118916133A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-in-the-loop reinforcement learning, and in particular to a human-in-the-loop reinforcement learning data enhancement method based on preference transitivity. Background Art
[0002] The Human-in-the-Loop Reinforcement Learning (HITLRL) task is a challenging and practical task in the field of artificial intelligence. Compared with the traditional task setting of reinforcement learning, the human-in-the-loop reinforcement learning algorithm is able to teach the desired behavior of the reinforcement learning agent without designing a reward function. Instead, the agent uses feedback, usually the (human) teacher's preference between two behaviors, to learn the ideal behavior instructed by the teacher. Therefore, the agent does not need to use carefully designed environmental rewards, but is able to learn a more flexible reward function that is better aligned with the teacher's feedback.
[0003] However, human-in-the-loop reinforcement learning usually requires a lot of teacher feedback, which may be time-consuming or sometimes infeasible. To improve feedback efficiency, previous studies have adopted different sampling strategies, aiming to select more informative queries to improve the quality of the learned reward function with less teacher feedback. Other works focus on policy initialization. For example, some studies initialize the agent's policy by imitating learning from expert demonstrations, and some studies use unsupervised agent pre-training to learn more diverse behaviors, thereby reducing the total amount of manual feedback required. However, previous works often assume that humans only provide partial order relationship preferences, ignoring the fact that human feedback in actual scenarios may be ambiguous, that is, equal preference. If this type of feedback cannot be handled well, the efficiency of human preference feedback will inevitably decrease. At the same time, previous works sample action sequences independently in each round, and do not connect the preference information between rounds. Summary of the invention
[0004] The purpose of the present invention is to propose a human-in-the-loop reinforcement learning data enhancement method based on preference transitivity in response to the deficiencies of the prior art. This method applies the transitivity of human preferences. By constructing a human preference dependency graph, the equal preference relationship for action sequences in human feedback can be converted into a partial order preference relationship for action sequences. By comparing the action sequences corresponding to the nodes with zero in-degree in each round of the human preference dependency graph, the human preference dependency graph can be merged, and the intrinsic partial order relationship between action sequences between different feedback rounds can be deeply mined. Compared with the prior art, this method can effectively convert the equal preference data in human feedback into optimizable partial order preference data, solving the problem that the previous method cannot use the equal preference data; at the same time, the human preference dependency graph is used to effectively improve the efficiency of human preference feedback and accelerate the convergence speed of the algorithm; in addition, this method has high versatility and modularity, and users can easily integrate the algorithm components into the existing human-in-the-loop reinforcement learning framework, greatly reducing the development and deployment costs and improving the flexibility of the system.
[0005] The object of the present invention is achieved in that:
[0006] A human-in-the-loop reinforcement learning data augmentation method based on preference transitivity is characterized in that the method samples action sequences from the action sequence library of the reinforcement learning agent, constructs a human preference dependency graph based on human preference annotations for action sequence pairs, and converts the equal preference relations of action sequences in the human preference dependency graph into partial order preference relations of action sequence pairs, and then uses these partial order preference relations to train the reward function to guide the learning of the reinforcement learning agent strategy.
[0007] The specific steps include:
[0008] 1) Sample action sequence σ from the reinforcement learning agent action sequence library; specifically including:
[0009] 1.1: Before the first sampling, in order to achieve the specified goal, the agent will first explore the environment autonomously. Each exploration can be regarded as an action sequence σ in which the agent tries to complete the task, which consists of several state-action pairs. Composition, the action sequence of each agent will be stored in the reinforcement learning action sequence library;
[0010] 1.2: Before each human annotation, a fixed number of action sequences σ are randomly sampled from the action sequence library.
[0011] 2) Construct a cross-round human preference dependency graph based on human preference annotations for action sequence pairs; specifically, it includes:
[0012] 2.1: This step expands the traditional preference types of people in loop reinforcement learning from only partial order preference types to partial order preference relations and equal preference relations. 1 , σ 0 )’s partial order preference types include: action sequence σ 1 Better than action sequence σ 0 , that is, 1 >σ 0 , action sequence σ 0 Better than action sequence σ 1 , that is, 0 >σ 1 , and equal preference type: action sequence σ 0 With the action sequence σ 1 Equal preference, that is
[0013] 2.2: First randomly select an action sequence σ 1 Construct the initial node of the dependency graph, and then select another action sequence σ 0 , and with σ 1 Constitute an action sequence pair (σ 1 , σ 0 ) is given to humans to label preference information; if σ 1 >σ 0 Then add a line σ to the graph 1 Pointing to σ 0 If σ 0 >σ 1 Then add a σ 0 Pointing to σ 1 If Then merge the two nodes into one node without adding edges;
[0014] 2.3: The action sequence σ sampled later i The action sequences corresponding to the nodes without in-degree in the current human preference graph are combined into action sequence pairs and handed over to humans for annotation. Then, according to the rules described in 2.2, σ i Add to the human preference graph;
[0015] 2.4: If it is not the first round of labeling, it is necessary to combine the action sequence corresponding to the node with in-degree 0 in the current round of human preference dependency graph and the action sequence corresponding to the node with in-degree 0 in the human preference dependency graph constructed in the previous round into an action sequence pair, and submit it to humans for preference feedback. After obtaining the preference relationship, merge the current and previous rounds of human preference dependency graphs according to the rules of 2.2.
[0016] 3) Convert the equal preference relationship of action sequences in the human preference dependency graph into the partial order preference relationship of action sequence pairs to obtain enhanced human preference data; specifically including:
[0017] 3.1: Extract the partial order preference relationship of all action sequence pairs according to the human preference dependency graph, perform a depth-first search on the human preference dependency graph starting from the node with in-degree 0, and obtain all possible depth-first search paths;
[0018] 3.2: For each depth-first search path, record all partial order preference relationships, that is, the action sequence corresponding to the first node and the action sequence corresponding to the later node have a partial order preference relationship;
[0019] 3.3: Using the rules described in 2.2, if Then the action sequence σ 1 Will inherit the action sequence σ 0 All partial order preference relations corresponding to the same action sequence σ 0 Will inherit the action sequence σ 1 All the corresponding partial order preference relations are used to obtain enhanced human preference data.
[0020] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:
[0021] 1) The present invention proposes to use the transitivity of human preferences to transform equal preferences in human feedback into partial order preferences, solving the problem that previous methods cannot utilize equal preference data.
[0022] 2) The present invention proposes a human preference dependency graph, which can maintain the connection between sampling results across rounds, so as to better mine the intrinsic relationship in the data, effectively improve the efficiency of human preference feedback, and accelerate the convergence speed of the algorithm.
[0023] 3) The sampling method proposed in the present invention has low coupling with the existing optimization method of human-in-the-loop reinforcement learning algorithm, and has high versatility and modularity. Users can easily integrate algorithm components into the existing human-in-the-loop reinforcement learning framework, which greatly reduces development and deployment costs and improves the flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flow chart of the present invention;
[0025] Figure 2 Schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The present invention is further described below in conjunction with specific embodiments and drawings.
[0027] In order to utilize the equal preference relationship in human feedback and deeply explore the internal connection of sampling results across rounds, the present invention proposes sampling action sequences from the action sequence library of reinforcement learning agents, constructing a human preference dependency graph based on human preference annotations for action sequence pairs, and converting the equal preference relationship of action sequences in the human preference dependency graph into a partial order preference relationship of action sequence pairs, and then using these partial order preference relationships to train reward functions to guide the learning of reinforcement learning agent strategies.
[0028] Example
[0029] See also Figure 1 , a human-in-the-loop reinforcement learning data enhancement method based on preference transitivity in this embodiment specifically includes the following steps:
[0030] 1) Sample action sequence σ from the reinforcement learning agent action sequence library;
[0031] 1.1: Before the first sampling, in order to achieve the specified goal, the agent will first explore the environment autonomously. Each exploration can be regarded as an action sequence σ in which the agent tries to complete the task, which consists of several state-action pairs. Composition, the action sequence of each agent will be stored in the reinforcement learning action sequence library;
[0032] 1.2: Before each human annotation, the algorithm randomly samples a fixed number of action sequences σ from the action sequence library.
[0033] 2) Construct a human preference dependency graph based on human preference annotations for action sequence pairs;
[0034] 2.1: This step expands the traditional preference types of people in loop reinforcement learning from only partial order preference types to partial order preference relations and equal preference relations. 2 , σ 0 ) In terms of partial order preference types, they include: action sequence σ 1 Better than action sequence σ 0 , that is, 1 >σ 0 , action sequence σ 0 Better than action sequence σ 1 , that is, 0 >σ 1 , and equal preference type: action sequence σ 0 With the action sequence σ 1 Equal preference, that is
[0035] 2.2: First randomly select an action sequence σ 1 Construct the initial node of the dependency graph, and then select another action sequence σ0 , and with σ 1 Constitute an action sequence pair (σ 1 , σ 0 ) is given to humans to label preference information. 1 >σ 0 Then add a line σ to the graph 1 Pointing to σ 0 If σ 0 >σ 1 Then add a σ 0 Pointing to σ 1 If Then merge the two nodes into one node without adding an edge.
[0036] 2.3: The action sequence σ sampled later i The action sequences corresponding to the nodes without in-degree in the current human preference graph are combined into action sequence pairs and handed over to humans for annotation. Then, according to the rules described in 2.2, σ i Add to the human preference graph;
[0037] 2.4: If it is not the first round of labeling, it is necessary to combine the action sequence corresponding to the node with in-degree 0 in the current round of human preference dependency graph and the action sequence corresponding to the node with in-degree 0 in the human preference dependency graph constructed in the previous round into an action sequence pair, and submit it to humans for preference feedback. After obtaining the preference relationship, merge the current and previous rounds of human preference dependency graphs according to the rules of 2.2.
[0038] 3) Convert the equal preference relationship of action sequences in the human preference dependency graph into the partial order preference relationship of action sequence pairs to obtain enhanced human preference data;
[0039] 3.1: Extract the partial order preference relationship of all action sequence pairs according to the human preference dependency graph, perform a depth-first search on the human preference dependency graph starting from the node with in-degree 0, and obtain all possible depth-first search paths;
[0040] 3.2: For each depth-first search path, record all partial order preference relationships, that is, the action sequence corresponding to the first node and the action sequence corresponding to the later node have a partial order preference relationship;
[0041] 3.3: Using the rules described in 2.2, if Then the action sequence σ 1 Will inherit the action sequence σ 0 All partial order preference relations corresponding to the same action sequence σ 0 Will inherit the action sequence σ 1 All the corresponding partial order preference relations.
[0042] 4) Use partial order preferences of action sequences to train reward functions;
[0043] 4.1: Method The cross entropy function is used to optimize the reward function, and the probability of modeling the partial order relationship of human action sequence pairs is as follows:
[0044]
[0045] in represents the predicted value of the reward function to be trained, P[σ 1 >σ 0 ]Description 1 Ratio 0 Better probability, s t represents the state of the environment observed by the agent at time t, a t Represents the action taken by the agent at time t.
[0046] 4.2: Specifically use the following loss function to optimize the reward function:
[0047]
[0048] in represents the data set, y(0) and y(1) represent label values 0 and 1 respectively.
[0049] 5) Use the Soft Actor-Critic algorithm to guide the learning of reinforcement learning agents.
[0050] See also Figure 2 This embodiment will solve the Finger turn hard task of intelligent agent control in the DeepMind Control Suite simulation environment. First, the action sequences σ of the intelligent agent in the simulation environment are stored in the reinforcement learning intelligent agent action sequence library. Then, a human preference dependency graph is constructed according to human preference annotations on action sequence pairs. The equal preference relationship of the action sequences in the human preference dependency graph is converted into a partial order preference relationship of the action sequence pairs. The partial order preference pairs of the action sequences are used to train the reward function, and the soft actor-critic algorithm is used to guide the reinforcement learning agent learning.
[0051] The above description is only a preferred embodiment of the present invention, and certain modifications may be made thereto within the scope defined by the claims of the present invention, but all modifications will fall within the protection scope of the present invention.
Claims
1. A human-in-the-loop reinforcement learning data augmentation method based on preference transitivity, characterized in that: The method comprises the following specific steps: 1) Sample action sequence σ from the reinforcement learning agent action sequence library; 2) Construct a cross-round human preference dependency graph based on human preference annotations for action sequence pairs; 3) Convert the equal preference relationship of action sequences in the human preference dependency graph into the partial order preference relationship of action sequence pairs to obtain enhanced human preference data; where: The step 2) specifically includes: 2.1: Expand the preference types of people in loop reinforcement learning to partial order preference relations and equal preference relations. For action sequence pairs (σ 1 , σ 0 )’s partial order preference types include: action sequence σ 1 Better than action sequence σ 0 , that is, 1 >σ 0 , action sequence σ 0 Better than action sequence σ 1 , that is, 0 >σ 1 , and equal preference type: action sequence σ 0 With the action sequence σ 1 Equal preference, that is 2.2: First randomly select an action sequence σ 1 Construct the initial node of the dependency graph, and then select another action sequence σ 0 , and with σ 1 Constitute an action sequence pair (σ 1 , σ 0 ) is given to humans to label preference information; if σ 1 >σ 0 Then add a line σ to the graph 1 Pointing to σ 0 If σ 0 >σ 1 Then add a σ 0 Pointing to σ 1 If Then merge the two nodes into one node without adding edges; 2.3: The action sequence σ sampled later i The action sequences corresponding to the nodes without in-degree in the current human preference graph are combined into action sequence pairs and handed over to humans for annotation. Then, according to the rules described in 2.2, σ i Add to the human preference graph; 2.4: If it is not the first round of labeling, the action sequence corresponding to the node with in-degree 0 in the current round of human preference dependency graph and the action sequence corresponding to the node with in-degree 0 in the human preference dependency graph constructed in the previous round should be combined into an action sequence pair, and submitted to humans for preference feedback. After obtaining the preference relationship, the human preference dependency graphs of the current and previous rounds are merged according to the rules of 2.
2.
2. The human-in-the-loop reinforcement learning data enhancement method according to claim 1, characterized in that The step 1) specifically includes: 1.1: Before the first sampling, the agent first conducts autonomous exploration in the environment. Each exploration is regarded as an action sequence σ in which the agent attempts to complete the task, which consists of several state-action pairs. Composition, the action sequence of each agent is stored in the reinforcement learning action sequence library; 1.2: Before each human annotation, a fixed number of action sequences σ are randomly sampled from the action sequence library.
3. The human-in-the-loop reinforcement learning data enhancement method according to claim 1, characterized in that the step 3) specifically comprises: 3.1: Extract the partial order preference relationship of all action sequence pairs according to the human preference dependency graph, perform a depth-first search on the human preference dependency graph starting from the node with in-degree 0, and obtain all possible depth-first search paths; 3.2: For each depth-first search path, record all partial order preference relationships, that is, the action sequence corresponding to the first node and the action sequence corresponding to the later node have a partial order preference relationship; 3.3: Using the rules described in 2.2, if Then the action sequence σ 1 Will inherit the action sequence σ 0 All partial order preference relations corresponding to the same action sequence σ 0 Will inherit the action sequence σ 1 All the corresponding partial order preference relations are used to obtain enhanced human preference data.
