Man-machine reinforcement learning method based on multi-dimensional human feedback fusion

Through generative adversarial imitation learning and dynamic adaptive weighting methods, a unified reward function is constructed, which solves the problem of complex design of reward function and insufficient single feedback in traditional reinforcement learning, and improves the RL system in dynamic environment performance and learning efficiency.

CN119005287APending Publication Date: 2024-11-22NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 8 Cited by

Patent Information

Application Number
CN202411023440.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Traditional reinforcement learning methods rely on carefully designed reward functions, are complex and time-consuming, and a single type of user feedback cannot provide comprehensive information, resulting in poor performance of RL systems in dynamic environments and potential conflicts between different feedback sources. Difficult to deal with.

Method used

Generative adversarial imitation learning imitation expert demonstration is adopted, combined with user trajectory preferences, and combined with dynamic adaptive weighting methods to integrate expert demonstration and preference feedback to build a unified reward function to resolve the conflict between demonstration and preference.

Benefits of technology

Enhanced the robustness and coherence of the reward function, reduce the dependence on human preferences, reduce the need for human labeling data, and achieve optimal performance under suboptimal demonstration conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005287A_ABST
    Figure CN119005287A_ABST
Patent Text Reader

Abstract

A man-machine reinforcement learning method based on multi-dimensional human feedback fusion comprises the steps of performing generative adversarial imitation learning based on a strategy generator and a discriminator through expert demonstration, so that a strategy learned by the strategy generator cannot be distinguished by the discriminator, and obtaining a demonstration reward function; obtaining a preference award function conforming to the user preference by using the track preference of the user; and fusing the demonstration reward function and the preference reward function to obtain a unified reward function, and solving the conflict between the demonstration reward function and the preference reward function through a dynamic reward weight adjustment strategy. According to the method, limitation caused by single feedback and potential conflicts existing among a plurality of feedback sources can be overcome, and the robustness and continuity of the reward function are enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a human-machine reinforcement learning method based on multi-dimensional human feedback fusion. Background Art

[0002] In recent years, reinforcement learning (RL) has made remarkable progress in solving complex decision-making tasks. Traditional reinforcement learning methods usually rely on carefully designed reward functions to guide the learning trajectory of intelligent agents. However, designing precise and effective reward functions for real-world applications remains a difficult and time-consuming task. This is because creating such functions requires deep domain expertise and must accurately encapsulate various aspects of the task at hand to prevent inadvertent suboptimal or harmful agent behaviors. In addition, the design of reward functions may introduce uncertainty and bias, causing RL systems to perform poorly in the face of dynamic and uncertain environments. For example, in the field of autonomous driving, it is almost impossible to design a reward function that can cover all possible situations, and any omissions or errors may lead to safety hazards. In medical diagnosis, inaccurate reward functions may lead to misdiagnosis or inappropriate treatment plans, which directly affects patient health. To address these challenges, scholars have begun to study techniques for deriving reward functions directly from user feedback, with the goal of seamlessly incorporating user feedback into the optimization of intelligent agent policies. This emerging paradigm is expected to reduce the burden associated with reward function engineering and enhance the adaptability and robustness of RL systems in real-world scenarios.

[0003] Currently, methods for learning from user feedback mainly rely on expert demonstrations, evaluation feedback, or trajectory preferences. Although these methods have proven effectiveness in many tasks, most of them utilize a single type of feedback, limiting the diversity and richness of the information available to intelligent agents. In addition, a single feedback mode may have problems with incomplete or too sparse information, making it difficult to provide comprehensive guidance to the agent. Different types of feedback can provide unique information perspectives and have different effects on the learning and performance of the agent. For example, while expert demonstrations can convey rich task-related information, the process of obtaining this information is usually challenging and costly. On the other hand, user preference feedback, although less informative, is easier and cheaper to generate. Therefore, no feedback type can be considered universally optimal, as each feedback type has its unique information value and application scenarios.

[0004] A few scholars have studied the work of learning from multiple feedback sources. For example, deep Q learning and preference learning are directly combined to train the reward function, but this method is mainly limited to Atari games and has limited generalization ability. Li uses the IRL-TAMER framework to learn from demonstration and evaluation feedback. Although it is effective, the algorithm complexity is very high. The method of Hsiung et al. learns from actions and scalar feedback, but has strict real-time requirements. Aleksandar innovatively integrates generative adversarial imitation learning and preferences, and proposes to optimize the agent's strategy by updating the log density ratio estimator (LDRE). These methods all ignore the potential conflicts between different types of feedback, such as inconsistencies between demonstration data and preference data, which may cause the model to fail to accurately learn the correct reward function or strategy. Summary of the invention

[0005] In order to overcome the limitations of single feedback, such as the inability to provide comprehensive information to accurately guide the agent's learning and the potential conflicts between multiple feedback sources, this application proposes a human-machine reinforcement learning method based on multi-dimensional human feedback fusion, which adopts generative adversarial imitation learning to simulate the complex behaviors in expert demonstrations, and uses preference-based reinforcement learning to fine-tune the reward function. In addition, expert demonstrations and trajectory preference feedback are integrated by using a dynamic adaptive weighted method to enhance the robustness and coherence of the unified reward function.

[0006] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0007] A human-machine reinforcement learning method based on multi-dimensional human feedback fusion, comprising the following steps:

[0008] Through expert demonstration, generative adversarial imitation learning is performed based on the strategy generator and the discriminator, so that the strategy learned by the strategy generator cannot be distinguished by the discriminator, and the demonstration reward function is obtained;

[0009] Using the user's trajectory preference, we obtain a preference reward function that meets the user's preference;

[0010] The demonstration reward function and the preference reward function are integrated to obtain a unified reward function, and the conflict between the demonstration reward function and the preference reward function is resolved through a dynamic reward weight adjustment strategy.

[0011] The policy generator is used to imitate the expert demonstration and adjust the agent strategy according to the feedback provided by the discriminator to try to deceive the discriminator and make it believe that the generated state-action pairs come from the expert demonstration. The discriminator is used to distinguish between the state-action pairs generated by the expert demonstration and the state-action pairs generated by the policy generator, and provides a dynamic reward signal to the policy generator to guide the learning process.

[0012] Through training with generative adversarial imitation learning, the policy generator continuously optimizes the agent strategy to get infinitely close to the expert strategy, so that the behavior of the policy generator cannot be effectively distinguished by the discriminator, and the policy generator successfully imitates the behavior of the expert.

[0013] The specific process of obtaining the demonstration reward function is as follows:

[0014] S1-1: Assume that the dataset of expert demonstration is in, For Expert E i A finite sequence of state-action pairs over time horizon T, a dataset of expert demonstrations is used to train generative adversarial imitation learning, so that the policy generator can imitate the expert E i Strategy

[0015] S1-2: Assume that the optimization goal of generative adversarial imitation learning is a minimax problem, the goal is to minimize the occupancy metric difference between the agent strategy and the expert strategy. The objective function of generative adversarial imitation learning is as follows:

[0016]

[0017] Among them, π E is the expert strategy, π θ is the agent strategy, D(s, a) is the output of the discriminator for the state-action pair (s, a), when the discriminator believes that the state-action pair is generated by the expert, the output is close to 1; when the discriminator believes that the state-action pair is generated by the agent, the output is close to 0;

[0018] S1-3: The discriminator is trained to distinguish samples generated by the expert strategy and the proxy strategy by minimizing the binary cross entropy loss. The loss function is as follows:

[0019]

[0020] S1-4: After training, the output of the discriminator is converted into the reward of the generator to guide the agent strategy optimization:

[0021] R D (s, a) = -log(1-D(s, a))

[0022] The generator uses the reward obtained from the conversion to adjust the agent policy parameters through the reinforcement learning algorithm to maximize the cumulative reward until the state-action pairs generated by the generator are indistinguishable from the expert demonstrations, and the demonstration reward function R is obtained. D (s, a).

[0023] The preference reward function that meets the user's preference is obtained by utilizing the user's preference for the trajectory, specifically:

[0024] S2-1: The agent follows the strategy π to generate a trajectory dataset such that Show the agent different trajectory segments σ, formalized as sequences of state-action pairs of length H A set of segment pairs (σ0, σ1) is sampled from the trajectory, and the preference feedback given by the user indicating his preference is obtained. The preference query and the corresponding preference feedback are taken as a triple (σ0, σ1, y comp ) is stored in the preference feedback dataset P;

[0025] S2-2: Preference-based reinforcement learning PbRL Consider a Markov decision process MDP\R without reward, and gradually learn the reward function by providing preferences between state-action pairs by users Update proxy strategy π θ And the reward function Will The prediction reward R predict (s, a|ψ) is used as the preference reward function.

[0026] The update agent strategy π θ And the reward function Specifically:

[0027] S2-21: Proxy Strategy π θ Continuously interact with the environment to collect trajectories and use the updated reward function Perform reward estimation, proxy strategy π θ Use the RL algorithm to update θ to maximize the sum reward function

[0028] S2-22: Sample the segment pair (σ0, σ1) from the trajectory, obtain the user's preference feedback indicating his preference for the segment pair, and the reward function Optimize based on preference feedback;

[0029] Repeat S2-21 and S2-22, following the Bradley-Terry model, using the preference predictor P ψ To model the reward function

[0030]

[0031] When given a stored triple (σ0, σ1, y comp ) and train the reward function by minimizing the cross entropy loss between the preference predictor and the actual trajectory preference.

[0032]

[0033] Among them, y comp (i), i∈{1,2} is an indicator when the trajectory segment σ1 is preferred, y comp (i)=1; when the reward function Optimized according to user preferences. The prediction reward R predict (s, a|ψ) is considered as the preference reward function returned from the environment.

[0034] The specific process of obtaining a unified reward function is:

[0035] The strategy consistency index is used to measure whether the agent strategy has learned the expert strategy. The specific definition of strategy consistency is as follows:

[0036]

[0037]

[0038] Where AC(s) refers to the behavioral consistency of state s in all rounds, which is defined by the standard deviation of actions in that state, n is the current round number, M is the number of all states included in the expert demonstration, PC(n) is the policy consistency obtained in the nth round, and a threshold of policy consistency is given. When the policy consistency of a certain round is greater than or equal to the threshold, the agent's training enters the second stage;

[0039] After entering the second stage, a time growth factor based on the stability of the agent strategy is introduced to promote the update of the preference reward weight. A benchmark growth amount is set to ensure that the preference weight increases gradually after entering the second stage. At the same time, the strategy stability is calculated by detecting the variance of the cumulative rewards of multiple rounds. Its definition is as follows:

[0040]

[0041] Among them, N total is the total number of rounds, N P1 is the number of rounds experienced in the first stage, m is the number of rounds experienced since the start of the second stage, and R i is the cumulative preference reward in the i-th round starting from the second stage;

[0042] The unified reward function has the following form:

[0043] R total (s, a) = α·R D (s,a|ω)+β·R predict (s, a|ψ)

[0044] Among them, α and β are weight parameters, representing the proportion of demonstration reward and preference reward in the unified reward model respectively.

[0045] The specific steps for dynamically updating the weights of demonstration rewards and preference rewards are as follows:

[0046] S4-1, first initialize the weights, set the demonstration reward weight α and the preference reward weight β to be equal, to ensure that the two feedbacks have equal influence in the initial stage: α = β = 0.5;

[0047] S4-2, weight adaptive adjustment:

[0048] As the expert demonstration is imitated, the demonstration reward weight is gradually increased until a preset threshold is reached;

[0049]

[0050] Where λ is the learning rate, PC(n) is the policy consistency obtained in the nth round;

[0051] According to the user's trajectory preference, the weight of the preference reward is continuously increased until the strategy converges:

[0052]

[0053] Where λ is the learning rate, ε is the baseline growth amount to ensure that the preference weights grow steadily even when the policy stabilizes, and m is the number of episodes since the start of the second phase.

[0054] By evaluating the performance of the demonstration reward and the preference reward on the same state-action pair, it is determined whether the demonstration reward and the preference reward conflict. When one reward mechanism gives a high reward and the other gives a low reward, the state-action pair is considered to be in conflict.

[0055] If a conflict is found in the first stage, the preference reward weight of the corresponding state-action pair is adjusted to zero; if a conflict is found in the second stage, the demonstration reward weight is reduced to zero.

[0056] The specific criteria for judging whether the demonstration reward and the preference reward conflict are as follows: a reward value exceeding 0.5 is considered a high reward, while a reward value below 0.5 is considered a low reward, that is:

[0057]

[0058] Among them, R D Representative Demo Reward, R predict Represents a preference for reward.

[0059] Under the suboptimal demonstration conditions in the first stage, the utilization of user preference feedback is gradually increased in the second stage.

[0060] Compared with the prior art, the beneficial effects of this application are:

[0061] This application integrates expert demonstration and trajectory preference through a dynamic adaptive weighted method to enhance the robustness and consistency of the reward function; this application adopts a staged optimization process, first imitating expert behavior through adversarial generative imitation learning, and then integrating preference rewards to improve the strategy for user-specified goals, and resolves the conflict between demonstration and preference through dynamic reward weight adjustment.

[0062] Our application outperforms single-feedback methods in tests, achieving optimal performance even under suboptimal demonstration conditions by gradually increasing utilization of trajectory preference, while reducing reliance on human preference and the necessity for human-labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of the method of this application.

[0064] Figure 2 This is an overview diagram of this application.

[0065] Figure 3 This is a case environment diagram for this application in Atari games.

[0066] Figure 4 A Django-based web application interface for collecting real human feedback on trajectory segments. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clear, the present application is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0068] Using formal adversarial imitation learning (GAIL) as the basic algorithm for demonstration learning, by leveraging human demonstration data, the agent strategy is made close to the expert strategy, thereby improving and accelerating the reinforcement learning process and preventing the agent from falling into a local optimal solution. However, GAIL requires a large amount of high-quality demonstration data, and demonstration data cannot be obtained in some states. In addition, reinforcement learning from human preferences (RLHP) is used as the implementation algorithm for preference learning. By leveraging human preference data for trajectory segments, a reward function close to the real reward function is generated. Compared with traditional hard-coded reward functions, RLHP does not require a predefined reward function and can personalize the agent to match human preferences. However, RLHP has low sample efficiency, requires a lot of human feedback, and may have unreasonable or difficult queries when actively querying preferences.

[0069] In general, the sample efficiency problem of preference learning can be improved by leveraging demonstration data, while the demonstration limitation of demonstration learning can be further supplemented and refined by leveraging human preferences. Therefore, this application aims to integrate expert demonstrations and trajectory preferences by constructing a comprehensive and unified reward function, such as Figure 1 As shown, the reward function consists of two components:

[0070] 1. Demonstration Demo Reward R D (s, a): obtained from expert demonstrations through generative adversarial imitation learning (GAIL), which aims to make the agent's behavior mimic the expert's decision as much as possible, thereby providing the agent with a preliminary behavior strategy;

[0071] 2. Preference reward R predict (s, a): Based on the user’s trajectory preferences, preference learning algorithms (e.g., Bradley-Terry model) are used for adjustment, focusing on refining the reward function to ensure that the agent’s behavior can better match the specific goals or preferences defined by the user;

[0072] 3. The form of the unified reward function is as follows:

[0073] R total (s, a) = α·R D (s,a|ω)+β·R predict (s, a|ψ)

[0074] Among them, α and β are weight parameters, representing the proportion of demonstration reward and preference reward in the unified reward model respectively.

[0075] This application works as follows: first, expert behavior is imitated through generative adversarial imitation learning, then user preference rewards are integrated to refine the strategy for user-specified goals, and finally the conflict between expert demonstration and trajectory preference is resolved through dynamic reward weight adjustment strategy.

[0076] A human-machine reinforcement learning method based on multi-dimensional human feedback fusion, comprising the following steps:

[0077] (1) Generative Adversarial Imitation Learning

[0078] Generative Adversarial Imitation Learning (GAIL) is an imitation learning technique based on Generative Adversarial Networks (GANs). It does not directly rely on explicit reward functions, but learns strategies by imitating the behavior of experts. GAIL consists of a generator (i.e., proxy strategy) and a discriminator. In GAIL, the task of the discriminator is to distinguish between state-action pairs generated by the expert strategy and state-action pairs generated by the proxy strategy. In this process, the discriminator provides a dynamic reward signal to the generator to guide its learning process. The generator (the proxy's strategy) tries to imitate the behavior of the expert and adjusts its strategy based on the feedback provided by the discriminator in an attempt to deceive the discriminator into believing that the generated state-action pairs come from the expert. Through this adversarial training process, the generator continuously optimizes its strategy to get infinitely close to the expert strategy, and finally reaches a Nash equilibrium, that is, the behavior of the generator cannot be effectively distinguished by the discriminator, and the proxy strategy successfully imitates the behavior of the expert.

[0079] (1-1) Demonstration Modeling

[0080] Demonstrations are a direct and intuitive way for humans to express their preferences. In imitation learning, an algorithm learns how to perform a specific task by observing how an expert performs that task. Expert demonstrations typically consist of a series of state-action pairs that reflect the expert's strategy when handling the task. We assume that the algorithm has access to a dataset of demonstrations generated by an expert. in For Expert E i A finite sequence of state-action pairs over a time horizon T. These data are usually used to train a learning algorithm so that the agent can imitate the expert E. i Strategy and ultimately reach or exceed the performance levels of experts.

[0081] (1-2) Algorithm Design

[0082] The optimization objective of GAIL can be expressed as a minimax problem, which aims to minimize the difference in occupancy measure between the agent strategy and the expert strategy (occupancy measure is a probability distribution that measures the frequency of state-action pairs under a given strategy). The objective function of GAIL is as follows:

[0083]

[0084] where π E is the expert strategy, π θis the agent policy. D(s,a) is the output of the discriminator for a state-action pair (s,a), which is close to 1 when it thinks the pair is generated by an expert and close to 0 when it thinks the pair is generated by the agent.

[0085] In the GAIL framework, the discriminator is trained to distinguish samples generated by the expert strategy and the proxy strategy by minimizing the binary cross entropy loss. The loss function is as follows:

[0086]

[0087] After training, the output of the discriminator for the generated samples is converted into the agent's reward function to guide the agent's policy optimization:

[0088] R D (s, a) = -log(1-D(s, a))

[0089] The generator uses this reward to adjust the policy parameters via a reinforcement learning algorithm to maximize the cumulative reward until the generated state-action pairs are indistinguishable from expert demonstrations.

[0090] (2) Preference learning

[0091] Traditional reinforcement learning (RL) relies on a well-defined reward function to guide the learning process. However, in many practical applications, it is extremely difficult to define a comprehensive and effective reward function. Therefore, human preference-based reinforcement learning (PbRL) came into being. Its core idea is to construct a reward function by learning directly from human preferences, so as to better reflect human values ​​and expectations.

[0092] (2-1) Preference Modeling

[0093] Another way for humans to provide information is to provide preferences for certain trajectory segments, which allows people to compare different trajectory segments of the agent and thus indirectly guide the agent's strategy. In general, the agent follows a strategy π to generate trajectory datasets such that Trajectories can be further divided into trajectory segments. Typically, we present the agent with different trajectory segments σ, which are shorter parts of the full trajectory, formalized as a sequence of state-action pairs of length H To collect human preferences, we sample a set of segment pairs (σ0, σ1) from the trajectory, and human annotators give feedback indicating their preferences. This preference is denoted by y comp ∈{(1, 0), (0, 1), (0.5, 0.5)}, where σ0>σ1 means σ0 is better, σ1>σ0 means σ1 is better, and σ1=σ0 means the two are indistinguishable. We regard the preference query and its corresponding preference feedback as a triple (σ0, σ1, y comp) is stored in the preference feedback dataset P.

[0094] (2-2) Algorithm Design

[0095] Unlike the standard RL setting, PbRL considers a Markov decision process (MDP\R) without reward. In fact, the whole process is not without reward function, but the reward function is gradually learned by humans providing preferences between trajectory segments. In the preference-based RL algorithm, the policy π θ And the reward function Updated as follows:

[0096] Step 1: (Agent Learning) Policy π θ Continuously interact with the environment to collect trajectories and use the updated The agent policy uses existing RL algorithms to update θ to maximize the sum of rewards

[0097] Step 2: (Reward Learning) Sample a pair of segments (σ0, σ1) from the trajectory, and the human gives feedback indicating his preference for the segment pair. Reward Function Optimize based on human feedback. Repeat steps 1 and 2.

[0098] Following the Bradley-Terry model, using the preference predictor P ψ To model the reward function

[0099]

[0100] When given a stored triple (σ0, σ1, y comp ), we can train the reward function by minimizing the cross entropy loss between the preference predictor and the actual labeled preference.

[0101]

[0102] where y comp (i), i∈{1,2} is an indicator when the trajectory segment σ1 is preferred, y comp (i) = 1. Once the reward function Optimized for human preferences, preference-based RL algorithms can use any existing RL algorithm to train an RL agent and The predicted reward of is considered as the reward function returned from the environment.

[0103] (3) Phased optimization strategy based on adaptive weights

[0104] Directly adding the demonstration reward and the preference reward will make it difficult for the agent to stably learn the optimal policy. This is because GAIL aims to evaluate the similarity between the agent's behavior and the expert's behavior and give a differential reward, while preference learning provides a static reward based on the subjective evaluation of the trajectory segment by humans. These two rewards are inconsistent in their goals and nature, which may lead to conflicts in optimization goals and make it difficult for the agent to optimize simultaneously.

[0105] In order to deal with the conflict problem caused by the direct combination of the two rewards, a phased optimization strategy based on dynamic adaptive weighting is adopted. The core idea of ​​this strategy is to flexibly adjust the weights from different feedback sources according to the actual performance of the agent in the task. This application divides the agent training process into a two-stage optimization process: (1) Phase 1: The agent quickly learns the basic strategy of the task based on the expert's demonstration data. (2) Phase 2: Based on the preference data provided by the user, the basic strategy is refined and fine-tuned. In Phase 1, we quickly optimize the imitation learning by gradually increasing the proportion of demonstration rewards. Similarly, in Phase 2, user preferences are learned by gradually increasing the weight of preference rewards.

[0106] How to measure when to switch between stages is a key issue. By observing whether the agent has mastered the basic behavior pattern. Since this application is conducted in a Markov decision environment without a true reward function, the strategy consistency indicator is used to measure whether the agent has learned the expert's strategy. The specific definition of strategy consistency is as follows:

[0107]

[0108]

[0109] Where AC(s) refers to the consistency of behavior in state s across all rounds (defined by the standard deviation of actions in that state), n is the current round number, and M is the number of all states included in the expert demonstration. Given a threshold PC for policy consistency threshold =0.8. When the strategy consistency of a certain round is greater than or equal to the threshold, the agent's training enters the second stage.

[0110] After the transition to the next stage, a time growth factor based on the stability of the agent strategy must be introduced to promote the update of the preference weight. A benchmark growth amount is set to ensure that the preference weight grows gradually after entering the second stage. At the same time, the strategy stability is calculated by detecting the variance of the cumulative rewards of multiple rounds, which is defined as follows:

[0111]

[0112] Among them, N total is the total number of rounds, N P1is the number of rounds experienced in the first stage, m is the number of rounds experienced since the start of the second stage, and R i It is the cumulative reward in the i-th round starting from the second stage.

[0113] The specific steps for dynamic updating of two weights are given:

[0114] (1) Initialization: First, initialize the weights, setting the demonstration weight α and the preference weight β to be equal to ensure that the two feedbacks have equal influence in the early stages of learning: α = β = 0.5

[0115] (2 Weight adaptive adjustment:

[0116] Phase 1: As the expert demonstration is imitated, the demonstration weight is gradually increased until a preset threshold is reached.

[0117]

[0118] Among them, λ is the learning rate and PC(n) is the policy consistency obtained in the nth round.

[0119] Phase 2: Based on the user’s preference feedback, the preference weight is continuously increased until the strategy converges:

[0120]

[0121] Where λ is the learning rate, ε is the baseline growth amount to ensure that the preference weights grow steadily even when the policy stabilizes, and m is the number of episodes since the start of the second phase.

[0122] (4) Conflict coordination strategies

[0123] In order to resolve the potential conflict between demonstration and preference. First, we need to identify when there is a conflict between demonstration and preference. This is achieved by evaluating the performance of the reward provided by the GAIL discriminator and the preference reward on the same state-action pair. When one reward mechanism gives a high reward and the other gives a low reward, we determine that the state-action pair is in conflict. The specific judgment criteria are: a reward value greater than 0.5 is considered a high reward, and a reward value less than 0.5 is considered a low reward. That is:

[0124]

[0125] For these conflicting states, a staged strategy is adopted to handle them. If a conflict is found in the first stage, we adjust the preference reward weight of the corresponding state-action pair to zero; if a conflict is found in the second stage, the demonstration reward weight is reduced to zero. This processing method is intended to reduce the interference of confusing signals on the agent learning process.

[0126] The present application is able to achieve optimal performance under suboptimal demonstration conditions by gradually increasing the utilization of preference feedback.

[0127] This application reduces the reliance on human preferences by approximately 30%, thereby reducing the necessity for human labeled data.

[0128] Figure 2 The workflow of this application is described, which accurately and efficiently learns the human reward function from multiple data sources. In this application, we focus on expert demonstrations and user preferences. Specifically, the agent learns in the context of MDP\R, assuming that there are experts in the loop who provide demonstrations (demonstrations are available from the beginning of the experiment) and annotators who provide preferences (preferences are collected during agent training). The goal of the agent is to try to maximize the true reward function inferred from human demonstrations and preferences to be as close as possible to human expected behavior. Formally, this application focuses on the following policy optimization problem:

[0129]

[0130] Where S is the state space, A is the action space, P: S×A×S→[0,1] is the state transition function (i.e., the probability that the agent transitions to state s′ when taking action a in state s), R: S×A→R is the reward function (the reward the agent receives when it transitions from s to s′ under action a), and γ∈[0,1] is the discount factor (indicating how much the agent values ​​immediate rewards compared to future rewards). The policy π is a mapping from S to a probability distribution over A, where π(a|s) specifies the probability of taking action a in state s.

[0131] The training algorithm diagram used in this application is as follows:

[0132]

[0133] Then, the method of the present application is applied to Atari games, such as Figure 3 The following four environments are used: Beamrider, Breakout, Hero, and Q*bert to test the effectiveness of this application, and the demonstration data provided in the Hester dataset is used as the input of the GAIL algorithm. In addition, in order to obtain the user's preference data, the model first needs to generate short video clips generated by the agent's interaction with the environment and upload them to the cloud. Figure 4 The web application interface built by Django is shown in the figure. These clips downloaded from the cloud are evaluated and feedback is submitted. Finally, the reward function is trained and iterated based on the feedback. In addition, four feedback plans are formulated to examine the impact of different feedback amounts on agent performance. The full schedule contains 6,800 labels, and the other three schedules reduce the total amount of feedback by 2 times, 4 times, and 6 times, respectively.

[0134] This application is implemented using Tensorflow in Python, which is divided into a feedback server and an algorithm application. In this embodiment, MDF-HMRL is deployed on a server equipped with an Intel i9-10900K 3.70GHz CPU and an NVIDIA GTX 3090Ti GPU to verify whether it is effective in Atari games. We use a small batch size of 64 samples, set the discount factor to 0.99, use the Adam optimizer, and set its step size to a value of 0.0003, while the overall learning rate is set to 0.0001. The discriminator has two hidden layers, containing 256 and 128 neurons respectively, and is nonlinearly transformed by the tanh activation function.

[0135] In Atari games, imitation learning is generally difficult. As shown in the data in Table 1, the method of this application surpasses the imitation baseline in all games, even with minimal feedback. This shows that integrating preference feedback into demonstration can significantly supplement the missing reward function. In addition, when comparing the method of this application with RLHP, it can be seen that the method of this application can achieve excellent performance with fewer labels. According to the data, the method of this application reduces the reliance on human preferences by about 30%, thereby reducing the need for human labeled data. In summary, the experiments confirm the effectiveness of the method of this application in guided demonstration and preference learning, especially in the challenging environment of traditional imitation learning.

[0136]

[0137] Table 1

[0138] This application is a human-machine reinforcement learning method based on multi-dimensional human feedback fusion to meet the challenges of designing reward functions in traditional reinforcement learning. Demonstration rewards and preference rewards are fused through a dynamic adaptive weighted strategy. By building a more comprehensive and unified reward model, the learning efficiency and performance of the agent in complex tasks are effectively improved. Experiments in the Atari game environment verify that this application uses the synergy between demonstration and preference to optimize performance, especially when the quality of demonstration data is poor. In addition, the reliance on large amounts of manually labeled data is reduced by about 30%, paving the way for practical applications. A large number of experiments have shown that this application achieves higher algorithm performance at a lower labor cost on a variety of challenging tasks.

[0139] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A human-machine reinforcement learning method based on multi-dimensional human feedback fusion, characterized in that: The following steps are involved: Through expert demonstration, generative adversarial imitation learning is performed based on the strategy generator and the discriminator, so that the strategy learned by the strategy generator cannot be distinguished by the discriminator, and the demonstration reward function is obtained; Using the user's trajectory preference, we obtain a preference reward function that meets the user's preference; The demonstration reward function and the preference reward function are integrated to obtain a unified reward function, and the conflict between the demonstration reward function and the preference reward function is resolved through a dynamic reward weight adjustment strategy.

2. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 1, characterized in that: The policy generator is used to imitate the expert demonstration and adjust the agent strategy according to the feedback provided by the discriminator to try to deceive the discriminator and make it believe that the generated state-action pairs come from the expert demonstration. The discriminator is used to distinguish between the state-action pairs generated by the expert demonstration and the state-action pairs generated by the policy generator, and provides a dynamic reward signal to the policy generator to guide the learning process. Through training with generative adversarial imitation learning, the policy generator continuously optimizes the agent strategy to get infinitely close to the expert strategy, so that the behavior of the policy generator cannot be effectively distinguished by the discriminator, and the policy generator successfully imitates the behavior of the expert.

3. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 2 is characterized in that: The specific process of obtaining the demonstration reward function is as follows: S1-1: Assume that the dataset of expert demonstration is in, For Expert E i A finite sequence of state-action pairs over time horizon T, a dataset of expert demonstrations is used to train generative adversarial imitation learning, so that the policy generator can imitate the expert E i Strategy S1-2: Assume that the optimization goal of generative adversarial imitation learning is a minimax problem, the goal is to minimize the occupancy metric difference between the agent strategy and the expert strategy. The objective function of generative adversarial imitation learning is as follows: Among them, π E is the expert strategy, π θ is the agent strategy, D(s,a) is the output of the discriminator for the state-action pair (s,a), when the discriminator believes that the state-action pair is generated by the expert, the output is close to 1; when the discriminator believes that the state-action pair is generated by the agent, the output is close to 0; S1-3: The discriminator is trained to distinguish samples generated by the expert strategy and the proxy strategy by minimizing the binary cross entropy loss. The loss function is as follows: S1-4: After training, the output of the discriminator is converted into the reward of the generator to guide the agent strategy optimization: R D (s,a)=-log(1-D(s,a)) The generator uses the reward obtained from the conversion to adjust the agent policy parameters through the reinforcement learning algorithm to maximize the cumulative reward until the state-action pairs generated by the generator are indistinguishable from the expert demonstrations, and the demonstration reward function R is obtained. D (s, a).

4. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 1, characterized in that: The preference reward function that meets the user's preference is obtained by utilizing the user's preference for the trajectory, specifically: S2-1: The agent follows the strategy π to generate a trajectory dataset such that The agent is presented with different trajectory segments σ, formalized as a sequence of state-action pairs of length H A set of segment pairs (σ0, σ1) is sampled from the trajectory, and the preference feedback given by the user indicating his preference is obtained. The preference query and the corresponding preference feedback are taken as a triple (σ0, σ1, y comp ) is stored in the preference feedback dataset P; S2-2: Preference-based reinforcement learning PbRL Consider a Markov decision process MDP\R without reward, and gradually learn the reward function by providing preferences between state-action pairs by users Update proxy strategy π θ And the reward function Will The prediction reward R predict (s, a) as the preference reward function.

5. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 4 is characterized in that: The update agent strategy π θ And the reward function Specifically: S2-21: Proxy Strategy π θ Continuously interact with the environment to collect trajectories and use the updated reward function Perform reward estimation, proxy strategy π θ Use the RL algorithm to update θ to maximize the sum reward function S2-22: Sample the segment pair (σ0, σ1) from the trajectory, obtain the user's preference feedback indicating his preference for the segment pair, and the reward function Optimize based on preference feedback; Repeat S2-21 and S2-22, following the Bradley-Terry model, using the preference predictor P ψ To model the reward function When given a stored triple (σ0, σ1, y comp ), the reward function is trained by minimizing the cross entropy loss between the preference predictor and the actual trajectory preference. Among them, y comp (i), i∈{1,2} is an indicator when the trajectory segment σ1 is preferred, y comp (i)=1; when the reward function Optimized according to user preferences. The prediction reward R predict (s, a) is considered as the preference reward function returned from the environment.

6. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 5, characterized in that: The specific process of obtaining a unified reward function is: The strategy consistency indicator is used to measure whether the agent strategy has learned the expert strategy. The specific definition of strategy consistency is as follows: Where AC(s) refers to the behavioral consistency of state s in all rounds, which is defined by the standard deviation of actions in that state, n is the current round number, M is the number of all states included in the expert demonstration, PC(n) is the policy consistency obtained in the nth round, and a threshold of policy consistency is given. When the policy consistency of a certain round is greater than or equal to the threshold, the agent's training enters the second stage; After entering the second stage, a time growth factor based on the stability of the agent strategy is introduced to promote the update of the preference reward weight. A benchmark growth amount is set to ensure that the preference weight increases gradually after entering the second stage. At the same time, the strategy stability is calculated by detecting the variance of the cumulative rewards of multiple rounds. Its definition is as follows: Among them, N total is the total number of rounds, N P1 is the number of rounds experienced in the first stage, m is the number of rounds experienced since the start of the second stage, and R i is the cumulative preference reward in the i-th round starting from the second stage; The unified reward function has the following form: R total (s, a)=α·R D (s, a|ω)+β·R predict (s, a|ψ) Among them, α and β are weight parameters, representing the proportion of demonstration reward and preference reward in the unified reward model respectively.

7. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 6 is characterized in that: The specific steps for dynamically updating the weights of demonstration rewards and preference rewards are as follows: S4-1, first initialize the weights, set the demonstration reward weight α and the preference reward weight β to be equal, to ensure that the two feedbacks have equal influence in the initial stage, α = β = 0.5; S4-2, weight adaptive adjustment: As the expert demonstration is imitated, the demonstration reward weight is gradually increased until a preset threshold is reached; Where λ is the learning rate, PC(n) is the policy consistency obtained in the nth round; According to the user's trajectory preference, the weight of the preference reward is continuously increased until the strategy converges: Where λ is the learning rate, ε is the baseline growth amount to ensure that the preference weights grow steadily even when the policy stabilizes, and m is the number of episodes since the start of the second phase.

8. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 7, characterized in that: By evaluating the performance of the demonstration reward and the preference reward on the same state-action pair, it is determined whether the demonstration reward and the preference reward conflict. When one reward mechanism gives a high reward and the other gives a low reward, the state-action pair is considered to be in conflict. If a conflict is found in the first stage, the preferred reward weight of the corresponding state-action pair is adjusted to zero; If a conflict is found in the second phase, the demonstration reward weight is reduced to zero.

9. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 8, characterized in that: The specific criteria for judging whether the demonstration reward and the preference reward conflict are as follows: a reward value exceeding 0.5 is considered a high reward, while a reward value below 0.5 is considered a low reward, that is: Among them, R D Representative Demo Reward, R predict Represents a preference for reward.

10. The human-machine reinforcement learning method based on multi-dimensional human feedback fusion according to claim 9, characterized in that: Under the suboptimal demonstration conditions in the first stage, the utilization of user preference feedback is gradually increased in the second stage.

Citation Information

Cited By

  • Cboth case generation method and device based on large model, equipment and medium

    CN120257948A

  • Intelligent content evaluation and optimization method and system based on multi-standard preference learning

    CN120494074A

  • Intelligent content evaluation and optimization method and system based on multi-criteria preference learning

    CN120494074B

  • Self-adaptive control method and system of autonomous mobile platform

    CN120848181A

  • Adaptive control method and system for autonomous mobile platform

    CN120848181B