A robot learning method based on model-based generative adversarial interactive imitation learning

Abstract

The application relates to a robot learning method based on model-based generative adversarial interactive imitation learning, which combines the advantages of model-based reinforcement learning, generative adversarial imitation learning and interactive reinforcement learning to form a model-based generative adversarial interactive imitation learning (MAILDH) method, and solves the problem of slow robot learning speed. Firstly, a forward dynamic model is simultaneously learned in the process of generative adversarial imitation learning, the model is used to generate simulation data to train the generator and discriminator parts in the generative adversarial imitation learning framework, so as to improve the sample utilization rate. In addition, different from the traditional generative adversarial imitation learning which learns from demonstration, the application also combines the judgment of human beings on the behavior of the robot, weakens the restriction of demonstration quality on the learning performance of the robot, achieves the effect of reaching or exceeding the demonstration performance, learns a better control strategy, and can improve the stability of the strategy and be suitable for large and complex task control.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robot learning technology, and particularly to model-based reinforcement learning, generative adversarial imitation learning, and interactive reinforcement learning. By constructing models and utilizing the evaluation of human trainers, it improves sample utilization and robot performance, enabling faster learning of better strategies. Specifically, it relates to a model-based generative adversarial interactive imitation learning robot learning method. Background Technology

[0002] Reinforcement learning (RL) attempts to teach robots an optimal policy through trial and error by interacting with the real world. With the development of deep neural networks, deep reinforcement learning (DRL) has achieved great success in many simulation tasks, such as games and mechanical manipulation. However, designing an effective reward function for each task is difficult, even impractical, especially in large and high-dimensional environments, which makes traditional DRL methods unsuitable for the real world.

[0003] In the real world, demonstrations are easier to provide than carefully designed reward functions, and this idea has led to the development of imitation learning. Robots can learn to perform tasks from expert demonstrations consisting of several state-action pair sequences. Behavior cloning (BC) is the simplest method in imitation learning, mapping states to optimal behaviors through supervised learning. However, BC requires a large amount of data to learn, cannot generalize to unseen states, and is typically used in reinforcement learning for policy initialization. Another approach to imitation learning is called inverse reinforcement learning. It uses a cost function extracted from expert trajectories to learn policies through reinforcement learning. However, many proposed inverse reinforcement learning algorithms require a model to solve a series of planning or reinforcement learning problems within an inner loop, which consumes significant runtime and limits their use in large and complex tasks. Furthermore, if the planning problem is not optimal, the robot's performance may also degrade significantly.

[0004] To address this issue, Generative Adversarial Imitation Learning (GAIL) was proposed—a general model-free imitation learning method that allows robots to learn policies directly from expert trajectories in large environments and extends inverse reinforcement learning to large environments. However, as a model-free method, despite not requiring a pre-defined reward function, GAIL suffers from low sampling efficiency in environmental interactions. To improve sampling efficiency, model-based reinforcement learning was proposed. Model-based reinforcement learning is more applicable to real-world scenarios than model-free methods. It can generate simulation data using existing or learned dynamic models to accelerate policy learning. However, model-based reinforcement learning suffers from model bias, which negatively impacts the learning process, slowing down or even reducing learning effectiveness. Furthermore, GAIL performance is constrained by demonstration quality, while Interactive Reinforcement Learning (InteractiveRL) has been shown to generally outperform trainers. Based on reward shaping methods in traditional reinforcement learning, Interactive Reinforcement Learning can solve the problem of low sampling efficiency in traditional and deep reinforcement learning, and even allow non-technical personnel to provide feedback by evaluating the robot's behavior for training, thereby improving the robot's learning effectiveness.

[0005] Therefore, combining model-based reinforcement learning, interactive reinforcement learning, and the GAIL method can improve sample utilization and extend to large and complex tasks. Furthermore, it is hypothesized that the complementary effects of demonstration and human evaluation feedback enhance policy stability, leading to better policies that approach or even surpass the demonstration.

[0006] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0007] (1) Most existing inverse reinforcement learning algorithms need to utilize model information in the inner loop, which consumes a lot of running time and limits their application in large and complex tasks; if the planning problem does not obtain the optimal solution, the performance of the algorithm will be greatly reduced.

[0008] (2) Traditional reinforcement learning, deep reinforcement learning and GAIL sampling efficiency is low. They need to interact with the environment multiple times to learn a more ideal policy, which consumes a lot of computing resources and time, thus hindering their application in reality.

[0009] (3) GAIL's performance is constrained by the quality of the demo, and it rarely, if ever, can surpass the demo. In many application scenarios, the present invention cannot provide the optimal demo in advance. The present invention aims to provide a method that can achieve near-optimal results without providing the optimal demo, while minimizing interaction with the environment and improving sample utilization.

[0010] Therefore, the difficulty in solving the above problems and defects is as follows:

[0011] 1. How to apply model-based reinforcement learning and interactive reinforcement learning simultaneously on the GAIL framework.

[0012] 2. How to obtain an accurate forward dynamic model and apply the model to the framework of generative adversarial imitation learning.

[0013] 3. How to obtain an accurate human reward network model to provide effective feedback for robot learning.

[0014] The significance of solving the above problems and defects is as follows:

[0015] 1. A novel framework and technical solution is proposed, which combines model-based reinforcement learning, interactive reinforcement learning and GAIL method to improve sampling efficiency, thereby improving the learning speed and stability of the robot, breaking away from the limitations of expert demonstration quality, and enabling the robot to learn better strategies.

[0016] 2. It broadens the application scope of traditional generative adversarial imitation learning, making it more suitable for real-world scenarios.

[0017] 3. The model is continuously improved during the training process, rather than being fully pre-trained, making it more realistic. The introduction of human evaluation makes the robot's learning more directional, which aligns with future trends. Summary of the Invention

[0018] In view of this, and in view of the problems and defects of the existing technology, the purpose of the embodiments of this application is to provide a model-based generative adversarial interaction imitation learning method for robot learning.

[0019] To achieve the above objectives, the present invention provides the following technical solution:

[0020] According to a first aspect of the present invention, a model-based generative adversarial interaction imitation learning robot learning method is provided, comprising the following steps:

[0021] Expert trajectories are sampled from expert policies, and the network parameters of robot policy, discriminator, human reward model and forward dynamic model are initialized.

[0022] The robot pre-trained a human reward model using samples from the demonstration, as well as random samples drawn from the task.

[0023] The robot samples the generated trajectory from the current strategy and the simulated trajectory from the forward dynamic model; it determines whether it has received human feedback. If so, it updates the human reward model; otherwise, it updates the discriminator network and the forward dynamic model.

[0024] The robot uses the loss function extracted from the discriminator network and the human feedback output by the human reward model to perform interactive reinforcement learning. It determines whether a stable policy has been learned. If so, the process ends; otherwise, the robot's policy is updated.

[0025] Model-based generative adversarial interaction imitation learning methods can be applied to robot task learning, such as social robots, to improve their efficiency in imitating the behavior of human experts, enabling trained robots to respond better to constantly changing environments.

[0026] As a further aspect of the present invention, the model-based generative adversarial interaction imitation learning robot learning method further includes a model-based generative adversarial interaction imitation learning (MAILDH) framework formed by model-based reinforcement learning, generative adversarial imitation learning, and interactive reinforcement learning. During the training process, the generative adversarial interaction imitation learning learns a forward dynamic model to generate simulated data. The data of real interaction with the environment is used to train the generator and discriminator; it is also used to train a human reward model so that the robot can learn by imitation from demonstrations or from human evaluation feedback.

[0027] As a further aspect of the present invention, the forward dynamic model adopts a reinforcement learning method based on the Dyna type, and uses a feedforward neural network to approximate the forward dynamic model; wherein, in each new iteration during the training process, the robot collects samples obtained from the real interaction between the current policy and the environment, and uses the samples to update the forward dynamic model.

[0028] As a further aspect of the present invention, the human reward model is also used to assign all state-action pairs [s] to the demonstration before the training process begins. d a d Allocate an artificial reward r h =+1, T random samples were drawn in the task; the current state s is selected. t and a random output action a t And manually allocate rewards r h ;[s d a d ]、[s t a t The reward r given by the human trainer is used as input to the human reward model. h It was used as a label to pre-train a human reward model. Where s d Corresponding to the state in the demonstration, a d Corresponding to the action in the demonstration, s t Given the current state, a t The action selected for the current time, r h The reward is artificial.

[0029] As a further aspect of the present invention, after the human reward model begins training, a pre-trained human reward model is loaded, and all tuples [s] are included. d a d +1], [s t a t r h The samples are stored in the replay buffer B to obtain samples for further training of the human reward model.

[0030] As a further aspect of the present invention, the MAILDH uses the simulated trajectory τ obtained by the forward dynamic model. s To update the generator and discriminator, use the demo trajectory τ E The actual trajectory τ of the robot's interaction with the environment under the current strategy. agent and simulated trajectory τ s Update discriminator D to reduce loss. Where L D Let E represent the loss function of the discriminator. τ This represents the trajectory sample τ = {s0, a0, s1, a1, ... s} T a T} experience expectation (i.e. γ is the discount factor for τ agent τ s Let c(s, a) = log(D(s, a)), for τ E c(s, a) = log(1 - D(s, a))); The cost function in GAIL derived from the latest discriminator D and the trained human reward model H will be integrated into a new reward function:

[0031] c maildh (s, a) = c gail (s, a) + H(s, a)

[0032] Where c maildh (s, a) represents the cost function of MAILDH, c gail (s, a) = log(D(s, a)) represents the cost function of GAIL, H(s, a) represents the predicted reward of the human reward model, and s and a represent the input state and action, respectively.

[0033] The reward function is used to update the policy π with PPO, and the above adversarial training process is repeated until the desired policy is obtained.

[0034] As a further aspect of the present invention, the MAILDH method finds the saddle point (π, D) in the expression:

[0035]

[0036] Where E π [log(D(s, a))+H(s, a)] represents the expected discounted return along the trajectory generated by strategy π. This represents the expected discounted return of the demonstration trajectory.

[0037] As a further aspect of the present invention, MAILDH also includes pre-training a human reward model before training begins, wherein the samples used to update the human reward model are partly derived from state-action pairs in expert demonstrations and partly derived from randomly sampled state-action pairs.

[0038] As a further aspect of the present invention, the model-based generative adversarial interaction imitation learning robot learning method further includes a generative adversarial imitation learning part, a forward dynamic model part, and a human reward model part;

[0039] The generative adversarial imitation learning part involves finding the saddle point (π, D) in the expression:

[0040] -λH(π)+E π [log(D(s, a))]+E πE [log(1-D(s,a))],

[0041] in E is the γ-discounted causal entropy of strategy π, where λ is the weight of the entropy H(π); π [log(D(s, a))] represents the expected discounted return along the trajectory generated by strategy π. Indicates along the expert strategy π E The expected value of the discounted return generated by the trajectory.

[0042] in It is the γ-discounted causal entropy of strategy π, where λ is the weight of entropy H(π);

[0043] The forward dynamic model part learns a forward dynamic model using real samples of robot-environment interaction, and generates simulated samples for updating the discriminator and generator by using virtual interaction between the robot and the forward dynamic model.

[0044] As a further aspect of the present invention, the human reward model portion minimizes the loss. The updated human reward model H, where L H Let H(s, a) represent the loss function, H(s, a) be the estimated human reward model, n be the number of samples, and r be the number of samples. h Artificial rewards are used as tags.

[0045] Compared with existing technologies, the technical solution of the model-based generative adversarial interaction imitation learning robot learning method provided by this invention can include the following beneficial effects:

[0046] This invention combines model-based reinforcement learning, human evaluation feedback-based interactive reinforcement learning, and generative adversarial imitation learning (GAIL) framework to form model-based generative adversarial interactive imitation learning (MAILDH).

[0047] This invention enables robots to leverage the synergy of demonstration and human evaluation, drawing upon both expert demonstrations and human knowledge to overcome the limitations of demonstration quality and achieve learning effects that surpass or even exceed those of demonstrations. This allows for the effective acquisition of superior strategies and improved strategy stability. The forward dynamic model further enhances sample efficiency and robustness. Simultaneously, the introduction of the forward dynamic model and the MAILDH method with human rewards broadens the application scope of traditional methods, improving sample efficiency, training speed, and stability.

[0048] This invention was tested on eight physics-based control tasks ranging from low to high dimensions and from discrete to continuous action spaces. The results show that, given either optimal or suboptimal expert demonstrations, the MAILDH robot can always learn a better policy than GAIL using fewer real samples. This method can be applied to robot task learning, such as social robots, enabling them to better mimic the behavior of human experts and perform a range of tasks.

[0049] These or other aspects of this application will become more apparent from the following description of embodiments. It should be understood that the foregoing general description and the following detailed description are merely illustrative and explanatory, and are not intended to limit the scope of this application. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a model-based generative adversarial interaction imitation learning robot learning method provided in an embodiment of the present invention.

[0052] Figure 2These are screenshots of eight physics-based control tasks provided in the embodiments of the present invention; in the figures: (a) CartPole; (b) Mountain Car; (c) Acrobot; (d) LunarLander; (e) Mountain Car Continuous; (f) Lunar Lander Continuous; (g) Hopper PyBullet; (h) HalfCheetah PyBullet.

[0053] Figure 3 This is a schematic diagram of the learning curves of different robots in the Cart Pole task under the given optimal expert demonstration provided by the embodiments of the present invention.

[0054] Figure 4 This is a schematic diagram of the learning curves of different robots in the Cart Pole task under a given suboptimal expert demonstration provided by an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of the learning curves of different robots in the MountainCar task under a given suboptimal expert demonstration provided by an embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram of the learning curves of different robots in the Acrobot task under the given suboptimal expert demonstration provided in the embodiments of the present invention.

[0057] Figure 7 This is a schematic diagram of the learning curves of different robots in the LunarLander task under a given suboptimal expert demonstration provided by an embodiment of the present invention.

[0058] Figure 8 This is a schematic diagram of the learning curves of different robots in the MountainCar Continuous task under a given suboptimal expert demonstration provided by an embodiment of the present invention.

[0059] Figure 9 This is a schematic diagram of the learning curves of different robots in the LunarLander Continuous task under a given suboptimal expert demonstration provided by an embodiment of the present invention.

[0060] Figure 10 This is a schematic diagram of the learning curves of different robots in the HopperPyBullet task under a given suboptimal expert demonstration provided by an embodiment of the present invention.

[0061] Figure 11 This is a schematic diagram of the learning curves of different robots in the HalfCheetah PyBullet task under a given suboptimal expert demonstration provided in the embodiments of the present invention.

[0062] Figure 12 This embodiment of the invention represents the number of interactions required for the MAILDH robot to achieve optimal performance in the Lunar Lander task under different model errors, compared with other agents. Note that the horizontal axis represents the error of initializing the dynamic model (A: 0.8, B: 2.9, C: 3.7, D: 4.2, E: 5 (random model)); the error bars and shaded areas indicate a 95% confidence level.

[0063] Figure 13 This is the number of samples required for the GAIL, MGAIL, and MAILDH robots provided in this invention example to achieve the same performance in eight tasks.

[0064] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0065] The present application will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0066] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0069] To address the problems existing in the prior art, this invention provides a model-based generative adversarial interaction imitation learning method for robot learning. The invention will be described in detail below with reference to the accompanying drawings.

[0070] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0071] like Figure 1 The diagram shows a flowchart of a model-based generative adversarial interaction imitation learning robot learning method provided in this embodiment of the invention. This invention provides a model-based generative adversarial interaction imitation learning robot learning method, comprising the following steps:

[0072] Expert trajectories are sampled from expert policies, and the network parameters of robot policy, discriminator, human reward model and forward dynamic model are initialized.

[0073] The robot pre-trained a human reward model using samples from the demonstration, as well as random samples drawn from the task.

[0074] The robot samples the generated trajectory from the current strategy and the simulated trajectory from the forward dynamic model; it determines whether it has received human feedback. If so, it updates the human reward model; otherwise, it updates the discriminator network and the forward dynamic model.

[0075] The robot uses the loss function extracted from the discriminator network and the human feedback output by the human reward model to perform interactive reinforcement learning. It determines whether a stable policy has been learned. If so, the process ends; otherwise, the robot's policy is updated.

[0076] This invention presents a model-based generative adversarial interactive imitation learning (MAILDH) robot learning method, which combines model-based reinforcement learning, generative adversarial imitation learning, and interactive reinforcement learning frameworks to form MAILDH. During training, MAILDH learns a forward dynamic model to generate simulated data, which is then combined with data from real-world interactions with the environment to train the generator and discriminator, accelerating robot training. It also trains a human reward model, enabling the robot to learn not only from demonstrations but also from human feedback.

[0077] The forward dynamic model employs a Dyna-type reinforcement learning method, using a feedforward neural network to approximate the forward dynamic model. In each iteration of the MAILDH method during training, the robot collects samples obtained from real interactions between the current policy and the environment, using these samples to update the forward dynamic model. The accuracy of the forward dynamic model plays a crucial role in the stability of the learning process. Unlike ordinary forward models, the performance of the forward dynamic model in the MAILDH method is improved in two ways: First, the forward dynamic model treats actions as operators in the state space. Since the states and actions in the samples are sampled from completely different distributions, they are first represented in a shared space, encoded using two independent neural networks, and then combined into a vector using the Hadamard product. Furthermore, to model multi-step dependencies, a Gate Repeated Unit (GRU) layer is used as part of the state encoder, employing recurrent connections from previous states. Finally, the discriminator and generator parts in the generative adversarial learning are updated using information predicted by the forward dynamic model and real samples obtained from the robot's interactions with the environment.

[0078] In the human reward model, the MAILDH method, before the training process begins, in order to reduce the burden on human trainers, assigns all state-action pairs [s] to the demonstration. d a d Allocate an artificial reward r h =+1, In addition, this invention also extracts T random samples in the task (the number of T is selected according to the complexity of the task, and the more complex the task, the larger the number of T will be), that is, selects the current state s. t and a random output action a t And manually allocate rewards r h [s] d a d ]、[s t a t The reward r given by the human trainer is used as input to the human reward model. h The labels are used to pre-train the human reward model. After training begins, the pre-trained human reward model is loaded. As training progresses, the state-action data obtained from real interactions with the environment is evaluated by humans, who then assign labels. All tuples [s] are then processed. d a d +1], [s t a t r hThe data is stored in the replay buffer B to obtain enough samples to continue training the human reward model. It's important to note that the human reward model is adjusted continuously during training based on pre-training. All parts of the MAILDH method training process that require human evaluation are output through the human reward model, not directly provided by humans. Reward r h The reward system follows this principle: -G is awarded when the robot fails; +G is awarded when the robot achieves the goal; +1 is awarded for relatively good or potentially good behavior; and -1 is awarded for poor performance. The updates to the human reward model and policy are asynchronous, and human trainers, who may not be experts in robot design or even programming, can provide rewards at any time during training based on their evaluation of the robot's behavior, thus continuing to train the human reward model.

[0079] 1. In the generative adversarial interactive imitation learning method of the present invention, in order to accelerate the robot's learning in the adversarial imitation learning environment in this embodiment, the MAILDH method uses the simulated trajectory τ obtained by the forward dynamic model. s This is used to update the generator (i.e., the policy) and the discriminator. Specifically, the demo trajectory τ is used. E The actual trajectory τ of the robot's interaction with the environment under the current strategy. agent and simulated trajectory τ s To update the discriminator D by reducing the loss Where L D Let E represent the loss function of the discriminator. τ This represents the trajectory sample τ = {s0, a0, s1, a1, ... s} T a T} experience expectation (i.e. γ is the discount factor for τ agent τ s Let c(s, a) = log(D(s, a)), for τ E c(s, a) = log(1 - D(s, a))); The cost function in GAIL derived from the latest discriminator D and the trained human reward model H will be integrated into a new reward function:

[0080] c maildh (s, a) = c gail (s, a) + H(s, a),

[0081] Where c maildh (s, a) represents the cost function of MAILDH, c gail(s, a) = log(D(s, a)) represents the cost function of GAIL, H(s, a) represents the predicted reward of the human reward model, and s and a represent the input state and action, respectively.

[0082] This function will be used to update the policy π (i.e., the generator) with PPO. The adversarial training process described above is repeated until the desired policy is learned. In summary, the MAILDH proposed in this invention can be summarized as finding the saddle point (π, D) of the expression:

[0083]

[0084] Specifically, at the beginning of the training process, in order to reduce the burden on human trainers, this invention will demonstrate state-action pairs [s]. d a d Distribute human rewards r h =+1, will add 1 to the tuple [s d a d r h The samples are stored in a replay buffer to obtain enough samples with good behavior. Additionally, T samples are obtained using a random policy, and a human reward r is assigned. h The human feedback model is pre-trained. The number T is 500 in low-dimensional tasks and 1000 in high-dimensional tasks. During training, a human trainer, who may not be a robot design or even programming expert, can provide rewards at any time during the training process by evaluating the robot's behavior based on their understanding of the task. The human reward r... h The definition is as follows:

[0085]

[0086] G is set to different values ​​for different tasks: 1 for Cart Pole, 5 for Hopper and Half Cheetah, 100 for Mountain Car, Mountain Car Continuous, and Acrobot, and 200 for Lunar Lander and Lunar Lander Continuous. The human reward model H minimizes the loss. Update the demo. Then, in each new iteration, the robot will sample the demo trajectory τ from the demo. E And under the latest strategy, the robot trajectory τ is sampled from its interaction with the environment. agent Human trainers can provide additional rewards at any time during the training process. h To further train the human reward model.

[0087] Furthermore, MAILDH learns a forward dynamic model and uses virtual interactions between the robot and the forward dynamic model to generate simulated samples τ. s This invention uses a Dyna-based reinforcement learning method and a feedforward neural network to approximate the forward dynamic model.

[0088] This invention uses two neural networks to encode the state and action separately, and then uses a Hadamard product to combine the encoded state and action into a vector. A GRU layer is used as part of the state encoder to predict the next state using multiple previous states. In MAILDH, the forward dynamic model can be used to predict the state transition under the robot's current state and action, i.e., f(s, a) = s′. Samples [s, a, s′] from the robot's trajectory interacting with the real environment are used to reduce the loss. To continuously update the forward dynamic model.

[0089] To accelerate the robot's learning, this invention uses a simulated trajectory τ. s This is used to update the policy and discriminator. Updating the discriminator reduces the loss. Then, the updated discriminator and the human reward model are integrated into a new reward function c. maildh (s, a) = c gail (s, a) + H(s, a). This reward function will be used to update the policy. That is, it maximizes the following formula using the virtual trajectory generated by the forward dynamic model, the real trajectory of the robot's interaction with the environment, and the new reward function involving the human reward model:

[0090]

[0091] Repeat the adversarial training process described above until the desired strategy is learned.

[0092] In this embodiment, the present invention is implemented by first pre-training the human reward model before training begins. A portion of the samples used to update the human reward model comes from state-action pairs in expert demonstrations. The present invention provides r... h =+1 labels, another part comes from randomly sampled state actions, this invention is given by humans r h As a tag, r hThe choices are ±1 and ±G, with the value of G adjusted according to different environments. At this point, the invention obtains a human reward model with a certain predictive ability. This invention loads the initially trained human reward model into subsequent training to prevent the MAILDH method from using a zero-precision human reward model in the early stages of training to provide incorrect feedback and slow down the training process. Unlike other methods that first train a high-precision human feedback model and then directly load it, the human feedback model in this invention can be adjusted at any time during training to optimize the human reward model during training. Specifically, during training, the human trainer can evaluate the robot's behavior at any time based on the robot's performance in the task, thereby providing a reward r. h Human-provided rewards r h The labels will be used as the corresponding state-action pair (s, a), tuple [s, a, r] h This will also be stored in the replay buffer B. A human reward model {H:s×a→R}, approximating a feedforward neural network, is trained using samples of human interaction and demonstrations in B to predict human rewards. H is obtained by minimizing the loss... Update.

[0093] In each new iteration, the robot samples the demonstration trajectory τ from the demonstration. E Sample agent trajector trajectory τ from the interaction between the latest strategy π and the environment. agent From τ agent Extract samples [s, a, s′] and minimize the loss. To continuously update the forward dynamic model To further accelerate the robot's learning in an adversarial imitation learning environment, this invention uses The obtained simulated trajectory τ s This is used to update the generator (i.e., the policy) and discriminator. In other words, unlike other methods, it only uses τ. E τ agent To update, the method of this invention also utilizes τ s Go and update, and τ s It utilizes τ agent What is learned makes τ agent The data was fully utilized. Specifically, the Adam optimizer was used, with τ... E τ agent and τ s By reducing losses To update the discriminator D, where L D Let E represent the loss function of the discriminator. τ This represents the trajectory sample τ = {s0, a0, s1, a1, ... s} T a T} experience expectation (i.e. γ is the discount factor for τ agent τ s Let c(s, a) = log(D(s, a)), for τ E c(s, a) = log(1 - D(s, a))). It is worth noting that τ E τ agent and τ s The trajectory length is the same as the trajectory length of one round. After this, the cost function c in GAIL is derived by the latest discriminator D. gail (s, a) = log(D(s, a)) and the trained human reward model H will be integrated into a new reward function:

[0094] c maildh (s, a) = c gail (s, a) + H(s, a)

[0095] Where c maildh (s, a) represents the cost function of MAILDH, c gail (s, a) = log(D(s, a)) represents the cost function of GAIL, H(s, a) represents the predicted reward of the human reward model, and s and a represent the input state and action, respectively.

[0096] The new reward function will be used to update the policy π (i.e., the generator) with PPO. The agent's goal is to maximize:

[0097]

[0098] Repeat the adversarial training process described above until the desired strategy is obtained. In summary, the MAILDH method proposed in this invention can be summarized as finding the saddle point (π, D) of the expression:

[0099]

[0100] Where E π [log(D(s, a))+H(s, a)] represents the expected discounted return along the trajectory generated by strategy π. This represents the expected discounted return of the demonstration trajectory.

[0101] In this embodiment, the entire framework of the model-based generative adversarial interaction imitation learning robot learning method is based on the Markov Decision Process (MDP) framework. MDP can be represented by the tuple M = {S, A, P, R, γ}. π: s → p(a|s) ∈ π: S → p(A|S) is a policy that maps the state s ∈ S to the optional action a ∈ A. In the model-free setting, subsequent states are obtained from the robot's interaction with the environment. In the model-based setting, subsequent states s′ can also be obtained from the transition probability distribution p(s′|s,a) ∈ P of a stochastic dynamics model or f(s,a) = s′ ∈ F of a deterministic dynamics model. During this process, the robot can obtain feedback c(s,a) from the cost function c: S × A → R. Let E represent the expected discounted return along the trajectory generated by strategy π, where γ is the discount factor, γ∈(0,1). Similarly, E τ This represents the empirical expectation of the trajectory samples τ = {s0, a0, s1, a1, ...}. It is expressed using π. E To represent expert strategies, use τ. E This represents expert trajectory samples. The goal of reinforcement learning is to find an optimal policy:

[0102]

[0103] In this embodiment, the generative adversarial imitation learning (GAIL) part of the model-based generative adversarial interaction imitation learning robot learning method uses a discriminator D: S×A→(0,1) trained to distinguish expert state-action pairs (s, a)~τ. E And robot state-action pairs (s, a) ~ τ agent This is represented as minimization:

[0104]

[0105] The generator (i.e., policy π) maximizes E π [log(D(s, a))] is used to "trick" the discriminator, causing it to interpret the robot's state-action pair (s, a) ~ τ. agent Consider the expert state-action pair (s, a) ~ τ E The GAIL robot learns the optimal policy directly by making the distribution of the robot's state-action pairs as close as possible to the distribution of the expert trajectory. Therefore, GAIL can be summarized as finding the saddle point (π, D) in the expression:

[0106]

[0107] in E is the γ-discounted causal entropy of strategy π, where λ is the weight of the entropy H(π); π[log(D(s, a))] represents the expected discounted return along the trajectory generated by strategy π. Indicates along the expert strategy π E The expected value of the resulting trajectory discounting revenue. Initially, Trust Region Policy Optimization (TRPO) was used to update the policy in GAIL. TRPO can prevent excessive policy oscillation caused by policy gradient noise. This invention uses the Proximal Policy Optimization (PPO) algorithm because PPO has the same advantages as TRPO, and implementing it using a first-order method is much simpler than implementing TRPO using a complex second-order method.

[0108] In this embodiment, the forward dynamic model part of the model-based generative adversarial interaction imitation learning robot learning method uses real samples of robot-environment interaction to learn a forward dynamic model, and then uses virtual interactions between the robot and the forward dynamic model to generate simulated samples for updating the discriminator and generator. This invention employs a Dynamoid model-based reinforcement learning method, using a feedforward neural network to approximate the forward dynamic model. To improve the accuracy of model predictions, this invention uses two different neural networks to encode states and actions, and then combines the encoded states and actions into a vector using the Hadamard product. Furthermore, this invention utilizes a gated recurrent unit (GRU) layer as part of the state encoder to achieve prediction of the next state using multiple previous states. This model structure yields better and more stable results than typical feedforward neural networks. In MAILDH, It can be used to predict the robot's current state and state transitions under its actions, denoted as f(s, a) = s′. It continuously updates the forward dynamic model by minimizing the loss using trajectory samples [s, a, s′] of the robot's interaction with the real environment. in For the estimated forward dynamic model, n is the number of samples. From The generated τ s , used for and τ E τ agent The generator and discriminator are updated iteratively together. Specifically, this is achieved by reducing the loss:

[0109]

[0110] To update the discriminator D, τ s The states and actions in GAIL also need to be derived from the cost function c in GAIL by the latest discriminator D. gail (s, a) and the trained human reward model H are evaluated and integrated into a new reward function c. maildh (s, a) are used to update the policy network.

[0111] In this embodiment, the human reward model portion of the model-based generative adversarial interaction imitation learning robot learning method, unlike traditional interactive reinforcement learning, does not require continuous feedback from a human trainer. This invention uses a model to replace the human trainer. At the beginning of the training process, to reduce the burden on the human trainer, all state-action pairs [s] in the demonstration are given... d a d Allocate an artificial reward r h =+1, and add all tuples [s d a d [+1] is stored in the replay buffer B to obtain enough samples with good actions for pre-training the human reward model. Furthermore, this invention also extracts T random samples from the task (the number of T is selected based on the complexity of the task; the more complex the task, the larger the number of T will be), i.e., selecting the current state s. t and a random output action a t [s] d a d The reward r given by the human trainer is used as input to the human reward model. h The labels are used to pre-train the human reward model. It's important to note that even in complex tasks, the number of samples (T) is far less than the number of samples required to achieve high accuracy in other methods. The updates to the human reward model and policy are asynchronous, and human trainers, who may not be experts in robot design or even programming, can provide rewards at any point during training by evaluating the robot's behavior, thus continuing to train the human reward model. This is achieved by minimizing the loss. The updated human reward model H, where L H Let H(s, a) represent the loss function, H(s, a) be the estimated human reward model, n be the number of samples, and r be the number of samples. h Artificial rewards are used as tags.

[0112] In this embodiment, the model-based generative adversarial interaction imitation learning robot learning method proposes a fixed cost function c from the discriminator network of GAIL. gail (s, a), and introduce human feedback to form a new cost function c. maildh (s, a) = c gail (s, a) + H(s, a). Where H(s, a) is used to approximate human feedback using a human reward model.

[0113] The advantages and positive effects of this invention are as follows:

[0114] This invention combines model-based reinforcement learning, human-evaluation-feedback-based interactive reinforcement learning, and the generative adversarial imitation learning (GAIL) framework to form a model-based generative adversarial interactive imitation learning (MAILDH). This invention enables robots to leverage the synergy of demonstration and human evaluation, both referencing expert demonstrations and utilizing human knowledge, thus overcoming the limitations of demonstration quality and achieving learning effects that surpass or even exceed those of demonstrations. This effectively acquires better strategies and improves strategy stability. The forward dynamic model in this embodiment improves sample efficiency and robustness. Furthermore, the introduction of the forward dynamic model and the human-reward-based MAILDH method broadens the application scope of traditional methods, improving sample efficiency, training speed, and stability.

[0115] The invention was tested on eight physics-based control tasks ranging from low-dimensional to high-dimensional and from discrete action space to continuous action space. The results show that the beneficial effect of the invention is that, given either optimal or suboptimal expert demonstrations, the MAILDH robot can always learn a better policy than GAIL with fewer real samples.

[0116] The technical effects of the present invention will be described in detail below with reference to experiments.

[0117] To demonstrate the potential effectiveness of the proposed method, experiments were conducted on eight physics-based control tasks ranging from low-dimensional to high-dimensional and from discrete action spaces to continuous action spaces.

[0118] (1) Experimental tasks, screenshots of the eight tasks are shown below. Figure 2 .

[0119] CartPole: A wooden pole is connected to a cart via a non-driven joint, and the cart moves along a frictionless track. The pole starts upright, and the goal is to keep the pole upright by pushing the cart to the left or right. This task has a four-dimensional continuous state space and a two-dimensional discrete action space, and a numerical reward of 1 is given for each time step.

[0120] MountainCar: A small car starts from the bottom of a valley and its goal is to climb to the top of the mountain. The task has a two-dimensional continuous state space and a three-dimensional discrete action space. Each step taken results in a negative reward.

[0121] Acrobot: A two-link pendulum with only the second joint driven. Initially, both links point downwards. The objective is to swing the end effector to the length of at least one link above the baseline. The task has a six-dimensional continuous state space and a two-dimensional discrete action space. The reward is -1 for each step and 0 for the end state.

[0122] Lunar Lander: This is a simulation environment for testing and solving rocket trajectory optimization—a classic optimal control problem. The mission has an eight-dimensional continuous state space and a four-dimensional discrete action space. The reward is given based on the relative motion between the lunar lander and the landing gear.

[0123] Mountain Car Continuous: A car starts from the bottom of a valley and its goal is to climb to the top of a mountain. This task has a two-dimensional continuous state space and a one-dimensional continuous action space. The reward is -0.1 * action for each step taken. 2 The negative reward is increased by 100 when the goal is achieved.

[0124] Lunar Lander Continuous: This is a simulation environment for testing and solving rocket trajectory optimization—a classic optimal control problem. The mission has an eight-dimensional continuous state space and a two-dimensional continuous action space. Rewards are given at each step based on the lander's relative motion to the landing pad.

[0125] Hopper PyBullet: The goal is to move Hopper forward as far as possible. This task has a 15-dimensional continuous state space and a 3-dimensional continuous action space. The reward is given for each step the object takes, based on its motion relative to the floor, control costs, etc.

[0126] HalfCheetah PyBullet: The goal is to move HalfCheetah forward as far as possible. This task has a 26-dimensional continuous state space and a 6-dimensional continuous action space. The reward is given for each step the object takes, based on its motion relative to the floor, control costs, etc.

[0127] (2) Experimental setup: For each different task, four different robots were tested, which could be divided into two groups: one group consisted of model-free robots, namely the GAIL robot which learned from the demonstration as a baseline and the GAILH robot which learned from the demonstration and human rewards through GAIL; the other group consisted of model-based robots, namely the MGAIL robot which learned from the demonstration only by embedding a forward dynamic model on top of GAIL and the MAILDH robot which learned from the demonstration and human rewards by embedding a forward dynamic model on top of GAIL. In each task, predefined reward functions in OpenAI's Gym environment were used as metrics to evaluate robot performance, but these reward functions were never used for learning.

[0128] Feedforward neural networks were used as approximators with different architectures: the policy network consisted of two hidden layers, each with 128 units, with tanh activations in between, and a softmax output layer; the value network and discriminator network in PPO each consisted of two 128-unit hidden layers and one output layer, with tanh activations in between. All networks were randomly initialized at the start of each trial. In each task, the environment simulator and model were randomly initialized using three random seeds, and learning curves for the four robots were plotted for comparison.

[0129] (3) Experimental results: The potential effectiveness of the proposed method will be demonstrated by comparing and analyzing the learning performance of the four robots in different tasks. Figures 3 to 11 The shaded area represents the 0.95 confidence interval, and the thick line represents the robot's average performance during the learning process. The performance demonstrated by experts is also shown in the figure for reference.

[0130] 1) The impact of introducing the forward dynamic model

[0131] This invention first analyzes the impact of the forward dynamic model on robot learning by comparing the learning curves of the GAIL robot and the MGAIL robot, such as... Figures 3-11 As shown. In four tasks with discrete action spaces, the MGAIL robot is able to reach or approach the demonstration faster than the GAIL robot. When CartPole provides the optimal demonstration, both the MGAIL and GAIL robots can reach the demonstration, but it is clear that the MGAIL robot is faster than the GAIL robot, as shown. Figure 3 As shown. When a suboptimal demo is provided, in the CartPole task, the performance of the MGAIL robot fluctuates around the demo level, but is more stable than that of the GAIL robot. Figure 4 As shown; in the MountainCar task, the MGAIL robot reached the demonstration significantly faster than the GAIL robot, as... Figure 5 As shown; in the Acrobot task, the MGAIL robot and the GAIL robot performed similarly, as... Figure 6 As shown; in the LunarLander mission, the MGAIL robot initially approached the demonstration faster than the GAIL robot, but its performance was slightly weaker than the GAIL robot in the later stages, such as... Figure 7 As shown. In the four tasks with continuous action spaces, similar to the performance in discrete action spaces, in MountainCar Continuous and Hopper PyBullet, the MGAIL robot is able to approach the demonstration faster than the GAIL robot, such as... Figure 8 and 10As shown. In Lunar Lander Continuous and HalfCheetah PyBullet, the MGAIL robot also showed a trend of approaching the demonstration faster and better than the GAIL robot, as... Figure 9 and 11 As shown. Furthermore, to highlight the impact of incorporating the forward dynamic model on sample efficiency, the number of samples required for the robot to achieve the same performance across eight tasks was statistically analyzed, such as... Figure 13 As shown, when achieving the same performance across eight tasks, the MGAIL and MAILDH robots require significantly fewer samples than GAIL, indicating that the introduction of the forward dynamic model plays a crucial role in improving sample efficiency. However, this invention finds that the performance of the MGAIL and GAIL robots is still at the upper limit of the demonstration.

[0132] 2) The impact of combining forward dynamics models with human rewards

[0133] To analyze the contributions of the forward dynamic model and human reward to the performance of the MAILDH robot during the learning process, this invention compares the performance of the MAILDH robot with that of the MGAIL robot (which learns an additional dynamic model on top of GAIL), the GAILH robot (which learns an additional human reward function on top of GAIL), and the GAIL robot, respectively. Figures 3-11 As shown. In the discrete action space, by further integrating the human reward and forward dynamics model with GAIL, the MAILDH robot proposed in this invention learns better and faster than the MGAIL robot. When the optimal demonstration is provided in CartPole, the GAILH robot learns slightly slower, but its final performance is as good as the MGAIL robot, as shown. Figure 3 As shown; when providing a suboptimal demonstration, the GAILH robot is far superior to the MGAIL robot, such as... Figure 4 As shown. When providing suboptimal demonstrations in other tasks, the GAILH robot performs worse in the Mountain Car compared to the MGAIL robot, such as... Figure 5 As shown; its performance in LunarLander is almost equivalent, such as Figure 7 As shown; its performance in Acrobot is far superior to the MGAIL robot, and even surpasses the demonstration, such as... Figure 6 As shown, tasks in continuous action space and tasks in discrete action space yield similar results, meaning the MAILDH robot learns better and faster than the MGAIL robot. However, when providing a suboptimal demonstration, the GAILMH robot performs worse than the MGAIL robot in Hopper PyBullet, as shown below. Figure 10As shown; its performance in Lunar LanderContinuous is almost equivalent, such as Figure 9 As shown; it performs much better in Mountain Car Continuous and HalfCheetahPyBullet, such as Figure 8 and Figure 11 As shown.

[0134] from Figures 3-11 The results show that by combining GAIL with a forward dynamic model and a human reward function, respectively, the MGAIL and GAILH robots learn much faster and achieve better learning results than the original GAIL. Combining the forward dynamic model and human reward function with GAIL as the proposed MAILDH method allows the robot to learn even faster and better than the MGAIL and GAILH robots. The MAILDH robot achieves optimal performance when providing the best demonstration and outperforms the second-best demonstration in most tasks when providing the second-best demonstration.

[0135] In summary, combining the forward dynamic model with GAIL alone results in the MGAIL robot learning faster and performing better than the GAIL robot, but its performance is still limited by the demonstration. The forward dynamic model shows significant improvement when a suboptimal demonstration is provided. Combining the human reward function alone with GAIL also allows the GAILH robot to learn faster and better than the GAIL robot, and in some tasks, its performance can even surpass the demonstration. Combining the forward dynamic model and the human reward function with GAIL as the proposed MAILDH method allows the robot to learn faster and better than both the MGAIL and GAILH robots. The MAILDH robot achieves optimal performance when providing the best demonstration and outperforms the suboptimal demonstration in most tasks when providing a suboptimal demonstration.

[0136] 3) Robustness to dynamic model errors

[0137] In the above experiments, both the MGAIL and MAILDH robots were trained using randomly initialized dynamic models. However, in real-world scenarios, dynamic models of varying quality may be provided in advance and further improved during model-based reinforcement learning. Therefore, this invention aims to investigate the impact of model error on the learning of the MAILDH robot by training MGAIL and MAILDH robots initialized with dynamic models of different qualities.

[0138] Specifically, this invention trains the MAILDH robot with a randomly initialized dynamic model for 2000 rounds, saving the trained dynamic model every 500 rounds to obtain pre-trained dynamic models of varying quality. The quality of the dynamic model is determined by... To measure, where k is the trajectory length, s i For the true state, s i ′ represents the predicted state.

[0139] Figure 12 The number of interactions required for the MGAIL and MAILDH robots to achieve optimal performance in the Lunar Lander task is shown, and compared with the GAIL and GAILH robots. Figure 12 The results show that adding a dynamic model significantly reduces the number of interactions required to achieve optimal performance, while adding only a human reward function provides only a small improvement in sample efficiency. Furthermore, as the model error increases from 0.8 to 3.7, the number of samples required for MGAIL increases slowly from around 50,000, while for MAILDH it fluctuates below 50,000. However, as the model error further increases from 3.7 to 5 (random model), the number of interactions required for the MGAIL robot increases sharply to around 150,000, while the number of interactions required for the MAILDH robot increases slightly but remains below 100,000. This demonstrates that compared to adding a dynamic model alone, combining a dynamic model and a human reward function with GAIL results in a significantly improved sample efficiency for the MAILDH robot, and the robot is more robust to model errors. This suggests a possible synergistic effect between the dynamic model and human reward in the method of this invention.

[0140] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A model-based generative adversarial interaction imitation learning method for robot learning, characterized in that, This robot learning method is applied in the field of social robot task learning, and includes the following steps: Expert trajectories are sampled from expert policies, and the network parameters of robot policy, discriminator, human reward model and forward dynamic model are initialized. The robot pre-trained a human reward model using samples from the demonstration, as well as random samples drawn from the task. The robot samples the generated trajectory from the current strategy and the simulated trajectory from the forward dynamic model; it determines whether it has received human feedback. If so, it updates the human reward model; otherwise, it updates the discriminator network and the forward dynamic model. The robot uses the loss function extracted from the discriminator network and the human feedback output by the human reward model to perform interactive reinforcement learning. It determines whether a stable policy has been learned. If so, the process ends; otherwise, the robot policy is updated. The model-based generative adversarial interaction imitation learning method further includes a model-based reinforcement learning, generative adversarial imitation learning, and interactive reinforcement learning framework forming a model-based generative adversarial interaction imitation learning MAILDH. During training, MAILDH learns a forward dynamic model to generate simulated data and trains the generator and discriminator with data from real-world interactions with the environment. It is also used to train a human reward model so that the robot can learn by imitation from demonstrations or from human evaluation feedback.

2. The robot learning method based on model-driven generative adversarial interaction imitation learning according to claim 1, characterized in that, The forward dynamic model employs a Dyna-type reinforcement learning method, using a feedforward neural network to approximate the forward dynamic model. During each new iteration in the training process, the robot collects samples obtained from the real interactions between the current policy and the environment, and uses these samples to update the forward dynamic model.

3. The model-based generative adversarial interaction imitation learning robot learning method according to claim 2, characterized in that, The human reward model is also used to assign rewards to all state-action pairs in the demonstration before the training process begins. Allocate an artificial reward Extracted from the task A random sample; select the current state. and a random output action And manually allocate rewards. ; , The rewards given by human trainers are used as input to the human reward model. Used as labels to pre-train human reward models; Corresponding to the state in the demonstration, Corresponding to the actions in the demonstration, The state at the current time. The action selected for the current time. The reward is artificial.

4. The model-based generative adversarial interaction imitation learning robot learning method according to claim 3, characterized in that, After training begins, the pre-trained human reward model is loaded, and all tuples are included. , Stored in the playback buffer In order to obtain samples for further training of the human reward model.

5. The model-based generative adversarial interaction imitation learning robot learning method according to claim 4, characterized in that, The simulated trajectory obtained by MAILDH using a forward dynamic model Update the generator and discriminator using the demo track. The actual trajectory of robot interaction with the environment under the current strategy. and simulated trajectory To update the discriminator Reduce losses ； in This represents the loss function of the discriminator. Indicates the trajectory sample Experiential expectations, i.e. , It is a discount factor, for , To be honest ,for , ; By the latest discriminator The derived cost function in GAIL and the trained human reward model It will be integrated into a new reward function: in This represents the cost function of MAILDH. This represents the cost function of GAIL. This represents the predicted reward from the human reward model. , These represent the input state and action, respectively. The reward function is used to update the strategy with PPO. Repeat the adversarial training process described above until the desired strategy is obtained.

6. The robot learning method based on model-driven generative adversarial interaction imitation learning according to claim 5, characterized in that, The MAILDH method finds saddle points in an expression. : in Indicates along the strategy The expected value of the trajectory discounted return. This represents the expected discounted return of the demonstration trajectory.

7. The robot learning method based on model-driven generative adversarial interaction imitation learning according to claim 1, characterized in that, Before training begins, MAILDH also includes pre-training a human reward model, where the samples used to update the human reward model come partly from state-action pairs in expert demonstrations and partly from randomly sampled state-action pairs.

8. The model-based generative adversarial interaction imitation learning robot learning method according to claim 1, characterized in that, The model-based generative adversarial interaction imitation learning method further includes a generative adversarial imitation learning part, a forward dynamic model part, and a human reward model part; The generative adversarial imitation learning part involves finding the saddle point of the expression. : ， In the formula, It is a strategy of -Discounted causal entropy, It is entropy The weights; Indicates along the strategy The expected value of the trajectory discounted return. Indicates following expert strategy The expected value of the trajectory discount return generated; The forward dynamic model part learns a forward dynamic model using real samples of robot-environment interaction, and generates simulated samples for updating the discriminator and generator by using virtual interaction between the robot and the forward dynamic model.

9. The model-based generative adversarial interaction imitation learning robot learning method according to claim 8, characterized in that, In the human reward model part, by minimizing the loss Updated human reward model ,in It represents the loss function. It is an estimated human reward model. It is the sample size. Artificial rewards are used as tags.

Images