Method and apparatus for manipulating a robot
By optimizing the training method of the deep neural network model, combining gradient descent and policy gradient, and using expert trajectories and model trajectories to optimize the discriminator and model parameters, the problem of low training efficiency in the existing technology is solved, and efficient imitation of robot behavior and environmental adaptation are achieved.
Patent Information
- Application Number
- CN202080022191.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-18
- Filing Date
- 2020-03-05
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2040-03-05
AI Technical Summary
Existing generative adversarial imitation learning and model-based policy learning methods are inefficient during training and have difficulty effectively matching real system behaviors.
By training a deep neural network model, using gradient descent and policy gradient methods, combining expert trajectories and model trajectories, optimizing the discriminator and model parameters, and adopting episode-based policy search and reinforcement learning, efficient reward determination and adaptive adjustment of the model are achieved.
The efficiency and accuracy of model training have been improved, enabling robots to better imitate human behavior, adapt to environmental changes, and enhance control and regulation effects.
Smart Images
Figure CN113614743B_ABST
Abstract
Description
BACKGROUND
[0001] J. Ho and S. Ermon, "Generative adversarial imitation learning," in Advances in Neural Information Processing Systems 29, pages 4565-4573. Curran Associates, Inc., 2016, disclose a scheme of generative adversarial imitation learning, GAIL. GAIL is a method for training a policy that should imitate an expert behavior.
[0002] A. Doerr, C. Daniel, D. Nguyen-Tuong, A. Marco, S. Schaal, T. Marc, and S. Trimpe, "Optimizing long-term predictions for model-based policy search," in Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 227-238. PMLR, 13-15 Nov 2017, disclose a scheme for learning long-term predictions for model-based policy search.
[0003] It is desirable to further improve the schemes of these methods of operation. SUMMARY
[0004] This is achieved by the method according to the invention and by the device according to the invention.
[0005] A computer-implemented method for training a parametric model, in particular a deep neural network, of an environment, determines a new model state from a model state, an action, and a reward, wherein the reward is determined from an expert trajectory and in particular from a model trajectory determined from a model state according to a policy, and wherein at least one parameter of the model is determined from the reward. Thereby, a long-time behavior of the model is learned that matches particularly well to a real system behavior of a system that is modeled.
[0006] It is preferably provided that the discriminator determines a reward from the expert trajectory and the model trajectory, wherein at least one parameter of the discriminator is determined from the expert trajectory and the model trajectory using a gradient descent method. In a roll out, the expert trajectory is used as a reference, and the generator, i.e. the pre-given device, determines a model trajectory from the model state according to a certain policy for comparison with the reference. The discriminator is parameterized by the gradient descent method. This makes it possible to parameterize the discriminator independently of the training of the model in a first step.
[0007] It is preferably provided that at least one parameter of the model is learned from the reward using an episode-based policy search or a policy gradient method, in particular REINFORCE or TRPO. This makes it possible to train the model independently of the training of the discriminator in a second step. It is preferably provided that the discriminator is trained first, and the model is trained using the reward then determined by the discriminator. These steps are preferably repeated alternately.
[0008] It is preferably provided that the reward is determined from a true expected value for the system dynamics of the environment and from a modeled expected value of the model. The expected value represents an approximation of the actual system dynamics based on training data points, which makes it possible to calculate said training more efficiently.
[0009] It is preferably provided that the expert trajectory is determined in particular from a demonstration, wherein an expert action is detected in an environment state, in particular according to an expert policy pre-given, wherein the environment is transformed into a new environment state with a probability by the expert action, and wherein the environment state, the expert action and the new environment state are determined as data points of the expert trajectory. Supervised learning can thus be represented particularly efficiently.
[0010] It is preferably provided that an action is detected in a model state, which is pre-given according to a policy, wherein the model is transformed into a new model state with a probability by the action, wherein a reward is determined from the model state, the action and the new model state. This makes it possible to train the discriminator using the model trajectory determined by the generator from the model. The model changes in the training. The model trajectory of the generator thus changes even if the policy used by the generator does not change. This makes it possible to adapt the discriminator to the changed model. The training thus becomes more efficient overall.
[0011] A parameter model of an environment including the regulated object is preferably trained according to the regulated object, wherein at least one state variable or a regulation variable is determined for manipulating the regulated object according to the model and according to at least one detected actual variable or an observed state variable of the regulated object. According to the regulated object, the model can be used in an untrained or partially trained manner. Thereby, a significant improvement of conventional methods for control or regulation is achieved, in particular in view of the learning aspect of the model.
[0012] Preferably, an action is determined according to a policy from a model state of the model by means of an agent, wherein a reward is determined according to the policy, the action or a new model state, wherein the policy is learned according to the reward in a reinforcement learning method. Thus, the policy can also be learned efficiently. The goal of the agent is to maximize the reward in the reinforcement learning method. The reward is determined, for example, according to a measure of the conformity of the policy to a predefined reference policy, a measure of the conformity of the model behavior to the actual behavior of the environment or to a reference model behavior.
[0013] A computer-implemented method for manipulating a robot provides that a parameter model of an environment is trained according to the method stated above, a policy is learned and the robot is manipulated according to the parameter model and the policy. This means that the robot is manipulated in such a way that the robot imitates human behavior. For this purpose, the policy is learned. For learning the policy, the model, i.e. the environment model, which is also learned, is used.
[0014] An apparatus for manipulating a robot is configured to carry out the computer-implemented method for manipulating a robot. The apparatus is configured to learn an environment model and a policy with which human behavior can be imitated. BRIEF DESCRIPTION OF DRAWINGS
[0015] Further advantageous refinements result from the following description and the drawings. In the drawings:
[0016] Figure 1 A schematic diagram showing a part of the apparatus for manipulating a robot,
[0017] Figure 2 A schematic diagram showing another part of the apparatus,
[0018] Figure 3 A schematic diagram showing a part of the regulation system,
[0019] Figure 4 A schematic diagram showing a part of the learning system,
[0020] Figure 5 A schematic diagram showing a step in the method for training a model,
[0021] Figure 6 A step in the regulation method,
[0022] Figure 7 Steps in a learning method for a policy are shown. DETAILED DESCRIPTION
[0023] Figure 1 A part of an apparatus for manipulating a robot is shown. The apparatus for manipulating a robot is configured for training a parametric model of an environment according to a method described below. The apparatus is configured for learning a policy as described below. In an aspect, the apparatus is furthermore configured for training a model according to a conditioning object as described below. The apparatus is furthermore configured for manipulating a robot according to the parametric model and the policy. The apparatus is configured for learning an environment model and a policy, wherein human behavior can be imitated with said policy.
[0024] The apparatus for manipulating a robot comprises at least one memory for instructions and at least one processor for executing said instructions. Execution of said instructions results in determining an action for a robot according to a policy for manipulating a robot. The apparatus comprises for example at least one manipulation device for manipulating a robot according to said action.
[0025] The robot can be an at least partially autonomous vehicle that performs said action.
[0026] The apparatus is configured for detecting an expert trajectory τ E for an environment 102. E The expert trajectory τ U comprises a plurality of triples (s E , a U , s′ E ) that collectively give rise to the expert trajectory τ E0 = (s E0 , a E1 , s E1 , a ET ,..., s ). The apparatus comprises a first pre-given device 104 configured for determining an expert action a E for an environment state s E of the environment 102 according to an expert policy π U (a U | s E ). The environment 102 is transformed by said expert action a E into a new environment state s′ U with a probability p(s′ E | a U , s U ). The pre-given device 104 can have a human-machine interface configured for inputting a state s UOutput to the expert and detect the expert action a based on the expert input E .
[0027] In this example, the apparatus is configured to determine an expert trajectory τ from a demonstration E During the demonstration, the following expert actions are detected: E , where the expert is in the environment state s U The expert action is predefined in U 、Expert Actiona E and the new environment state s′ U Determined to be the expert trajectory τ E For example, the device is configured to repeat these steps in order to detect the expert trajectory τ E In this example, the expert trajectory τ is set E The first memory 106.
[0028] Figure 2 Another part of the device is shown schematically. The device is configured to train a parameter model 202 of the environment 102. The model 202 is, for example, a deep neural network. The device includes a second presetting device 204 for the action a. The model 202 is configured to train a parameter model 202 of the environment 102 according to the model state s. M , action a and reward r to determine the new model state s M In this example, the second presetting device 204 is a generator, which is configured to generate a signal according to the model state s M According to the strategy π(a|s M ) to determine the action a. The presetting device 204 is designed to be used as a plurality of (s M ,a,s′ M ) triples, where these triplets collectively yield a model trajectory τ = (s M0 , a M0 , s M1 , a M1 ,...,s MT ). For example, the device is configured to repeat these steps to detect data points for the model trajectory τ until the training of the model is completed. In this example, a second memory 206 for the model trajectory τ is provided.
[0029] The device comprises a discriminator 208 which is designed to discriminate based on the expert trajectory τ E and the model trajectory τ determine the reward r.
[0030] The apparatus comprises a training device 210 configured to train a model using a gradient descent method according to an expert trajectory τE and the model trajectory τ determine at least one parameter w of the discriminator 208 .
[0031] Training device 210 is designed to learn at least one parameter θ of model 202 as a function of reward r using an episode-based policy search or using a policy gradient method, in particular REINFORCE or TRPO.
[0032] The training device 210 is configured to train the system dynamics of the environment 102 according to the actual expected value. And according to the expected value modeled for model 202 To determine the reward r. The training device 210 is configured, for example, to detect the reward r in the model state s. M The model 202 calculates the action a with probability p by taking the action a. θ (s′ M |a,s M ) is transformed into a new model state s′ M , and where according to the model state s M , action a and new model state s′ M Determine the reward r. The training device 210 is configured to determine at least one parameter θ of the model 202 as a function of the reward r.
[0033] In one aspect, a device for controlling or regulating in a model-based manner is also provided. Figure 3 1 shows a device for control, which includes a parameter model 202 of an environment 102, wherein the environment 102 includes a control object 302. The device is designed to train the model 202 based on the control object 302. The device is designed to train the model 202 based on the model 202 and based on an observed state variable or at least one detected actual variable s of the control object 302. R At least one state variable or manipulated variable y is determined for controlling the manipulated object 302. The control is then processed accordingly independently of the actual variable.
[0034] exist Figure 4 Schematically, an apparatus for learning a policy in a model-based manner is shown in FIG. The apparatus comprises a parameter model 202 of an environment 102. The apparatus comprises an agent 402 configured to learn a policy according to a model state s of the model 202 in accordance with a policy π. M To determine action a. The reinforcement learning device 404 is configured to determine action a according to the policy π, action a or the new model state s′. M Determine the reward r π , and in the reinforcement learning method according to the reward r π Let’s learn the policy π.
[0035] Figure 5 Steps of a computer-implemented method for training a parametric model 202 of an environment 102 are shown.
[0036] In step 502, an expert trajectory τ is determined, in particular from a demonstration, in particular according to the steps 501. E .
[0037] In this example, an expert action a is detected in an environment state s U of the expert. E For example, the environment state s U , the expert action a E and the new environment state s' U are determined as data points of the expert trajectory τ E . For example, this step is repeated until the demonstration ends.
[0038] Subsequently, step 504 is performed. In step 504, the discriminator 208 and the model 202 are initialized, for example with random values for the respective parameters.
[0039] Subsequently, step 506 is performed. In step 506, a new model trajectory is generated using the model parameters of the model 202. For example, a new model state s M is determined for the model trajectory τ from the respective model state s M and the respective action a for the model 202.
[0040] Subsequently, step 508 is performed. In step 508, a reward r is determined from the expert trajectory τ E and the model trajectory τ. In this example, the reward r is determined for the generated model trajectory τ. For this purpose, the discriminator 208 is used.
[0041] Subsequently, step 510 is performed. In step 510, the model 202 is trained by means of the reward r or using the accumulated reward R. Here, at least one parameter θ of the model 202 is learned, for example using episodic-based policy search or using a policy gradient method, in particular REINFORCE or TRPO.
[0042] Subsequently, step 512 is performed. In step 512, the discriminator 208 is trained by means of the expert trajectory and the model trajectory. In this example, at least one parameter w of the discriminator 208 is determined from the expert trajectory τ E and the model trajectory τ using a gradient descent method.
[0043] These steps are repeated in this example. The accumulated reward R is determined from the rewards r that have been determined for different model trajectories.
[0044] Based on the Markov decision process M = (S, A, p, r π, μ0, γ) determines the initially unknown reward r, which assumes that an expert trajectory has been created in the decision process M case. For example, the reward is a binary value, i.e. true or false. The purpose of the subsequent optimization is to learn a function p θ ≈ p, where p(s U , a E , s′ U ) describes the actual system behavior of the environment.
[0045] To this end, the following Markov decision process with unknown reward r is used:
[0046]
[0047] where
[0048] action space
[0049] state space
[0050] initial distribution
[0051] dynamic transition probability
[0052]
[0053] If the GAIL method is used in this decision process, the reward r can be determined and the model p θ can be learned.
[0054] For example, the reward r is determined from the true expected value for the system dynamics of the environment 102 and the modeled expected value of the model 202 .
[0055] For example, for the gradient descent method, a plurality of roll outs with an arbitrary policy π are set, where
[0056] where D w is a discriminator with at least one parameter w
[0057] For example, the log(D) is used as the reward r in order to exploit
[0058]
[0059] The model p θ is trained, where
[0060]
[0061] where
[0062] is the expected value of the i-th model trajectory based on model state s0, action a0, and new model state s'0.
[0063] An exemplary algorithm based on scenario-based policy search is illustrated below as pseudocode. The algorithm is based on the expert trajectory τ E determined by the expert policy π E ,
[0064] for i = 1 to... do
[0065] sample model parameters from model parameter proposal distribution
[0066] for each model parameter do
[0067] sample K trajectories from model
[0068] evaluate model parameters where
[0069]
[0070] end for
[0071] update discriminator by maximizing
[0072]
[0073] update model parameter proposal distribution from ω i to ω i+1
[0074] using maximum likelihood weighted with weights R [j] to update model parameters from ω i to ω i+1
[0075] end for.
[0076] The following definitions of R, r, H, γ, and λ are used in the algorithm:
[0077] R total cumulative reward of a model trajectory consisting of multiple (s, a, s') triples,
[0078] r reward of a single (s, a, s') triple, which is determined via the discriminator,
[0079] H causal entropy of the model q,
[0080] γ discount factor of an infinite-time discounted MDP
[0081] λ regularization factor, which can be chosen freely
[0082] In Figure 6 the steps in a method for regulating in a model-based manner are shown. A parametric model 202 of an environment 102 of a regulating object 302 is trained in step 602 according to the regulating object 302 as described. Subsequently in step 604, at least one state variable or regulating variable y for manipulating the regulating object 302 is determined from the model 202 and from at least one detected actual variable s R or observed state variable of the regulating object 302. These steps can be repeated recursively, for example, after initializing the model 202 for a self-learning regulator. It can also be provided that the model is trained first and only the steps 604 for regulating are repeated.
[0083] In Figure 7 the steps in a method for learning a strategy in a model-based manner are shown. A parametric model 202 of an environment 102 is trained in step 702 as described.
[0084] In a subsequent step 704, an action a is determined from a model state s M of the model 202, in particular by an agent 402 according to a strategy p.
[0085] Subsequently, in step 706, a reward r M is determined from the strategy p, the action a or a new model state s' π . Subsequently in step 708, the strategy p is determined in a reinforcement learning method from the reward r π .
[0086] The reward r π for example depends on the task that should be learned by the strategy p in the end. The reward can be defined as a function r(s, a) or r(s). The action a contributes to fulfilling the task in the state s to what extent or how well it is used for fulfilling the task in a particular state is rewarded by this.
[0087] If for example the task is to navigate to a destination in a 2D environment modeled by the model 202. Then the reward can for example be the distance to the destination. A strategy for navigating can accordingly be learned for a vehicle.
[0088] The steps 702 to 708 are preferably repeated recursively. It can be provided that steps 704 and 706 are repeated recursively before step 708 is performed.
[0089] In the following a computer-implemented method for manipulating a robot is described. The method provides that a parametric model 202 of an environment 102 is trained according to the method described according to Figure 5 . The method furthermore provides that a strategy p is learned according to the method described according to Figure 7The described method learns a policy p and manipulates the robot according to the parametric model 202 and the policy p. This means: The robot is manipulated such that it imitates human behavior.
[0090] In one approach, the model 202 is trained according to Figure 6 The described method depends on the conditioning object on which it is trained.
Claims
1. A computer-implemented method for training a parametric model (202) of an environment (102) for navigation of a robot, characterized in that The model (202) is based on the model state (s M ), action (a) and at least one parameter (θ) of the model (202) to determine (506) a new model state (s M '), wherein the parameters (θ) of the model (202) are determined according to the reward (r), wherein the parameters (θ) of the model (202) are determined according to the expert trajectory (τ E ) and the model trajectory (τ) to determine (508) the reward (r), and wherein the expert trajectory (τ) is determined (502) based on the demonstration E ), where the detection expert is in the environment state (s U ) in accordance with the expert strategy (π E (a E |s U ))The expert action (a E ), wherein the environment (102) is controlled by the expert action (a E ) with probability (p(s′) U |a E , s U )) is transformed into a new environmental state (s' U ), and wherein the environmental state (s U )), said expert action (a E ) and the new environmental state (S' U ) is determined (502) as the expert trajectory (τ E ) data points, where the expert trajectory τ E Including multiple triples (s U , a E , s′ U ), these triplets jointly derive the expert trajectory τ E =(s E0 , a E0 , s E1 , a E1 ,...,s ET ), wherein the robot is an at least partially autonomous vehicle that performs the action, and wherein the reward is the distance to the destination, and wherein a policy for navigation can be learned accordingly for the vehicle.
2. The method according to claim 1, characterized in that The discriminator (208) is based on the expert trajectory (τ E ) and the model trajectory (τ) to determine (508) the reward (r), wherein the reward (r) is determined by the gradient descent method according to the expert trajectory (τ E ) and the model trajectory (τ) to determine (512) at least one parameter (w) of the discriminator (208).
3. The method according to claim 1 or 2, characterized in that The at least one parameter (θ) of the model (202) is learned (510) based on the reward (r) using an episode-based policy search or a policy gradient method.
4. The method according to claim 1 or 2, characterized in that According to the true expected value of the system dynamics of the environment (102) And according to the expected value modeled by the model (202) The reward (r) is determined (512).
5. The method according to claim 1 or 2, characterized in that In the model state (s M ) detects (506) an action (a) predetermined according to a policy (π), wherein the model (202 ...) θ (s′ M |a,s M )) is transformed to the new model state (s' M ), where according to the model state (s M ), the action (a) and the new model state (s' M ) determines the reward (r).
6. The method according to claim 1 or 2, characterized in that A parameter model (202) of an environment (102) including a control object (302) is trained (602) based on the control object (302), wherein the control object (302) is trained based on the model (202) and based on at least one detected actual variable (s) of the control object (302). R ) or the observed state variable determines (604) at least one state variable or control variable (y) for controlling the control object (302).
7. The method according to claim 1 or 2, characterized in that According to the model state (s) of the model (202) M ) determines (704) an action (a) according to a policy (π), wherein the action (a) or the new model state (s') is determined according to the policy (π), the action (a) or the new model state (s') M ) Determine (706) reward (r π ), wherein in the reinforcement learning method according to the reward (r π ) learn (708) the policy (π).
8. The method according to claim 1, characterized in that The parametric model is a deep neural network.
9. The method according to claim 1, characterized in that According to the expert trajectory (τ E ) and according to the strategy (π(a|s M ))According to the model state (s M ) is used to determine (508) the reward (r) based on the model trajectory (τ) determined by .
10. The method according to claim 3, characterized in that The at least one parameter (θ) of the model (202) is learned (510) based on the reward (r) using REINFORCE or TRPO.
11. The method according to claim 7, characterized in that By means of an agent (402) according to the model state (s) of the model (202) M ) determines (704) an action (a) according to a policy (π), wherein the action (a) or the new model state (s') is determined according to the policy (π), the action (a) or the new model state (s') M ) Determine (706) reward (r π ), wherein in the reinforcement learning method according to the reward (r π ) learn (708) the policy (π).
12. A computer-implemented method for controlling a robot, characterized in that: A parametric model (202) of an environment (102) is trained according to the method of claim 7 or 11, and a policy (π) for controlling the robot is learned, and the robot is controlled according to the parametric model (202) and the policy (π).
13. The method according to claim 12, characterized in that The parametric model is a deep neural network.
14. A computer program product, characterized in that The computer program product comprises computer-readable instructions which, when executed by a computer, execute the method according to any one of claims 1 to 13 .
15. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises a memory on which the computer program product of claim 14 is stored.
16. A device for controlling a robot, characterized in that: The device is designed to carry out the method according to claim 12 or 13 .