Methods for controlling robotic devices and robot device controllers

Abstract

A method for controlling a robotic device is described according to various embodiments, comprising obtaining a demonstration for controlling the robotic device; performing initial training of an actor neural network through imitation learning of the demonstration; controlling the robotic device through the initially trained actor neural network to generate multiple trajectories of the robotic device, wherein each trajectory includes a sequence of actions selected by the initial actor neural network in a state sequence, and observing a reward for each action selected by the initial actor neural network; performing initial training of a critic neural network through supervised learning, wherein the critic neural network is trained to determine the observed reward for the actions selected by the initial actor neural network; training the actor neural network and the critic neural network through reinforcement learning starting from the initially trained actor neural network and the initially trained critic neural network; and controlling the robotic device through the trained actor neural network and the trained critic neural network.

Description

Technical Field

[0001] This disclosure relates to apparatus and methods for controlling robotic devices, as well as robotic device controllers. Background Technology

[0002] Controllers for robotic devices can be trained, for example, in manufacturing using reinforcement learning to perform specific tasks. Performing a task typically involves selecting an action for each state in a sequence of states, which can be viewed as a sequential decision problem. Each action, depending on the state reached by the selected action, particularly the final state, provides a reward, which is caused, for example, by whether the action allows reaching the rewarding final state (e.g., to achieve the task's objective). Reinforcement learning typically involves a trade-off between exploration (trying new actions) and development (using actions known to give good rewards), and the training time and data required for successful training typically depend particularly on how much exploration must be conducted during training. Therefore, methods for controlling robots are desirable, requiring little exploration and thus being data-efficient and utilizing minimal training time.

[0003] The publication “Continuous control with deep reinforcement learning” by Lillicrap, Timothy P. et al., arXiv preprint arXiv:1509.02971 (2015), referred to below as reference [1], describes DDPG (Deep Deterministic Policy Gradient), a model-free actor-critic RL method aimed at solving decision problems without any prior knowledge of the environment (i.e., the problem to be solved). It uses a replay buffer and a target network for both the actor and critic networks. According to DDPG, the state space and new policies are discovered by exploration.

[0004] The publication “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards” by Vecerik, Mel, et al., arXiv preprint arXiv:1707.08817 (2017), referred to below as reference [2], describes DDPGfD, a model-free actor-critic RL method that aims to enable agents to learn policies from expert demonstrations, thereby safely interacting with the environment. DDPGfD modifies the DDPG algorithm to take advantage of demonstrations. It loads demonstration transitions into a replay buffer before the learning process begins and retains all transitions during training. DDPGfD uses a PER (prioritized experience replay) memory to enable efficient propagation of rewards. To propagate Q-values ​​along a trajectory, DDPGfD incorporates an n-step action-value objective. Summary of the Invention

[0005] According to various embodiments, a method for controlling a robotic device is provided, comprising obtaining a demonstration for controlling the robotic device, performing initial training of an actor neural network by imitation learning of the demonstration, controlling the robotic device by the initially trained actor neural network to generate a plurality of trajectories of the robotic device, wherein each trajectory includes a sequence of actions selected by the initial actor neural network in a state sequence, and observing a reward for each of the actions selected by the initial actor neural network, performing initial training of a critic neural network by supervised learning, wherein the critic neural network is trained to determine the observed reward for the actions selected by the initial actor neural network, training the actor neural network and the critic neural network by reinforcement learning starting from the initially trained actor neural network and the initially trained critic neural network, and controlling the robotic device by the trained actor neural network and the trained critic neural network.

[0006] According to another embodiment, a robot device controller is provided according to the method for controlling a robot device described above.

[0007] The aforementioned method for controlling robotic devices and the robotic device controller allow for training sequential decision problems (for controlling robotic devices) with efficient data volumes and minimal training time, as it requires only a small number of data samples and interactions with the environment due to improvements derived from prior knowledge from demonstrations (e.g., by experts, such as inadequately designed controllers). Furthermore, it allows for addressing the three main caveats of model-free actor-critic reinforcement learning settings: high sample complexity, overestimation of the action-value function, and self-destructive interaction loops. This method allows for the efficient initial learning of the behavior of any previously inadequately designed controller to achieve the objective, and then further improvement of the controller's behavior through reinforcement learning.

[0008] Various examples are given below.

[0009] Example 1 is a method for controlling robotic equipment as described above.

[0010] Example 2 is a method of Example 1, wherein training an actor neural network and a critic neural network by reinforcement learning includes executing multiple episodes, wherein in each episode, the actor neural network selects a control action for each of multiple time steps, wherein the method includes copying the actor neural network to generate a target actor neural network and copying the critic neural network to generate a target critic neural network under delays of multiple time steps, and the method includes using the target actor neural network to provide one or more control actions following the control action selected by the actor neural network to the target critic neural network for each control action selected by the actor neural network, wherein the target actor neural network determines the reward of the one or more control actions selected by the target actor neural network, wherein the critic's loss is determined based on the reward of the action selected by the actor neural network determined by the critic, the sum of the rewards of the one or more control actions selected by the target actor neural network, and the reward of the action selected by the actor neural network.

[0011] Since the actor neural network and the critic neural network are pre-trained (initial training) using prior knowledge, it is sufficient to copy the actor neural network to the target actor neural network and the critic neural network to the target critic neural network. Therefore, training is more efficient.

[0012] Example 3 is the method of Example 2, wherein training the actor neural network involves updating the actor neural network every predetermined number of time steps to select actions with higher rewards, as specified by the critic neural network.

[0013] Similar to the delayed updates of the target actor neural network and the target critic neural network described above, the updates of the actor neural network can be delayed (i.e., not performed at every time step, but only after a certain number of time steps). This reduces training time.

[0014] Example 4 is a method of any of Examples 1 to 3, wherein obtaining a demonstration includes obtaining a demonstration from a human user and / or a pre-designed robot device controller.

[0015] Therefore, prior knowledge, such as that provided by human users or pre-designed robot device controllers, is used as a starting point for training.

[0016] Example 5 is a method of any one of Examples 1 to 4, wherein obtaining a demonstration includes obtaining a sequence of states generated by a sequence of control actions taken in the demonstration for each demonstration, and obtaining a reward for the states in the state sequence observed in the demonstration, and wherein training the critic neural network by reinforcement learning includes training the critic neural network with the rewards for the states in the obtained state sequence of the demonstration.

[0017] Using the obtained demonstrations in reinforcement learning further reduces the number of episodes necessary for training.

[0018] Example 6 is a robot controller configured to execute any of the methods in Examples 1 through 5.

[0019] Example 7 is a computer program that includes instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 1 to 6.

[0020] Example 8 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 1 to 7. Attached Figure Description

[0021] In the accompanying drawings, similar reference numerals are used throughout different views to generally refer to the same parts. The drawings are not necessarily to scale; rather, the emphasis is generally placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, wherein:

[0022] Figure 1 The robotic device is shown;

[0023] Figure 2 The diagram illustrates the controller training process according to the first embodiment;

[0024] Figure 3 The diagram illustrates the controller training process according to the second embodiment;

[0025] Figure 4A flowchart illustrating a method for controlling robotic equipment is shown. Detailed Implementation

[0026] The following detailed description refers to the accompanying drawings, which illustrate by way of explanation specific details and aspects in which the invention can be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0027] The various examples will be described in more detail below.

[0028] Figure 1 Robotic device 100 is shown.

[0029] Robotic device 100 includes a robotic arm 101, such as an industrial robotic arm for handling or assembling a workpiece (or one or more other objects). The robotic arm 101 includes manipulators 102, 103, and 104 and a base (or support) 105, which is supported by the base (or support) 105. The term "manipulator" refers to a movable component of the robotic arm 101, whose actuation enables physical interaction with the environment, for example, to perform a task. For control, robotic device 100 includes a (robotic) controller 106 configured to interact with the environment according to a control program. The last component 104 of the manipulators 102, 103, and 104 (farthest from the support 105) is also referred to as an end effector 104 and may include one or more tools, such as a welding torch, a gripping device, a painting apparatus, etc.

[0030] Other manipulators 102, 103 (closer to support 105) can form a positioning device to provide a robotic arm 101 with an end effector 104 at its end, together with the end effector 104. The robotic arm 101 is a mechanical arm capable of providing functions similar to a human arm (possibly with a tool at its end).

[0031] The robotic arm 101 may include joint elements 107, 108, and 109 that interconnect manipulators 102, 103, and 104 with each other and with support 105. Joint elements 107, 108, and 109 may have one or more joints, each joint providing rotational (i.e., rotational movement) and / or translational (i.e., displacement) motion relative to the associated manipulator. Movement of manipulators 102, 103, and 104 may be initiated by means of actuators controlled by controller 106.

[0032] The term "actuator" can be understood as a component adapted to affect a mechanism or process in response to being driven. An actuator can realize (so-called activation) a command issued by controller 106 as mechanical movement. An actuator, such as an electromechanical converter, can be configured to convert electrical energy into mechanical energy in response to a drive.

[0033] The term "controller" can be understood as any type of logical implementation entity, which may include, for example, circuitry and / or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and which may, for example, issue instructions to the actuator in this example. The controller may, for example, control the operation of the system (the robotic device in this example) through program code (e.g., software).

[0034] In this example, controller 106 includes one or more processors 110 and a memory 111 for storing code and data, based on which processors 110 control robotic arm 101. According to various embodiments, controller 106 controls robotic arm 101 based on a control strategy 112 stored in memory 111.

[0035] One approach to generating control policies is through reinforcement learning (RL). Reinforcement learning is characterized by trial-and-error search and delayed rewards. Unlike supervised learning, which requires labels to learn from, reinforcement learning uses a trial-and-error mechanism to learn the mapping from states to actions, thereby maximizing the reinforcement signal known as the reward. Through trial and error, reinforcement learning algorithms attempt to discover actions that lead to higher rewards by trying different actions. The choice of action affects not only the reward for the current state but also the rewards for all upcoming states, thus employing the concept of delayed rewards. Deep reinforcement learning (DRL) refers to using supervised learning to learn a neural network that can approximate a delayed reward approximation or directly map states to actions.

[0036] Model-free DRL algorithms are limited by their sample inefficiency, which leads to random initial behavior far removed from proficiency, making them unsuitable for real-world applications. Such poor initial behavior can potentially result in unsafe actions or unacceptable damage. In such cases, it is beneficial to initialize the learning agent with a significantly higher performance level than that produced by an agent initialized randomly, using demonstrations from a previously inadequate control agent (also known as learning from demonstrations (LfD)).

[0037] According to various embodiments, an actor-critic method is used. The actor and critic are implemented by actor neural networks and critic neural networks, respectively, trained during the learning process. For learning, target actor neural networks and target critic neural networks also exist, which are copies of the actor neural networks and critic neural networks, and are used to compute the loss functions of the actor neural networks and critic neural networks. The target actor neural network may, for example, slowly track the actor neural network (i.e., its weights are updated to change slowly in the direction of the actor neural network's weights), and the target critic neural network slowly follows the critic neural network (i.e., its weights are updated to change slowly in the direction of the critic neural network's weights). Using target networks for the actor and critic networks increases the stability of the learning process.

[0038] The references [1] and [2] mentioned above provide examples of actor-critic reinforcement learning methods that include a target network. Based on DDPG and DDPGfD described in these two references, the actor and critic networks and their target network are randomly initialized.

[0039] Conversely, according to various embodiments, the actor neural network and the critic neural network are pre-learned.

[0040] An initial actor network capable of mimicking a previously designed controller can be trained by using a controller that is deficient in the previous design (i.e., a control policy defined using prior knowledge, pre-knowledge, anticipation, or expert knowledge).

[0041] The critic network can be pre-trained using the initialization of the critic network to estimate the true reward of the initial actor's policy (i.e., the T-step reward), with the help of supervised training, where T is the total number of time steps in each episode, i.e., from time step... t The total discounted cumulative reward encountered up to the end time T). It should be noted that a plot refers to the sequence of agent-environment actions (and possible resulting state changes) between the initial state and the final state. At each time step, an action that can lead to a state change is selected.

[0042] Furthermore, according to various embodiments, the updates of the actor neural network, the critic neural network, and the target neural network are delayed, rather than being updated at every time step.

[0043] According to one embodiment, only policy development is used instead of exploration, or only a small amount of exploration is used. This means, for example, that actions in the reinforcement learning process are performed as given by the current control policy implemented by the actor.

[0044] Demonstrations collected from the initial strategy (which mimics a previously designed controller or expert (i.e., an expert's demonstration)) can also be used during the RL learning process.

[0045] The following sections will describe two implementations in detail. They are referred to as D3P2G version 1 and D3P2G version 2. D3P2G is an abbreviation for Delayed Deep Deterministic Pre-learned Policy Gradient.

[0046] Table 1 summarizes the variables used in the two versions.

[0047]

[0048] Figure 2 The diagram illustrates the training process based on the first version of the controller.

[0049] Figure 3 The diagram illustrates the training process based on the second version of the controller.

[0050] The controller training process includes initialization, followed by a reinforcement learning process.

[0051] Environments 201 and 301 can be viewed as defining a process to be performed under specific conditions (e.g., a task to be performed by robotic device 100, including interaction with one or more objects), and when in state s t The following is provided with action a t At that time, in state s t+1 A response is then initiated. According to various embodiments, it is assumed that prior knowledge exists in the form of a previously designed, inadequate control strategy (labeled as an expert). The expert is initially used to gather demonstrations from the environment.

[0052] These demonstrations, along with imitation learning, were used to train policy neural networks. Its parameters Used to initialize actor neural networks 202 and 302 (which implement the control strategy) ) and target actor neural networks 203 and 303 (which implement target control strategies) ) parameters , (i.e., weight).

[0053] Then use the initial control strategy. Collect trajectories (e.g., robot trajectories) from environment 201, 301. Each trajectory corresponds to a plot and includes state-action tuples (pairs) for each time step of that plot.

[0054] For each state-action tuple of these trajectories, the real reward is calculated as follows: The collected trajectory state-action tuples are used as input, and their computed true rewards are used as labels in a supervised learning process to learn the critic neural network. Its parameters Used to initialize commentator neural networks 204 and 304 (which implement the value function) ) and target commentator neural networks 205 and 305 (which achieve the target value function) ) parameters , (i.e., weight).

[0055] exist Figure 3 In the second version illustrated in the diagram, by The collected trajectories are also stored in PER memory 306 for use during the reinforcement learning process (demonstration 312). In the first version, there is playback memory 206 (not used to store demonstration 312).

[0056] The reinforcement learning process consists of multiple episodes, each of which includes multiple time steps T.

[0057] In each episode at each time step t At this point, the actor's neural networks 202 and 302 apply actions. a t (To the controlled system, such as robotic arm 101) interact with the environment 201, 301, this action a t Without any additional noise during development, the actor's neural network In state s t The output is located at [location]. The state, action, reward, and next state are stored in PER playback memories 206 and 306 (with n-step transitions in version 2). (together) are stored as tuples .

[0058] Furthermore, at each time step t At this point, N transitions are sampled from playback / PER memories 206 and 306. For each of the N sampled transitions... i The action value target is calculated as follows:

[0059] -For version 1:1 action value target Is it using rewards? Calculated

[0060] - For version 2: Use accumulated n-step rewards Calculate the value target of n-step actions

[0061] Using the action value objective, calculate the critic losses 207 and 307, and update the parameters of the critic neural networks 204 and 304 based on the loss gradient (arrows 208 and 308).

[0062] Every d Each time step, using The actors (arrows 209, 309) are updated, and the parameters of the actors and critics are soft-copied to the parameters of the target actor neural networks 203, 303 (arrows 210, 310) and the target critic neural networks 205, 305 (arrows 211, 311).

[0063] Tables 2 and 3 provide the pseudocode for two versions of the training process.

[0064]

[0065] .

[0066]

[0067]

[0068]

[0069]

[0070] Table 3.

[0071] According to one embodiment, version 1 includes the following modifications regarding DDPG:

[0072] - Pre-learning actors for actor initialization: Using imitation learning on demonstrations collected through interactions between a previously designed controller and the environment, actors with neural network parameters are learned. Initial strategy for actors

[0073] - Pre-learning of critics for initialization: Supervised learning, along with an appropriate loss function, is used to learn the initial critics, i.e., those with network parameters. of It can approximate the learned strategy. Real returns

[0074] - Use delay when updating actors, critics, and target networks.

[0075] - Development replaces exploration during the RL learning process

[0076] - Use L2 regularization on the network parameters for actors and critics.

[0077] According to one embodiment, version 1 includes the following modifications regarding DDPGfD:

[0078] - Pre-learning actors for actor initialization: Using imitation learning on demonstrations collected through interactions between a previously designed controller and the environment, actors with neural network parameters are learned. Initial strategy for actors

[0079] - Pre-learning of critics for initialization: Initial critics are learned by supervised learning along with an appropriate loss function, i.e., those with network parameters. of It can approximate the learned strategy. Real returns

[0080] - Use delay when updating actors, critics, and target networks during the RL learning process.

[0081] - Development replaces exploration during the RL learning process

[0082] - Change the critic's loss function during the RL learning process.

[0083] The main difference between the two versions of D3P2G is the difference between DDPG and DDPGfD. These differences are:

[0084] Version 2 loads expert demo transformations into the playback buffer before the learning process begins and retains all transformations for use during training.

[0085] Version 2 uses a PER memory instead of the normal non-prioritized replay memory used in version 1, enabling efficient reward propagation.

[0086] Version 2 uses an n-step action value target instead of the 1-step action value target used by version 1, thus propagating the Q value along the trajectory in a better way.

[0087] In summary, according to various embodiments, a method is provided as follows: Figure 4 The method is illustrated in the figure.

[0088] Figure 4 A flowchart 400 illustrating a method for controlling a robotic device is shown.

[0089] In section 401, a demonstration of how to control robotic equipment is obtained.

[0090] In 402, the initial training of the actor's neural network is performed through imitation learning demonstrated.

[0091] In 403, the robotic device is controlled by an initially trained actor neural network to generate multiple trajectories for the robotic device, wherein each trajectory includes a sequence of actions selected by the initial actor neural network in a sequence of states, and the reward for each of the actions selected by the initial actor neural network is observed.

[0092] In 404, the initial training of the critic neural network is performed through supervised learning, where the critic neural network is trained to determine the reward for the observed action selected by the initial actor neural network.

[0093] In 405, the actor neural network and the critic neural network are trained by reinforcement learning starting from the initially trained actor neural network and the initially trained critic neural network.

[0094] In 406, the robotic equipment is controlled by trained actor neural networks and trained critic neural networks.

[0095] According to various embodiments, in other words, the actors in the actor-critic reinforcement learning are pre-learned using imitation learning on a demonstration performed by, for example, a previously designed controller or a human user (for their initialization). Furthermore, supervised learning is used to pre-learn the critics (for their initialization), making it possible to approximate the true reward of the control policy implemented by the initially trained (i.e., pre-trained) actors (in the above terms, ...). ).

[0096] Delays can be used when updating actors, critics, and target networks.

[0097] It should be noted that the method described in this paper can be applied to any kind of modelless actor-critic deep reinforcement learning.

[0098] Figure 4The method can be performed by one or more computers including one or more data processing units. The term "data processing unit" can be understood as any type of entity that allows the processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include analog circuits, digital circuits, composite signal circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof. Any other manner of implementing the corresponding functions, which will be described in more detail below, can also be understood as a data processing unit or logic circuit. It should be understood that one or more of the method steps described in detail herein can be performed (e.g., implemented) by the data processing unit through one or more specific functions performed by the data processing unit.

[0099] The term "robot device" can be understood to refer to any physical system having mechanical parts whose movement is controlled, such as computer-controlled machines, vehicles, household appliances, power tools, manufacturing machines (e.g., for grinding, coating, cutting, etc.), personal assistants, or access control systems. In particular, the term "robot device" includes robots having one or more arms and / or one or more legs controlled by a robot controller.

[0100] Various embodiments can receive and utilize sensor signals from a variety of sensors—such as (e.g., RGB) cameras, video sensors, radar, LiDAR, ultrasonic sensors, thermal imaging sensors, etc.—for example, to obtain sensor data showing an object. Embodiments can be used to train robot device controllers to autonomously perform various manipulation tasks in different scenarios. In particular, embodiments can be applied to the control of manipulation tasks, for example, on an assembly line. Specifically, sensor data (e.g., by the robot device controller) can be used to observe the reward or reward of an action. For example, it can be determined from sensor data whether the objective of a task has been achieved. The reward or reward can also be observed through simulation.

[0101] The embodiments can operate on time-series state vector data obtainable by receiving sensor signals to compute control signals for controlling robotic devices. Sequential decision-making processes can be provided using prior knowledge employed in the form of previously designed but insufficient controllers. The embodiments are applicable to controlling any process that can be represented as a sequence of states and controlled by a sequence of actions (sequential processes). The embodiments can be applied to autonomously improve any previously designed controller through deep reinforcement learning (DRL). This may be a one-off improvement (over a finite time period) or continuous improvement during regular operation. Another application area is controlling complex or hazardous physical systems where the controller was previously designed to implement sequential tasks or processes.

[0102] While specific embodiments have been illustrated and described herein, those skilled in the art will understand that various alternatives and / or equivalent implementations may be used to replace the specific embodiments shown and described without departing from the scope of the invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, the invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for controlling a robotic device, comprising: Obtain a demonstration for controlling the robotic equipment; The initial training of the actor's neural network is performed by imitating the demonstration; The robotic device is controlled by an initially trained actor neural network to generate multiple trajectories of the robotic device, wherein each trajectory includes a sequence of actions selected by the initially trained actor neural network in a state sequence, and a reward for observing each of the actions selected by the initially trained actor neural network; The initial training of the critic neural network is performed through supervised learning, wherein the critic neural network is trained to determine the reward for the observed action selected by the initially trained actor neural network; Starting with initially trained actor neural networks and initially trained critic neural networks, the actor neural networks and critic neural networks are trained through reinforcement learning; and The robotic equipment is controlled by trained actor neural networks and trained critic neural networks. The method of training an actor neural network and a critic neural network through reinforcement learning includes executing multiple episodes, wherein in each episode, the actor neural network selects a control action for each of a plurality of time steps, and for each of the plurality of time steps, the robot is controlled to execute the selected control action in the time step. The method includes copying the actor neural network to generate a target actor neural network only after a delay of several time steps in the plurality of time steps, and copying the critic neural network to generate a target critic neural network. The method also includes using the target actor neural network to provide one or more control actions following the control action selected by the actor neural network to the target critic neural network for each control action selected by the actor neural network. The target critic neural network determines a reward for the one or more control actions selected by the target actor neural network, and the critic's loss is determined based on the reward determined by the critic for the action selected by the actor neural network, the sum of the rewards for the one or more control actions selected by the target actor neural network, and the reward for the action selected by the actor neural network.

2. The method of claim 1, wherein training the actor neural network comprises updating the actor neural network at predetermined time steps to select actions with higher rewards as specified by the critic neural network.

3. The method according to any one of claims 1 to 2, wherein obtaining a demonstration includes obtaining a demonstration from a human user and / or a pre-designed robot device controller.

4. The method according to any one of claims 1 to 2, wherein obtaining the demonstration comprises obtaining, for each demonstration, a state sequence generated by a sequence of control actions taken in the demonstration, and obtaining a reward for the state in the state sequence observed in the demonstration, and wherein training the critic neural network by reinforcement learning comprises training the critic neural network by the reward for the state in the obtained state sequence of the demonstration.

5. A robot controller configured to perform the method of any one of claims 1 to 4.

6. A computer program including instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4.

Images