Method and apparatus for controlling a robot

Abstract

Methods and apparatus for controlling a robot. According to various embodiments, a method for controlling a robot is described, the method comprising: obtaining sensor data representing a robot environment; identifying one or more objects in the robot environment from the sensor data; associating each of the one or more objects and the robot with a respective agent of a multi-agent system; determining a quality metric for each agent of the multi-agent system, the quality metric comprising a reward term for a movement action at a location and a coupling term dependent on a probability that other agents occupy the same location as the agent at a time; determining a movement policy for the robot, the movement policy selecting a movement action with a higher value of the quality metric with a higher probability than a movement action with a lower value of the quality metric determined for the robot; and controlling the robot in accordance with the movement policy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an apparatus and a method for controlling a robot. BACKGROUND

[0002] Multi-agent learning is a challenging problem in machine learning, which has applications in different domains such as distributed control, robotics, including autonomous driving, and economics. In many multi-agent systems, an agent does not necessarily choose its optimal strategy with respect to other agents. For example, an autonomous vehicle will not suddenly replace a human-controlled vehicle and will not be isolated from human-controlled traffic on a separate route. Therefore, an autonomous vehicle will have to move safely in mixed traffic with human-controlled other objects, whereby these external objects also include pedestrians as weaker road users. In the case of human-controlled external objects, there is uncertainty as to which movement action these other objects will perform next.

[0003] The publication DE 102018210280 A1 describes a trajectory planning method for an autonomous vehicle, in particular in mixed traffic with human-controlled other objects. In this method, a Q-function is established for each other object and an optimal movement strategy is determined for the autonomous vehicle and the other objects by maximizing the Q-function. From the optimal movement strategy, a trajectory of the other objects and a trajectory of the autonomous vehicle are determined.

[0004] However, this method has an exponential runtime and memory requirement. The memory requirement of this method is TxAxP x S P where S is the number of states, P is the number of players, A denotes the number of actions, and T is the time span (number of time steps). Therefore, this method can be applied to small state and action spaces with a small number of players, but the computation time for a multi-agent game of 20 states and 4 players and 5 time steps can easily exceed a computation time of three days.

[0005] Therefore, a more efficient method for determining trajectories in multi-agent scenarios is desirable. SUMMARY

[0006] The method for controlling a robot and the robot controller with features corresponding to the first example given below and corresponding to the seventh example given below provide a more efficient method for trajectory planning than the above described methods. Since the interaction between agents is only modeled through a coupling term that includes an occupancy measure, the computational complexity and memory can be reduced. In particular, the exponential memory and runtime requirements resulting from modeling the behavior of each agent as a Q-function over the product space of the states of all agents can be avoided. For example, for a multi-agent game with 20 states and 4 players and 5 time steps, the computation time can be reduced to 1 second and thus make the method practically usable.

[0007] In the following, exemplary embodiments are given.

[0008] Example 1 is a method for controlling a robot, the method comprising: obtaining sensor data representing an environment of the robot; identifying one or more objects in the environment of the robot from the sensor data; associating each of the one or more objects and the robot with a respective agent of a multi-agent system; determining a quality measure for each agent of the multi-agent system, the quality measure comprising a reward term for a movement action at a position and a coupling term that depends on a probability of other agents occupying the same position at a time; determining a movement policy for the robot, the movement policy selecting a movement action with a higher value of the quality measure with a higher probability than a movement action with a lower value of the quality measure determined for the robot; and controlling the robot according to the movement policy.

[0009] Example 2 is the method according to example 1, wherein the coupling term is a functional of the occupancy measures of the other agents, wherein, for each agent, the occupancy measure of a position and a time indicates a likelihood of the agent being in the position at the time.

[0010] The quality measures of the various agents via the coupling of the occupancy measures allow to efficiently solve the multi-agent reinforcement learning problem and to derive a control policy for the robot.

[0011] Example 3 is the method according to example 1 or 2, wherein determining the quality measure comprises iteratively determining the quality measure in a plurality of iterations, wherein each iteration comprises a forward pass over a plurality of time steps from an initial time to an end time and a backward pass over the plurality of time steps from the end time to the initial time.

[0012] The iterative method comprising the forward pass and the backward pass allows to solve the interdependencies of the quality measure system of the set of agents in an efficient way.

[0013] Example 4 is the method of example 3, wherein the coupling term is a functional of the occupancy measures of the other agents, and the forward pass comprises updating the occupancy measure of each agent for the next time step by propagating the occupancy measure of the agent at the current time step using the policy of the agent at the current time step.

[0014] This allows the occupancy measure to be updated in an iteration.

[0015] Example 5 is the method of example 3 or 4, wherein the coupling term is a functional of the occupancy measures of the other agents, and the backward pass comprises updating the quality measure and the policy of each agent at the current time step for the next time step by using the occupancy measures of the other agents at the next time step.

[0016] This allows the quality measure to be updated in an iteration.

[0017] Example 6 is the method of any one of examples 1 to 5, wherein the movement policy is determined such that, depending on the quality measure determined for the robot, actions for the system state of the multi-agent system are distributed according to a Boltzmann distribution.

[0018] Determining the movement policy as a Boltzmann distribution gives a degree of freedom to deviate from strictly optimal behavior.

[0019] Example 7 is a robot controller configured to perform the method of any one of examples 1 to 6.

[0020] Example 8 is a computer program comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of examples 1 to 6.

[0021] Example 9 is a computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of examples 1 to 6. BRIEF DESCRIPTION OF DRAWINGS

[0022] In the drawings, like reference numerals refer to same parts throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed on illustrating the principles of the application. In the following description, various aspects will be described, reference being made to the accompanying drawings, in which:

[0023] Figure 1 An example of an autonomous driving scenario is shown.

[0024] Figure 2 A graphical model showing the dependencies between the variables in a multi-agent reinforcement learning algorithm according to an embodiment with two agents is shown.

[0025] Figure 3 A flowchart illustrating a method for controlling a robot according to an embodiment is shown. Detailed Implementation

[0026] The following detailed description refers to the accompanying drawings, which illustrate by way of example specific details and aspects of the present disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the invention. The various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0027] The various examples will be described in more detail below.

[0028] Figure 1 An example of autonomous driving scenario 100 is shown.

[0029] exist Figure 1 In the example, a vehicle 101, such as a car, truck, or motorcycle, is provided with a vehicle controller 102.

[0030] The vehicle controller 102 includes a data processing component, such as a processor (e.g., a CPU (central processing unit)) 103, and a memory 104 for storing control software of the vehicle controller 102 according to its operation and data of the processor 103 on its operation.

[0031] For example, the stored control software includes instructions that, when executed by processor 103, cause the processor to implement movement strategy determiner 107.

[0032] The movement strategy determiner 107 determines a movement strategy for vehicle 101 (and thus the (desired) trajectory for vehicle 101) to avoid collisions with other objects 108. Other objects 108 may be, in particular, people or vehicles controlled by people, such as conventional motor vehicles or bicycles. However, there may also be other objects that are uncontrollable or only controllable to a limited extent, such as a vehicle that has rolled away after parking on a slope or a trailer that has detached from its towing vehicle.

[0033] The vehicle controller 102 first identifies the other object 108. For this purpose, a time series of physical observations of the environment can be used, such as a sequence of camera images acquired by one or more cameras 105 and stored in memory 104, or a sequence of events output by event-based sensors. Alternatively or in combination, information received via the wireless interface of the vehicle 101 can be used. This information can be transmitted by the other object 108 itself, for example, via a vehicle-to-vehicle (V2V) interface. However, the information can also be transmitted by infrastructure, for example, via a vehicle-to-infrastructure (V2I) interface.

[0034] The movement policy determiner 107 can determine a movement policy (e.g., by determining trajectories of other objects and a desired trajectory for the vehicle 101) and the vehicle controller 102 controls the vehicle 101 in accordance with the determined result. For example, the vehicle controller 102 can control the actuators 106 to control the speed of the vehicle (e.g., to actuate the brakes of the vehicle) and the direction of the vehicle 101.

[0035] According to various embodiments, the movement policy determiner 107 models the vehicle 101 and the other objects by a non-cooperative multi-agent system that models the interaction between the agents via an occupancy metric of the agents. This means that an agent chooses its next action based on its individual reward and the occupancy metric of the state (e.g., the geographical position). The occupancy metric represents the likelihood of an agent being in a given state at a given time. It should be noted that the states of all agents at a certain time (time step) are considered to form the (overall) system state of the multi-agent system.

[0036] According to various embodiments, the model uses a set of Q-functions (which can be seen as Q-functions of a multi-agent reinforcement learning problem) that are coupled by the occupancy metric (for each state of each agent and time step). These Q-functions capture the behavior of the agents at each time step and system state.

[0037] The movement policy determiner 107 models the behavior of traffic participants using the multi-agent system. More precisely, the movement policy determiner can use the resulting set of Q-functions to predict the future states of individual traffic participants. This helps to control a self-driving vehicle without colliding with other traffic participants and to create a smooth driving experience.

[0038] It should be noted that while embodiments are described for autonomous driving scenarios, the methods described herein can also be used in other scenarios, e.g., to predict the position of an airplane or a ship. Similarly, the methods described herein can be used not only to derive a control policy for an autonomous vehicle, but also for any other robot or autonomous agent.

[0039] The term "robot" can be understood to refer to any physical system (with mechanical components whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0040] Generally speaking, the methods described herein can be used to solve any kind of multi-agent game, e.g., in the context of one of the above-mentioned application scenarios.

[0041] Modeling the behavior of each agent at a time step t as a Q-function over the product space of all agent states leads to exponential memory and runtime requirements.

[0042] Instead, according to various embodiments, each agent's Q-function at each time step is made independent of the exact position of other agents. The individual Q-function of player i at system state x and time step t depends only on player i's position and actions, and the occupancy metric (probability) of other agents i at state x, by... Give, Used to indicate that agent j is in state x at time τ i The possibility of [a specific location]. More precisely, the Q-function for agent i is defined as [a specific function].

[0043]

[0044] in

[0045] Where M is the number of agents (or players), and Ψ(·) is a functional. As an example, in a self-driving car scenario, for Possible options are Where μ j >0.

[0046] Expectations are taken over by probabilistic state transitions, and the value function is defined as follows: and Where R i,F (x) indicates the final reward of agent i at state x.

[0047] The move policy determiner 107 uses the Q function of player i at system state x at time τ of action a to compute a policy that defines the probability of choosing action a given system state x and time τ.

[0048] These Q-functions are coupled using the agent's occupancy probability. Obtaining the agent's Q-function hints at its strategy. Therefore, finding these Q-functions is the objective in trajectory determination methods. A Q-function is a solution to a set of coupled equations. According to various embodiments, a forward-backward algorithm is used to solve this set of coupled equations. An example of this algorithm is given in pseudocode below.

[0049]

[0050] The algorithm starts with an initial guess of the Q-function, then updates the occupancy metric in the forward pass and updates the Q-function in the backward pass.

[0051] More precisely, in the forward pass, the occupancy metrics of all players at all system states are propagated to future time steps τ using the Q function and the occupancy metrics from time τ-1.

[0052] Action probability can be used for the following propagation occupancy metric

[0053]

[0054] for i e [M]

[0055] where

[0056] The Q-function of player i at time t for action a at system state x can be used to compute a policy that defines the probability of choosing action a at a given state x and time t:

[0057] for i e [M].

[0058] This means that all agents choose their mixed strategies such that actions with a higher Q-function are chosen with a higher probability (according to the Boltzmann distribution in this example).

[0059] During the backward pass, the Q-function Q i The occupancy measure O -i (where -i denotes the opponent of agent i) and the already computed Q-functions from future time steps are updated according to

[0060]

[0061] for i e [M],

[0062] which describes the dependency between the Q-function and the occupancy measure.

[0063] The initial conditions of the algorithm are the occupancy measure at time 0 and the Q-functions at the final time step (corresponding to the given final reward).

[0064] It should be noted that it is assumed that the agents perceive each other's objectives, i.e. the reward function {R j , R j , F} j is known to all agents.

[0065] It can be shown that under certain conditions, the above algorithm converges to a unique set of Q-functions as the number of iterations K tends to infinity.

[0066] Figure 2 A graphical model 200 is shown that demonstrates the dependencies between the variables in the above algorithm with two agents {i, j} and a time span T = 2. Here, k denotes the k-th iteration of the algorithm, denotes the Q-function of player i at time step 1 at the k-th iteration of the algorithm.

[0067] In summary, according to various embodiments, there is provided a method as illustrated in Figure 3 Fig. 2.

[0068] Figure 3 A flowchart 300 illustrating a method for controlling a robot according to an embodiment is shown.

[0069] In 301, sensor data representing an environment of the robot is acquired.

[0070] In 302, one or more objects in the environment of the robot are identified from the identified sensor data.

[0071] In 303, each of the one or more objects and the robot are associated with a respective agent of a multi-agent system.

[0072] In 304, for each agent of the multi-agent system, a quality measure is determined, the quality measure comprising a reward term for a movement action at a position and a coupling term depending on a probability that other agents occupy the same position as the agent at a time.

[0073] In 305, a movement policy for the robot is determined, the movement policy selecting a movement action with a higher value of the quality measure with a higher probability than a movement action with a lower value of the quality measure determined for the robot.

[0074] In 306, the robot is controlled according to the movement policy.

[0075] According to various embodiments, in other words, a multi-agent reinforcement learning is used to determine a movement policy for the robot, wherein the Q-function (i.e., the quality measure) of an agent i is decomposed into two terms: an individual objective of the agent, which depends only on (x i , a i ); and another term, which encodes the interaction between i and its opponents by means of an occupancy measure.

[0076] Figure 3The methods of the present application can be performed by one or more computers comprising one or more data processing units. The term "data processing unit" can be understood as any kind of entity that allows processing data or signals. For example, data or signals can be handled in accordance with at least one, i.e. one or more, specific function performed by the data processing unit. The data processing unit can comprise or be formed by an analog circuit, a digital circuit, a complex signal circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA) integrated circuit, or any combination thereof. Any other means of implementing a respective function, which will be described in more detail below, can also be understood as a data processing unit or logic circuit. It will be understood that one or more method steps described in detail herein can be performed, e.g. implemented, by a data processing unit by one or more specific functions performed by the data processing unit.

[0077] While specific embodiments have been illustrated and described, it will be appreciated that various alternative and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present application. This application is intended to cover any adaptations or variations of the specific embodiments discussed. Therefore, it is intended that the application be limited only by the claims and the equivalents thereof.

Claims

1. A method for learning a movement strategy to control a robot, the method comprising: Acquire sensor data representing the robot's environment; Identify one or more objects in the robotic environment based on sensor data; Associate each of the one or more objects and the robot with the corresponding agent of a multi-agent system of multi-agent reinforcement learning; For each agent in a multi-agent system, a Q-function is determined, the Q-function comprising a reward term, an expectation term, and a coupling term for a movement action at a location, wherein the reward term and the expectation term are independent of the other agents, and wherein the coupling term is a functional of the occupancy metrics of the other agents, wherein for each agent, the occupancy metrics of location and time indicate the probability that the agent is in the location at the time. The multi-agent reinforcement learning is used to determine the robot's movement policy, wherein the movement policy selects a movement action with a higher value of the Q function with a higher probability than a movement action with a lower value of the Q function determined for the robot.

2. The method according to claim 1, comprising the step of controlling the robot according to the movement strategy.

3. The method of claim 1, wherein determining the Q function comprises iteratively determining the Q function over multiple iterations, wherein each iteration comprises a forward pass from an initial time to an end time over multiple time steps and a backward pass from the end time to the initial time over multiple time steps.

4. The method of claim 3, wherein the coupling term is a functional of the occupancy metrics of other agents, and the forward propagation includes propagating the occupancy metrics of the agents at the current time step by using the agents' policies at the current time step, and updating the occupancy metrics for the next time step for each agent.

5. The method of claim 3 or 4, wherein the coupling term is a functional of the occupancy metric of other agents, and the backward propagation includes updating the Q-function and policy of each agent at the current time step by using the occupancy metric of other agents at the next time step.

6. The method according to any one of claims 1 to 4, wherein a movement strategy is determined such that actions for the system state of the multi-agent system are distributed according to a Boltzmann distribution, depending on the Q-function determined for the robot.

7. A robot controller configured to perform the method of any one of claims 1 to 6.

8. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

Images