Apparatus and method for controlling a robot

Abstract

The present disclosure relates to an apparatus and method of controlling a robot. The method comprises: receiving an indication of a target configuration to be reached; determining a coarse scale value map by value iteration, wherein transition probabilities are determined using a transition probability model; and for each of a sequence of fine scale states, starting from an initial coarse scale state and until either the target configuration is reached or a maximum number of fine scale states has been reached, determining a fine scale sub-goal from the coarse scale value map; performing a fine scale control action by an actuator and obtaining sensor data to determine a reached fine scale state, starting from the current fine scale state and until either a different coarse scale state is transitioned to, or a maximum sequence length of the sequence of fine scale states has been reached; and determining the next one of the sequence of coarse scale states from the last one of the sequence of fine scale states.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an apparatus and a method for controlling a robot. BACKGROUND

[0002] Reinforcement learning (RL) is a paradigm of machine learning that allows a machine learning to perform desired behavior in relation to a task specification, e.g. it controls the actions taken in a robot navigating a scene to reach a target position. Learning a policy that generates these behaviors with reinforcement learning differs from learning a policy with supervised learning in the way the training data is constructed and obtained: while in supervised learning the training data provided consists of matching input pairs for the policy (e.g. observations like sensor readings) and desired outputs (actions to take), in the case of reinforcement learning no fixed training data is provided. The policy is learned from experience data collected by the machine’s interaction with its environment, whereby the machine is provided with a feedback (reward) signal that scores / evaluates the actions taken in a particular context (state).

[0003] Data efficiency of RL algorithms is low in cases where the provided reward signal is very sparse. Thus, training with a trained control policy to satisfy some quality criterion can take a very long time.

[0004] The publication “Value Iteration Networks” by Tamar et al. (20 March 2017) describes a value iteration network (VIN) as a fully differentiable planning module that learns an approximation of the value iteration planning algorithm. In continuous control tasks, they hierarchically combine a high-level VIN planner on a discrete, coarse grid representation of the continuous environment with a low-level continuous control policy trained via guided policy search. The low-level policy receives as additional input a local excerpt of the value function (mapping) generated by the VIN planner.

[0005] The HiDe algorithm described in Jendele et al., “Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks”, September 25, 2019, combines a Value Propagation network (VProp), an improvement of VIN, and reinforcement learning hierarchically for solving robot navigation tasks in continuous state and action space. The high-level Vprop planner generates subgoals for the low-level reinforcement learning. The VProp planner receives as input a 2D bird’s eye view image of the environment (a coarse discrete map representation) and computes as output a value map. The current 2D position of the agent falls into one of the discrete cells. The next goal cell for the low-level (RL) policy is chosen by picking the neighboring cell with the highest value in the value map. The goal vector is generated by subtracting the current 2D position from the goal grid cell position. This goal vector is refined by an “interface layer policy” before being provided as input to the low-level continuous control RL policy, which additionally receives as input the internal agent state and computes a continuous action applied to the actuators of the agent. The whole hierarchical three-level planning / policy is trained end-to-end via reinforcement learning.

[0006] However, end-to-end reinforcement learning, including learning the value iteration process itself, requires a lot of training effort and, in particular, a lot of training data.

[0007] In view of the above, a more data-efficient method for controlling a robot is desirable. SUMMARY

[0008] The method and apparatus disclosed herein allow for controlling a robot by training with less observed data, as only a model of the transition probabilities needs to be learned for the high-level planner (e.g., using supervised learning), as opposed to learning the value iteration process itself end-to-end via reinforcement learning with a low-level continuous control policy, as in the above-mentioned HiDe method, which is a more difficult learning task (as the transition and reward functions also have to be learned implicitly).

[0009] The method of the present invention is also computationally more efficient than the method of learning a value iteration model such as HiDe, as it is not necessary to update the (recurrent) planner by backpropagating gradients through the planner via the continuous control policy.

[0010] In the following, exemplary embodiments are given.

[0011] Example 1 is a method for controlling a robot, comprising: receiving an indication of a target configuration (state) to be reached from an initial configuration (state) of the robot; determining a coarse-scale value map by value iteration, wherein a transition probability is determined using a transition probability model that maps a coarse-scale state and a coarse-scale action to a transition probability of a coarse-scale state; and for each coarse-scale state in a coarse-scale state sequence of the robot, starting from an initial coarse-scale state determined from the initial configuration (state) of the robot and until the robot reaches the target configuration (state) or a maximum number of fine-scale states has been reached, determining a fine-scale subgoal from the coarse-scale value map; performing, by an actuator of the robot, a fine-scale control action to reach the determined fine-scale subgoal and obtaining sensor data to determine a fine-scale state reached as a result of performing the fine-scale control action for each fine-scale state in a resulting fine-scale state sequence of the robot, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale subgoal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the fine-scale state sequence has been reached; and determining a next coarse-scale state of the coarse-scale state sequence from a last fine-scale state of the fine-scale state sequence.

[0012] Example 2 is the method of example 1, wherein the transition probability model is a model that is trainable by supervised learning.

[0013] Example 3 is the method of example 1 or 2, wherein the transition probability model is a neural network.

[0014] Using a model that is trainable by supervised learning, such as a neural network for determining the transition probabilities, allows for the use of value iteration planning on a high level (i.e., coarse scale) and thus allows for a data-efficient training of the robot control.

[0015] Example 4 is a robot controller configured to perform the method of any one of examples 1 to 3.

[0016] Example 5 is a method for training a robot controller according to example 4.

[0017] Example 6 is a method for training the robot controller of example 5, comprising training the transition probability model using supervised learning.

[0018] As mentioned above, using supervised learning to learn a transition probability model for value iteration is more data-efficient than learning to approximate performing a value iteration algorithm end-to-end with a low-level control policy via reinforcement learning (implicitly including learning transition dynamics).

[0019] Example 7 is a method for training the robot controller of example 6, comprising training the transition probability model by maximizing the probabilities of the transition probability model to predict the coarse scale state reached by the fine scale action performed to reach the subgoal.

[0020] Training the transition probability model in this way allows to train the high level planner while performing reinforcement learning for the low level policy, i.e. learning the low level control policy by performing training rollouts to collect data to update the policy.

[0021] Example 8 is a method for training the robot controller of any one of examples 5 to 7, comprising training the fine scale control policy via reinforcement learning, said reinforcement learning determining the fine scale control action to reach the subgoal.

[0022] Thus, the low level control can be trained using reinforcement learning methods of selection such as Q-learning, Deep Q-Networks (DQN), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), (Asynchronous) Advantage Actor Critic (A2C / A3C).

[0023] Example 9 is a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of examples 1 to 3 or 5 to 8.

[0024] Example 10 is a computer readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of examples 1 to 3 or 5 to 8. BRIEF DESCRIPTION OF DRAWINGS

[0025] In the drawings, like reference numerals refer to like parts throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. In the following description, various aspects of the application are described with reference to the following drawings, in which:

[0026] Figure 1 A control scenario according to an embodiment is shown;

[0027] Figure 2 A neural network for predicting a high level state is shown;

[0028] Figure 3 A flowchart illustrating a method for controlling a robot according to an embodiment is shown.

[0029] The following detailed description references the accompanying drawings, which illustrate specific details in the context of present disclosures. Other aspects can be utilized, and structural, logical and electrical changes can be made without departing from the scope of the present disclosure. The various aspects of the disclosure are not necessarily mutually exclusive, as some aspects of the disclosure can be combined with one or more other aspects of the disclosure to form new aspects. DETAILED DESCRIPTION

[0030] In the following, various examples will be described in more detail.

[0031] Figure 1 A control scenario is illustrated.

[0032] A robot 100 is located in an environment 101. The robot 100 has a start position 102 and should reach a target position 103. The environment 101 contains obstacles 104 that the robot 100 should avoid. For example, they cannot be passed by the robot 100 (e.g., they are walls, trees or rocks) or should be avoided because the robot would damage or harm them (e.g., pedestrians).

[0033] The robot 100 has a controller 105 (which can also be remote from the robot 100, i.e., the robot 100 can be controlled by remote control). In Figure 1 In the exemplary scenario, the goal is that the controller 105 controls the robot 100 to navigate from the start position 102 to the target position 103 for the environment 101. For example, the robot 100 is an autonomous vehicle, but it can also be a robot with legs or tracks or other kinds of propulsion systems (such as a deep sea or Mars rover).

[0034] Further, embodiments are not limited to scenarios where the robot should move (as a whole) between the positions 102, 103, but can also be used for controlling a robot arm etc. whose end effector should move between the positions 102, 103 (without hitting the obstacles 104).

[0035] Ideally, the controller 105 has learned a control policy that allows it to successfully control the robot 101 (from the start position 102 to the target position 103 without hitting the obstacles 104) for any of the particular scenarios (i.e., environments, start and target positions) that the controller 105 has not encountered before.

[0036] Thus, various embodiments are concerned with learning a control policy for a particular (assigned) task(s) by interacting with the environment 101. In training, the scenario (in the particular environment 101) can be simulated, but it will typically be real in deployment.

[0037] Reinforcement learning (RL) is a technique for learning a control policy. An RL algorithm iteratively updates parameters of a parametric policy that maps a state s (e.g., a (pre-processed) sensor signal) to an action a (control signal), e.g., represented by a neural network. During training, the policy is interactively engaged with the (possibly simulated) environment 101 in a rollout. During a (simulated training) rollout in the environment 101, according to the current control policy, the controller 105 performs an action a at each discrete time step according to the current state s, which leads to a new state s' in the next discrete time step. Further, a reward r is received, which is used to update the policy. A (training) rollout ends as soon as a target state is reached, the cumulative (potentially discounted) reward exceeds a threshold, or a maximum number of time steps, i.e., a time horizon T, is reached. During training, a reward-related objective function (e.g., a discounted sum of rewards received during a rollout) is maximized by updating the parameters of the policy. Training ends as soon as the policy satisfies a certain quality criterion with respect to the objective function, a maximum number of policy updates has been performed, or a maximum number of steps in the (simulated) environment has been taken.

[0038] RL algorithms are data inefficient in cases where the provided reward signal is very sparse: e.g., a binary reward indicating completion of a task is only provided at the end of an interaction episode. Thus, training with a policy satisfying a certain quality criterion can take a very long time, requiring many interaction steps in the (simulated) environment and / or policy updates, or fail otherwise.

[0039] Given some prior knowledge about the (simulated) environment (e.g., availability of a map in case of a robot / vehicle navigation task), a combination of planning algorithms that guide reinforcement learning can be used in order to improve data efficiency.

[0040] According to various embodiments, the controller 105 uses a high-level planning on a coarse discrete (mapping) representation of the environment 101 to set (relative) sub-goals (goal vectors) for a low-level controller trained via reinforcement learning acting in a continuous state and action space of the environment 101. The coarse discrete representation of the environment is, e.g., a grid 106 (as shown in dashed lines in Figure 1

[0041] ​​Instead of trying to solve the difficult learning problem of learning an end-to-end high-level planner and low-level policy via reinforcement learning, the learning of planner and policy parameters is correspondingly split into two (related) problems: the planner parameters (representing the transition probabilities between high-level (i.e., coarse-scale) states) are inferred via supervised learning, which is a simpler learning problem than reinforcement learning, and the policy parameters are learned via reinforcement learning. Unlike the HiDe method described above, a precise value-iteration planning algorithm is used, rather than learning an approximation (VIN / VProp). To this end, a model of transition probabilities between coarse discrete high-level states is trained and used, given a selected neighboring goal coarse-scale state and the current low-level reinforcement learning policy. This model of transition probabilities reflects the current capabilities of the low-level reinforcement learning agent. It is learned from data collected during reinforcement learning episodes. Different types of models of transition probabilities are in principle possible. Specifically, according to various embodiments, a parametric model (neural network) whose parameters are the parameters of the planner is trained by supervised learning, which works with local agent observation features and thereby generalizes across different environment layouts.

[0042] In the following, embodiments can be described in more detail.

[0043] Task

[0044] Consider a distribution of Markov Decision Processes (MDPs) that share the same state space and action space A particular MDP, can be sampled. The start state and goal state are sampled from the MDP-specific state space The (goal-dependent) reward function has the form: where is some distance metric such that there is a reward of 1 only if the goal (i.e., destination) has been reached (where is a small number, e.g., the size of a label to avoid that the distance has to become exactly zero). is the MDP-specific transition dynamics, which models the transition from state s to next state s’ given as a result of action a. is a discount factor, and T is the time horizon (maximum number of steps to reach goal state 103 from start state 102).

[0045] Goal

[0046] The goal is to make the MDP m ​reward maximization in expectation (for task completion) of:

[0047] .

[0048] hierarchical state and action space

[0049] continuous low-level state space is split into an external part describing the agent (e.g. robot 101) in its environment (e.g. agent position) and an internal part describing the internal state of the agent (e.g. joint angles of robot 101) where where denotes the Cartesian product. Surjective mapping transforms the continuous low-level state space into a finite high-level state space . Another mapping transforms the high-level state back to the reference external low-level state . The high-level action space consists of a finite number of time-extended options (corresponding to subgoals can be seen). The ideal / expected outcome (next high-level state) of taking an option o in a high-level state z is denoted by .

[0050] low-level reinforcement learning

[0051] The agent perceives local feature observations based on its external state . For example, the robot 101 has sensors that observe the vicinity of the robot - e.g. the presence of obstacles 104 in its vicinity. In particular, for each coarse scale (i.e. high-level) state z (position of the robot on the coarse grid), there can be a feature observation which may, for example, include information for each tile in a part of the coarse grid 106 around the robot 101 (e.g. in a 3x3 segment of the coarse grid 106 in which the robot is centered). This may, for example, include information about the presence of obstacles in the respective coarse grid tile, the type of tile (e.g. rugged terrain, flat terrain). The feature observation can also include other information of the coarse scale state such as the velocity and orientation of the robot 101.

[0052] option-dependent low-level policy is denoted by where is the option-dependent goal vector from the current external state to the subgoal: .

[0053] initial set is denoted by ​; At each place any option can be chosen.

[0054] Termination function .

[0055] Option-dependent (subgoal) reward function for low-level reinforcement learning policy where when option o is chosen, is the agent state.

[0056] Low-level reinforcement learning policy is trained using sub-episodes that occur within training episodes from repeated option activations and terminations with horizon T. Thus, execution of option o has a variable maximum time horizon which depends on the time step at which the option was activated.

[0057] High-level (value iteration) planner

[0058] Value iteration iteratively refines estimates of state value functions V or state-action value functions Q. Let V k be the state value function at the kth iteration, and let Q k be the state-action value function at the kth iteration. These can be defined recursively. Value iteration starts with an arbitrary function (which can be thought of as an initial guess) V0. Value iteration uses the following equation to obtain the function for k + 1

[0059]

[0060] denotes the transition dynamics of reaching the next state s' from the current state s given action a.

[0061] When considering the high-level state corresponding to the expanded state s of the low-level policy it can be observed that the high-level state follows transition dynamics that depend on the low-level policy . From data, a parametric model with parameters is learned that models these transition dynamics well enough. To generalize across MDPs within the distribution , the model is learned with respect to feature observations corresponding to the high-level state: .

[0062] The high-level planner runs value iteration to obtain the value mapping . The options that implicitly define the subgoals for low-level reinforcement learning are greedily selected by the policy over the options .

[0063] Retrospective objective

[0064] Hence, the reward maximization in the objective is with respect to the parameters of the low-level reinforcement learning policy and the parameters of the learned high-level transition dynamics model of the high-level (value-iteration) planner :

[0065] .

[0066] Components

[0067] According to various embodiments, the controller 105 applies a hierarchical planning guided RL control policy that comprises a high-level (coarse scale) value-iteration planner trained via reinforcement learning and a low-level (fine scale) policy.

[0068] The high-level value-iteration planner performs value-iteration planning in a coarse state space abstraction (high-level state space). The sub-objective of the low-level reinforcement learning is generated by selecting a high-level action (option) based on the generated (high-level) value mapping. The high-level planning maintains a transition probability model of the coarse scale state space. This model can have the following specific architecture.

[0069] Figure 2 A neural network 200 for predicting a high-level state is shown.

[0070] The neural network 200 implements a classifier that predicts the next high-level state by selecting one of the (fixed amount of) neighboring high-level states provided as input 201, the current high-level state (in terms of the feature observations of the current high-level state) and the selected option (from which the sub-objective of the low-level reinforcement learning is derived). The neural network 200 comprises a plurality of hidden layers 202 followed by a softmax layer 203. The softmax probabilities 204 of the classifier are used as transition probabilities 204 to all possible neighboring high-level states and are used to determine the output 205 of the neural network.

[0071] The low-level RL module comprises a control policy that interacts with the environment and is rewarded for achieving the sub-objective set by the high-level planning. It can be trained by a reinforcement learning algorithm of choice - e.g. Q-learning, Deep Q-Networks (DQN), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), (Asynchronous) Advantage Actor Critic (A2C / A3C), etc.

[0072] Training

[0073] The hierarchical policy is trained over several training iterations until a maximum number of training iterations is reached or it satisfies a predefined quality criterion on the objective function. In each iteration:

[0074] I. Data is collected by interacting with the (simulated) environment(s) over multiple episodes. In each episode:

[0075] a. A particular (simulated) environment is provided / sampled as well as a starting state and a desired target state of the agent;

[0076] b. Given a coarse representation of the environment, the desired target state of the agent and the current transition probability model, a value iteration planner is run that generates a value map for the high-level states;

[0077] c. Starting from the starting state and providing an initial subgoal high-level state from the high-level planner, which is determined based on the initial agent state and the value map as one of the neighboring high-level states, the agent interacts with the (simulated) environment until the target state is reached or a maximum number of interaction steps (time horizon) is reached. In each interaction step:

[0078] i. The agent receives some observations (potentially pre-processed sensory information);

[0079] ii. Based on the observations, the internal state of the agent and the current subgoal, the low-level RL policy computes a control signal (action);

[0080] iii. The control signal (action) is applied to the actuators of the agent;

[0081] iv. Consequently, the agent / environment is in a new state and the reward for this transition is provided by the (subgoal-dependent) low-level reward function;

[0082] v. The low-level transition <state, action, new state, reward> is stored in the low-level transition buffer. If not completed in vi, the high-level transition <high-level state (feature observations), option (high-level action), new high-level state (feature observations)> is stored in the high-level transition buffer;

[0083] vi. If the subgoal high-level state is reached or the agent transitions to the wrong high-level state, the high-level planner provides a new subgoal high-level state based on the current high-level state and the value map as one of the neighboring high-level states. If not completed in v, the high-level transition <high-level state (feature observations), option (high-level action), new high-level state (feature observations)> is stored in the high-level transition buffer;

[0084] If the maximum number of interactions (time horizon) is reached and a high-level transition has not yet been stored in v, a high-level transition < high-level state (feature observations), option (high-level action), new high-level state (feature observations)> is stored in the high-level transition buffer;

[0085] II. Using the selected reinforcement learning algorithm, update the low-level reinforcement learning policy using the data in the low-level transition buffer;

[0086] III. Update the transition probability model parameters using the data in the high-level transition buffer. In the case of the specific neural network architecture of Figure 2 the network parameters are iteratively updated by an optimizer that maximizes the predicted probability of the actual new high-level state given the high-level state (feature observations) and option (high-level action) as network input.

[0087] Deployment

[0088] Once the hierarchical policy is trained, the controller 105 performs the following during deployment:

[0089] I. Provide / sample a specific (simulated) environment as well as a starting state and a desired target state of the agent;

[0090] II. Given a coarse representation of the environment, the desired target state of the agent, and the current transition probability model, run a value-iteration planner that generates a value map for high-level states;

[0091] III. Starting from the starting state and providing an initial subgoal high-level state from the high-level planner that is determined based on the initial agent state and the value map as one of the neighboring high-level states, the agent interacts with the (simulated) environment until the target state is reached or the maximum number of interactions (time horizon) is reached. In each interaction step:

[0092] a. The agent receives some observations (potentially pre-processed sensory information);

[0093] b. Based on the observations, the internal state of the agent, and the current subgoal, the low-level RL policy computes a control signal (action);

[0094] c. The control signal (action) is applied to the actuators of the agent;

[0095] d. This is repeated until the agent / environment is in a new high-level state;

[0096] e. If the subgoal high-level state is reached or the agent has transitioned to the wrong high-level state, the high-level planner provides a new subgoal high-level state based on the current high-level state and the value map as one of the neighboring high-level states.

[0097] In summary, according to various embodiments, as Figure 3 A method for controlling a robot is provided, as illustrated in the flowchart 300.

[0098] Figure 3 A flowchart 300 illustrating a method for controlling a robot is shown.

[0099] In 301, an indication of a target configuration to be reached from an initial configuration of the robot is received.

[0100] In 302, a coarse-scale value map is determined by value iteration, wherein the transition probabilities are determined using a transition probability model that maps coarse-scale states and coarse-scale actions to coarse-scale states.

[0101] In 303, for each coarse-scale state in a sequence of coarse-scale states of the robot, starting from an initial coarse-scale state determined from the initial configuration of the robot, and until the robot reaches the target configuration or has reached a maximum number of fine-scale states, a fine-scale subgoal is determined in 304 from the coarse-scale value map. The fine-scale subgoal can correspond to a coarse-scale action, e.g. can be a reference fine-scale state for the coarse-scale state of the coarse-scale action to be reached (e.g. the center of the coarse-scale state in fine-scale coordinates).

[0102] In 305, by actuators of the robot, fine-scale control actions to reach the determined fine-scale subgoal are performed, and sensor data is obtained to determine the fine-scale state reached as a result of performing the fine-scale control actions for each fine-scale state in a resulting sequence of fine-scale states of the robot, starting from a current fine-scale state of the robot (e.g. an initial fine-scale state of the current coarse-scale state), and until the robot reaches the determined fine-scale subgoal, the robot transitions to a different coarse-scale state, or has reached a maximum sequence length of the sequence of fine-scale states.

[0103] In 306, from the last fine-scale state of the determined sequence of fine-scale states, the next coarse-scale state of the sequence of coarse-scale states is determined.

[0104] According to various embodiments, in other words, the robot control is split into (at least) two levels, where the lower level performs fine-scale control given by a (fine-scale) control policy (trained via reinforcement learning - such as Q-learning, Deep Q-Networks (DQN), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic policy gradient (TD3), (Asynchronous) Advantage Actor Critic (A2C / A3C) - and the higher level - also referred to as planner - performs value iteration. Value iteration can be seen as an iterative application of the Bellman equation to generate a value map. The transition probabilities for value iteration are given by a transition probability model, which according to various embodiments can be trained by supervised learning, e.g. represented by a neural network. The parameters (e.g. weights) of the neural network, denoted by in the above example can be seen as parameters of the planner.

[0105] For example, using the learned (hierarchical) control policy, a control signal is determined based on the latent pre-processed sensory input. The high-level planner generates (subgoal) goals based on the provided coarse environment representation (e.g. map). Given these goals and the latent pre-processed sensory input, a low-level control module operating according to the low-level control policy computes the control signal (for performing the low-level control action).

[0106] In the case of a vehicle (e.g. car) or mobile robot, the coarse environment representation for the high-level planner is e.g. a tiled map (indicating obstacles), such that a finite number of cells is produced, which are potential subgoals. The goal vector (i.e. subgoal) is e.g. the difference between the center of the selected cell and the current vehicle position. A low-level control module operating according to the low-level control policy receives this goal vector together with the current latent pre-processed sensory reading.

[0107] In the case of a robot task, the high-level planner can receive some symbolic representation of the task (e.g. known subtasks to be fulfilled: install screw, move object...) as coarse environment representation. It then generates a plan, which orders these subtasks (or possibly some object position offset vectors) as goals (i.e. subgoals) for the low-level policy.

[0108] Figure 3The method and the method for training a robot controller to perform the method can be executed by one or more computers comprising one or more data processing units. The term “data processing unit” can be understood as any kind of entity that allows processing data or signals. For example, the data or signals can be processed in accordance with at least one, i.e. one or more than one, specific function performed by the data processing unit. The data processing unit can comprise or be formed by an analog circuit, a digital circuit, a complex signal circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA) integrated circuit, or any combination thereof. Any other means of implementing a respective function (which will be described in more detail below) can also be understood as a data processing unit or a logic circuit. It is to be understood that one or more method steps detailed herein can be performed, e.g. implemented, by a data processing unit by one or more specific functions performed by the data processing unit.

[0109] The term “robot” can be understood to refer to any physical system (with mechanical components whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0110] The sensor data, i.e. the observation data, can be data from almost any kind of sensor, such as video, radar, LiDAR (light detection and ranging), ultrasound, motion, thermal imaging camera, etc.

[0111] While specific embodiments have been shown and described herein, one of ordinary skill in the art will appreciate that many alternative and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein.

Claims

1. A method for controlling a robot (100), comprising: Receive an instruction to achieve the target configuration (103) from the initial configuration (102) of the robot (100); The coarse-scale value mapping is determined by value iteration, wherein a transition probability model (200) is used to determine the transition probability, which maps the coarse-scale state and coarse-scale action to the transition probability of the coarse-scale state. and For each coarse-scale state in the coarse-scale state sequence of the robot (100), starting from the initial coarse-scale state determined according to the initial configuration (102) of the robot (100), and until the robot (100) reaches the target configuration (103) or has reached the maximum number of fine-scale states, the fine-scale sub-objective is determined from the coarse-scale value mapping. The actuators of the robot (100) execute fine-scale control actions to achieve the determined fine-scale sub-goal, and obtain sensor data to determine the fine-scale state reached as a result of executing fine-scale control actions for each fine-scale state in the obtained fine-scale state sequence of the robot (100), starting from the current fine-scale state of the robot (100) and continuing until the robot (100) reaches the determined fine-scale sub-goal, the robot (100) transitions to a different coarse-scale state, or the maximum sequence length of the fine-scale state sequence has been reached; The next coarse-scale state in the coarse-scale state sequence is determined from the last fine-scale state in the fine-scale state sequence.

2. The method according to claim 1, wherein the transition probability model (200) is a model that can be trained through supervised learning.

3. The method according to claim 1 or 2, wherein the transition probability model (200) is a neural network.

4. A robot controller (105) configured to perform the method of any one of claims 1 to 3.

5. A method for training a robot controller (105) according to claim 4, comprising training a transition probability model (200) using supervised learning, and comprising training a fine-scale control policy via reinforcement learning, the reinforcement learning determining fine-scale control actions to achieve a sub-objective.

6. The method of claim 5, comprising training the transition probability model (200) by maximizing the probability of the transition probability model (200) to predict the coarse-scale state achieved by performing fine-scale actions to achieve a sub-objective.

7. A computer program product comprising a computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 3 or 5 to 6.

8. A computer-readable medium comprising a computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 3 or 5 to 6.

Images