A computer-implemented method for deep reinforcement learning using analogous mapping

Abstract

Deep reinforcement learning of a Markov decision model 24 of a mobile agent 10, e.g. a robot that is able to move through a training environment is performed by, in part, obtaining trajectory data of

Description

TECHNICAL FIELD The invention relates to a computer-implemented method for deep reinforcement learning of a Markov decision model of a mobile agent. BACKGROUND Reinforcement Learning (RL) is a powerful tool for learning ideal control policies in various fields, including robotics, gaming, and autonomous systems. However, RL-based local motion planners may have difficulties generalizing their learned policies to new environments with different features, such as illumination, camera viewpoints, and background textures, from the training environments. This is typically associated with RL algorithms relying on raw inputs from the environment, making them highly sensitive to changes in input features. Consequently, the learned policies may not be suitable to new environments, which can cause to decreased performance. The main challenge for RL-based motion planners to be able to generalize to unseen environments with feature changes is developing methods that can effectively extract relevant information from raw high-dimensional inputs that are invariant to environments variations. Furthermore, such methods should be scalable, sample efficient, and computationally efficient, making them practical for real-world applications, in particular applications that are (real) time critical. Addressing these challenges can significantly expand the range of environments in which RL-based controllers can operate, enabling them to perform more complex tasks and facilitate applications in various fields. A common approach to help RL-based controllers to generalize is data augmentation, as described by I. Kostrikov et al.: “Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels”, available from https: / / arxiv.org / abs / 2004.13649, for example. This approach -'focuses on data augmentation techniques and regularization of the value function to reduce overfitting and improve generalization. Specifically, it applies image transformation to the input image observations to augment the training data, and regularize the Q-function to ensure similar Q-values for different image transformations. Another known approach is generating increasingly complex and diverse environments, as described by R. Wang et al.: “Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions”, available from https: / / arxiv.org / abs / 1901.01753, for example. This approach generates the learning environments with increasing complexity and diversity. It evolves the population of environments as automatic curriculum learning for RL and expands the diverse challenges and solutions. Another approach is learning invariant representations, as described by A. Zhang et al.: “Learning Invariant Representations for Reinforcement Learning without Reconstruction”, available from https: / / arxiv.org / abs / 2006.10742, for example. The method estimates behavioral similarity between states in Markov decision processes (MDPs). By approximating the behavioral similarity on representation space, the task-relevant information from observations can be encoded in state representations. This method is robust and invariant to the background distractors which is irrelevant to the learning task. CN 115 338 862 A and US 2019 / 0 025 851 A1 disclose a process to plan a robot or manipulator path using a (partially observable) Markov decision process. SUMMARY OF THE INVENTION It is the object of the invention to improve RL-based motion planners, preferably to improve their capability for generalization of environmental features. The object is achieved by the subject-matter of the independent claims. Preferred embodiments are subject-matter of the dependent claims. The invention provides a computer-implemented method for deep reinforcement learning of a Markov decision model of a mobile agent that is able to move through a training environment, the method comprising: a) gathering trajectory data that are indicative of a plurality of trajectories along which the mobile agent is able to move through the training environment, wherein each trajectory is defined by a sequence of trajectory data points that are defined by a state, an action and a reward; b) randomly selecting a trajectory data point pair of trajectory data points from the same trajectory but from different time steps and feeding that trajectory data point pair to a state embedding function and to an analogy mapping function, wherein the state embedding function determines an intra-trajectory embedding for each selected trajectory data point, wherein the intra-trajectory embedding has a reduced dimensionality compared to the selected trajectory data points, wherein the analogy mapping function determines one inter-trajectory embedding for the trajectory data point pair, wherein the inter-trajectory embedding has a reduced dimensionality compared to the trajectory data points; e) randomly selecting a pair of the previously selected trajectory data point pairs from different trajectories and feeding them to the analogy mapping function; f) feeding the intra-trajectory embeddings and the inter-trajectory embeddings to a policy network that is configured to determine a next action to be taken by the mobile agent; and g) determining a training loss that includes an analogy embedding loss and a state embedding loss, wherein the analogy embedding loss is based on an analogy embedding determined by the analogy mapping function and an expected reward determined by a reward estimator, wherein the state embedding loss is determined based on a state embedding determined by the state embedding function; h) updating learnable parameters of the state embedding function, the analogy mapping function and the policy network based on the previously determined training loss. A modification to the known methods is the introduction of an analogy mapping function. The analogy mapping function is trained towards capturing sequences of states that are analogous to each other, although they may differ in the details, such as illumination, camera viewpoints, and background textures. In other words, the analogy mapping function maps input states into an embedding space, in which despite the different details, the states can be recognized as substantially the same. As a result, differences between a training environment and a deployment environment can be better generalized and allow a more accurate decision making. Preferably, each state is defined based on robot information that is indicative of a robot property, e.g., robot position, robot orientation, robot velocity, robot angular velocity, and / or based on environmental information that is indicative of an area around the robot given by a sensor range, e.g., RGB images, RGB images with depth, lidar point cloud, costmap, road network and the like. The robot information allows to better define the intrinsic state of the mobile agent or the extrinsic state of the environment and their respective effects on each other. As a result, the decision making can be improved, since the next action is based on more relevant information. Preferably, wherein step a) includes taking sensor data that are indicative of the state and constructing the trajectory data based on the taken sensor data. Sensor data can give good information about the environment and the mobile agent. The sensor data can be chosen to represent the current state with a minimum of computational effort. Preferably, the state embedding function comprises a deep neural network or a deep convolutional neural network. Depending on the input data, deep neural networks or deep convolutional neural networks are better suited to determine the relevant embeddings. The computational effort is comparatively low and the system can be easily implemented. Preferably, the analogy mapping function comprises a deep neural network or a deep convolutional neural network. Similar to the state embedding function, an implementation involving deep neural networks can be computationally efficient. Preferably, the action is chosen from a set of actions that includes or consists of a change of velocity and a change of angular cost map velocity. A simple action space to choose actions from allows a more efficient implementation and accurate results. Preferably, the reward is chosen from a set of rewards that includes or consists of a positive reward for reaching a goal, a negative reward for colliding with an object in the environment, and a neutral reward otherwise. With this implementation of rewards, only severe issues are taught to be avoided. In other words, taking a longer, but potentially safer route is not at a disadvantage compared to a shorter but more dangerous route. In other words, the planned paths can be safer. The invention provides a Markov decision model that is obtainable by performing a previously described training method. The Markov decision model that is obtained with this training method is able to better generalize between different environments. As a result a more accurate and reliable decision making and path planning is possible. The invention provides a method for controlling an autonomous mobile robot in a deployment environment, the method comprising: a) at least one sensor device of the autonomous mobile robot detecting sensor data in an area around the autonomous mobile robot, wherein the sensor data are indicative of a state; b) a robot controller of the autonomous mobile robot determining robot information that is indicative of the current kinematic state of the autonomous mobile robot; c) feeding the sensor data and the current kinematic state to a path planner that includes a Markov decision model and, with the Markov decision model, determining a next action to be taken by the autonomous mobile robot; d) the robot controller generating a control signal for a drive system such that the control signal causes the drive system to move the autonomous mobile robot through the deployment environment in accordance with the control signal. Robots equipped with the disclosed Markov decision model can move through different environments with greater accuracy and reliability, as they are able to handle different details, e.g., illumination, camera viewpoints, etc. The invention provides a computer program comprising instructions that, when executed by a computer cause the computer to perform a previously described method The invention provides a machine readable storage medium having stored thereon the computer program. The invention provides an autonomous mobile robot comprising at least one sensor device configured for detecting sensor data in an area around the autonomous mobile robot, wherein the sensor data are indicative of a state; a robot controller configured for determining robot information that is indicative of the current kinematic state of the autonomous mobile robot; a path planner that includes a Markov decision model that is configured for determining a next action to be taken by the autonomous mobile robot based on the sensor data and the kinematic state; a drive system configured to move the autonomous mobile robot in accordance with a control signal; wherein the robot controller is further configured for generating a control signal based on the next action and transmitting the control signal to the drive system to cause the drive system to move the autonomous mobile robot in accordance with the control signal. This invention can be applied for autonomous mobile robots. Such robots are seen as the future of transport of goods in logistics and road traffic. An example for such an autonomous robot is the Correrie Delivery Robot currently under development by Continental. During deployment, the robot may face the challenge of high-dimensional feature changes, for example, diverse background or object color or shape changing. This invention can improve robot performance when deploying the robot in deployment environments that are significantly different from the training environments. The idea is based on a generalization on the RL planner and involves learning the analogies with time-series trajectory data, formulate the analogies as an implicit distance / metric between states, measure the analogies in state representation level to improve generalization and improving data efficiency for accelerating the RL training process. Initially the navigation task is formulated as an MDP that is represented by states, actions, a probability and a reward. Each state is an input to the MDP and includes robot information, such as current position, current velocity, current goal and the like, as well as environment information, such as RGB camera images, lidar point clouds, costmaps, and the like. Each state is being sampled at fixed time intervals. The actions define an action space of the robot and typically involves changes to the robot information, such as desired velocity and desired angular velocity. The transition probability of the next state is determined by taking a particular action in the current state. This is usually unknown for the navigation task and determined by taking actions in the environment. The transition probability also depends on the robot dynamics and environment factors such as obstacles and terrain conditions. The reward is received after transiting from the current state to the next state and is in general positive for reaching a goal, negative for a particular failure, e.g., colliding with another robot or obstacle, and neutral otherwise. The actions are preferably determined by a policy that is preferably represented by a deep neural network. For each given input state, the policy network can output an action. The policy network is trained such that the expected sum cumulative reward defined for the MDP can be maximized. The policy network can collect data by taking a sequence of actions in the environment and the collected data is in a form of trajectories. Each trajectory represents a sequence of {state, action, reward} across the timesteps. The policy network is trained using well known actor-critic methods. An optional value function can be used as a critic during policy updates. The method employs an analogies learning unit that enables the generalization of the training environment to the deployment environment. Trajectory data is collected which represents a sequence of {state, action, reward} across the timesteps. Based on the trajectory data intra-trajectory pairwise data are constructed by selecting two states of the same trajectory but at different timesteps. A deep mapping function is trained such that it maps states to low-dimensional embeddings. A neural network-based analogies mapping is trained to map the states pair to a single low-dimensional embedding. The deep mapping function is optimized by approximating the behavioral metric for low-dimensional state embeddings. The analogies mapping is trained by measuring a well-defined metric which is computed according to state embeddings from the deep mapping function. Based on the previously determined intra-trajectory state pairs inter-trajectory pairwise data is constructed by sampling two data pairs and computing state-pair embeddings from the analogies mapping. A reward function is trained regarding a metric function between inter-trajectory data pairs. The RL policy takes the input of state embeddings from the deep mapping function and the analogies mapping function, which transform the original state to lowdimensional embeddings. The analogies component is learned jointly with deep reinforcement learning framework with suitable loss functions. The robot can be deployed in various environments, where the learned policy network can generalize features in unseen environment by computing the state embeddings with the deep mapping function and the analogies mapping function. The policy network predicts the action based on the given input of state embeddings. BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings that are listed below. Fig. 1 depicts an environment in which autonomous mobile robots are navigating; and Fig. 2 depicts a data flow through a Markov decision model according to the invention during training. DETAILED DESCRIPTION OF EMBODIMENT Fig. 1 illustrates an example of an environment that may be a training environment or a deployment environment. An mobile agent 10 is navigating through the environment. The mobile agent 10 may be configured as an autonomous mobile robot 12. The environment can be a logistics building, a production facility or even an entire city. The autonomous mobile robot 12 includes a plurality of sensor devices 14. Some of the sensor devices 14 may be configured to detect robot information, such as position, orientation, velocity, angular velocity, state of charge or the like. Typical sensor types used for detecting robot information include, but are not limited to, navigation sensors, rotational sensors, voltage and current sensors, temperature sensors, accelerometers, etc. In general robot information is indicative of a property that is internal to the autonomous mobile robot 12 and can describe a robot state. Some sensor devices 14 may have a sensor range 16 in which the autonomous mobile robot 12 is able to detect environmental information about the environment surrounding the autonomous mobile robot 12. It should be noted that, while the sensor range 16 is illustrated as being 360° it is possible to have other types of sensor coverage that need not be complete. Typical sensors types used for detecting environmental information include, but are not limited to, RGB cameras, IR cameras, lidar devices, ultrasonic sensors. The environmental information may be indicative of objects 18 that are also present in the environment. The objects 18 may be stationary or mobile and may represent other traffic participants, e.g., vehicles or pedestrians, or obstacles. In general the environmental information is indicative of properties of the environment that are external to the autonomous mobile robot 12 and can describe an environmental state. In addition to sensor data, the environmental information may in particular include costmaps (for example travel time, fuel consumption and the like), road network information, such as road links. These non-sensor type data can be transmitted to the autonomous mobile robot 12 via a telecommunication network to keep certain information current, e.g., by updating the travel time costmap in case of congestion and the like. The autonomous mobile robot 12 includes a robot controller 20 and a path planner 22. The robot controller 20 receives the sensor data from the sensor devices 14 and passes the sensor data on to the path planner 22. The path planner 22 includes a Markov decision model 24 that is trained to generate a next action at to be taken by the autonomous mobile robot 12 based on both the robot state and the environmental state, collectively designated as the state st. The next action at advances the current state st to the next state st+i. The next action at typically includes a change to the robot state, in particular to the velocity and / or angular velocity to influence the movement path taken by the autonomous mobile robot 12 through the environment. The next action at is then fed to the robot controller 20 which generates a control signal in accordance with the next action at. The autonomous mobile robot 12 includes a drive system 26 that moves the autonomous mobile robot 12 through the environment according to the control signal previously generated by the robot controller 20. Due to the Markov decision model 24, the autonomous mobile robot 12 follows a trajectory 28 towards a goal position 30 depending on the states s encountered and the actions a taken. Each trajectory 28 is defined by trajectory data that includes a sequence of trajectory data points. Each trajectory data point for a time step t and a trajectory i is defined by the state s't, the action a't, and a reward r't. Referring to Fig. 2, a method for deep reinforcement learning of the Markov decision model 24 is described in more detail. The components that are depicted with hatching are trained, i.e., have their parameters updated. The process of deep reinforcement learning is well-known in the art and will only be described in detail insofar as it is relevant to understand the disclosed invention. As a general overview, the Markov decision model 24 is trained using a deep reinforcement learning method in the form of an actor-critic method. It should be noted that other deep reinforcement learning methods are also able to train the Markov decision model 24 such that it is usable in deployment with satisfactory results. In other words, the results that can be achieved by the Markov decision model 24 are more or less independent of the reinforcement learning method used for training. The Markov decision model 24 includes a sensor data input 32. The sensor data input 32 receives the sensor data that is gathered by the sensor devices 14. The sensor data input 32 thereby compiles the data that define the current state st. The Markov decision model 24 includes a trajectory data construction unit 34. The trajectory data construction unit 34 constructs at least a first trajectory t' and a second trajectory tj based on the sensor data and the forward simulation of the movement of the mobile agent 10. The Markov decision model 24 includes an intra-trajectory selector 36 that randomly selects a trajectory data point pair from the same trajectory 28, say the first trajectory t', at different time steps t and t+N. The trajectory data point pair can be designated as (t't, t't+N) where it should be kept in mind that each trajectory data point includes the state s't, the action a't and the reward 6 and state s't+N, the action a't+N and the reward r't+N, respectively. The intra-trajectory selector 36 does this once per time step for each trajectory 28. The Markov decision model 24 includes an inter-trajectory selector 38 that randomly selects a pair of trajectory data point pairs (t't, t't+N) and (tt, tW) from different trajectories 28, say the first trajectory t' and the second trajectory fl, and different time steps t and t+N and t and t+M, respectively. The inter-trajectory selector 38 selects the pair of trajectory data point pairs from the trajectory data point pairs that were previously selected by the intra-trajectory selector 36. In other words, the inter-trajectory selector 38 builds on the selection results from the intra-trajectory selector 36. The Markov decision model 24 includes a state embedding function 40. The state embedding function 40 is fed the trajectory data point pair (t't, t't+N), specifically the states (s't, s't+N), determined by the intra-trajectory selector 36. The state embedding function 40 receives the high-dimensionality input, e.g., RGB images, RGB images with depth dimension, 2D or 3D LIDAR point clouds. The state embedding function 40 determines from the high-dimensional input a low-dimensional state embedding 0(s't) for each state. High-dimensional can be typically upwards of 64,000 dimensions, e.g. for a grey scale image 1x320x200 = 64,000 or significantly more for RGB with depth, where typically 4x320x200 = 256,000 dimensions are present. Low-dimensionality is typically from 100 to 2,000 dimensions. The state embedding function 40 typically reduces the number of dimensions by a factor of from 10 to 100. The state embedding function 40 is implemented as a neural network that is adapted to the type of input data. For images, RGB images and RGB images with depth, the state embedding function 40 is typically configured as a convolutional neural network. The convolutional neural network may include typically 16 to 512 hidden nodes and one layer. For 2D LIDAR point clouds the state embedding function 40 can be implemented as a 1D convolutional neural network, with a similar structure as the convolutional neural network. The state embedding function 40 for 3D LIDAR point clouds is preferably chosen as VoxelNet, which is disclosed in https: / / arxiv.org / abs / 1711.06396 and available from https: / / www.voxelnet.com / . The state embedding function 40 may also be implemented as a fully connected neural network or a recurrent network, such as RNN or LSTM. The fully connected neural network typically involves 16 to 512 hidden nodes organized in 1 to 4 layers. The recurrent network typically involves 16 to 512 hidden nodes organized in 1 to 4 layers. The Markov decision model 24 includes an analogy mapping function 42. The analogy mapping function 42 is fed the pair of trajectory data point pairs (t't, t't+N), (Vt, tJt+M), specifically the states (s't, s't+N) and (sjt, sU), determined by the inter-trajectory selector 38. The analogy mapping function 42 receives these states as high-dimensionality input and determines a single low-dimensional analogy embedding qj(s't, s't+N) for each trajectory data point pair, specifically state pair. The analogy mapping function 42 is typically implemented as the same type as the state embedding function 40, with the difference that the state embedding function 40 processes trajectory data point pairs, specifically state pairs, whereas the analogy mapping function 42 processes pairs of trajectory data point pairs, specifically pairs of state pairs. The overall structure of the analogy mapping function 42 is similar to the state embedding function 40 with regards to number of layers and hidden nodes. The Markov decision model 24 includes a policy network 44. The policy network 44 is fed the state embeddings 0(s't) determined by the state embedding function 40 and the analogy embedding ip(s't, s't+N) determined by the analogy mapping function 42. The policy network 44 determines the next action at that is to be taken by the mobile agent 10. The policy network 44 can be implemented as a fully connected neural network and may include 16 to 512 hidden nodes organized in 1 to 4 layers. The output of the policy network 44, i.e. the next action at, involves a change to the velocity The Markov decision model 24 includes a value function network 46. The value function network 46 is fed the state embeddings 4>(s't) determined by the state embedding function 40 and the analogy embedding ip(s't, s't+N) determined by the analogy embedding function. The value function network 46 estimates the value of the current state st in which the mobile agent 10 is currently in. The value is an estimate of the expected cumulative return that can be achieved starting from this state. The value function network 46 can be implemented as a fully connected feed forward neural network that includes 16 to 256 nodes organized in 1 to 4 layers. The output is a single number, i.e. the value of the input state st. The Markov decision training model 24 is trained using deep reinforcement learning in an actor-critic process, wherein the policy network 44 functions as the actor and the value function network 46 functions as a critic. The different components of the Markov decision model 24 are jointly trained, i.e. the respective loss values of the respective loss functions are added. The total loss function is given by: Ltotal =   + where is the state embedding function 40 loss and is the analogy mapping function loss. The state embedding function 40 is updated based on the following state embedding loss function: h (4- ) = || 2 (¢(4)- 0 (4)) - d (4,4) ||, where 0 designates the state embedding function 40, slt and 4 are states at time step t from trajectories i and j, say the first trajectory and the second trajectory, d is a metric, e.g., the Euclidean or L2 metric, and d is a behavioral metric, e.g. the bisimulation metric or the metric by J. Chen and S. J. Pan available from https: / / arxiv.org / abs / 2006.10742. The state embedding loss function causes states that are similar to cluster together in embedding space, i.e. to be mapped closer together than wildly different states. The state embedding loss function is not limited to a particular time step or different trajectories. During training the pairs of states are sampled from different trajectories and time steps from a replay buffer. . Please confirm, whether this is correct. The analogy mapping function 42 is updated based on the following analogy mapping loss function: Liintra T ^interi where ip designates the analogy mapping function 42, Lintra is the intra-trajectory loss function, and Linter is the inter-trajectory loss function. The intra-trajectory loss function is given by: Lintra^t.S^ = lid (¢(4, 4+jv),       4+ / v)) - d where ip designates the analogy mapping function 42, and are states of trajectory i at time steps t and t+N, respectively, d is a metric, e.g., the Euclidean or L2 metric, d is a behavioral metric as previously discussed, and 0 designates the state embedding function 40. The intra-trajectory loss function causes the analogy mapping function 42 to learn the embedding for sequential behaviour, in particular the sequential analogy of the states. The expression d (0(4-4+^)-V'fct+iv-4+n)) gives the distance between 4 to in terms of the analogy mapping function 42. The result of the expression d is learned from the state embedding loss function. As a result, the embedding for the sequential behaviour is approximated towards the state embedding. The inter-trajectory loss function is given by: where 4-4+n-4-4+m are states of trajectory i and j at time steps t, t+N and t+M, respectively, d is a metric, such as the Euclidean or L2 metric, r is a reward estimating function, ip designates the analogy mapping function 42, ^ktNtrlk is the sum cumulative reward of trajectory i from time step t to time step t+N. The expression d [r (^p(sP si+N)j ,r (ip^ measures the learned reward differences between two rollouts (si,...,4+w) and (4, ...,4+M) in terms of the analogy mapping function 42. The expression computes the actual reward differences between two rollouts. Llnter aims to learn the analogy mapping function to capture the accumulated reward signals in the rollout, where rewards are the critic to the RL objective. The reward estimating function r can be implemented with a well-known approach, It can be implemented by parameterized neural network, e.g., a multi-layer perceptron (MLP), and trained jointly with loss function Linter. The configuration can be 16 to 512 hidden nodes organized in 1 to 4 layers. It can also be implemented by nonparameterized linear or non-linear function, e.g., the Euclidean or L2 metric. The Markov decision model 24 is trained with a learning rate of 10~5 to 10 3, a batch size of 4 to 512, a discount factor of 0.95 to 0.99 and a total number of training steps from 0.2 million to 10 million. The training can be done in an artificial environment, i.e. a simulation, that includes the relevant features. A suitable training environment is httpg; / / rnuk)w^ which allows physics simulations. It should be noted that, while a training using a simulated environment is possible, the ideas presented herein can also be applied to a real training environment. During the deployment phase, the mobile agent 10 detects the current state st using its sensor devices 14. The current state st is fed to the state embedding function 40 and the analogy mapping function 42. The state embedding function 40 receives the current state st as an input, whereas the analogy mapping function 42 receives the current state st as the state pair (st, st). The output of the state and analogy mapping functions 40, 42, i.e. the respective state embedding and analogy embedding, are fed to the policy network 44 which determines the next action at to be taken by the mobile agent 10. Based on the next action at, the robot controller 20 generates a control signal that causes the drive system 26 of the mobile agent 10 to execute the action at. As a result, the mobile agent 10 follows a path determined by the path planner 22. REFERENCE SIGNS mobile agent autonomous mobile robot sensor devices sensor range objects robot controller path planner Markov decision model drive system trajectory goal position sensor data input trajectory data construction unit intra-trajectory selector inter-trajectory selector state embedding function analogy mapping function policy network value function network

Claims

1. A computer-implemented method for training a Markov decision model (24) of a mobile agent (10), the method comprising:a) gathering trajectory data that are indicative of a plurality of trajectories (28) along which the mobile agent (10) is able to move through a training environment, wherein each trajectory (28) is defined by a sequence of trajectory data points that are defined by a state, an action and a reward;b) randomly selecting a trajectory data point pair of trajectory data points from the same trajectory (28) but from different time steps and feeding that trajectory data point pair to a state embedding function (40) and to an analogy mapping function (42), wherein the state embedding function (40) determines an intra-trajectory embedding for each selected trajectory data point, wherein the intra-trajectory embedding has a reduced dimensionality compared to the selected trajectory data points, wherein the analogy mapping function (42) determines one inter-trajectory embedding for the trajectory data point pair, wherein the inter-trajectory embedding has a reduced dimensionality compared to the trajectory data points;e) randomly selecting a pair of the previously selected trajectory data point pairs from different trajectories (28) and feeding them to the analogy mapping function (42);f) feeding the intra-trajectory embeddings and the inter-trajectory embeddings to a policy network that is configured to determine a next action to be taken by the mobile agent (10);g) determining a training loss that includes an analogy embedding loss and a state embedding loss, wherein the analogy embedding loss is based on an analogy embedding determined by the analogy mapping function (42) and an expected reward determined by a reward estimator, wherein the state embedding loss is determined based on a state embedding determined by the state embedding function (40); andh) updating learnable parameters of the state embedding function (40), the analogy mapping function (42) and the policy network (44) based on the previously determined training loss.

2. The method according to claim 1, wherein each state is defined based on robot information that is indicative of a robot property and based on environmentalinformation that is indicative of an area around the robot given by a sensor range (16).

3. The method according to any of the preceding claims, wherein step a) includes taking sensor data that are indicative of the state and constructing the trajectory data based on the taken sensor data.

4. The method according to any of the preceding claims, wherein the state embedding function (40) comprises a deep neural network or a deep convolutional neural network.

5. The method according to any of the preceding claims, wherein the analogy mapping function (42) comprises a deep neural network or a deep convolutional neural network.

6. The method according to any of the preceding claims, wherein the action is chosen from a set of actions that includes or consists of a change of velocity and a change of angular velocity.

7. The method according to any of the preceding claims, wherein the reward is chosen from a set of rewards that includes or consists of a positive reward for reaching a goal, a negative reward for colliding with an object (18) in the environment, and a neutral reward otherwise.

8. A Markov decision model (24) that is obtainable by performing a method according to any of the preceding claims.

9. A method for controlling an autonomous mobile robot (12) in a deployment environment, the method comprising:a) at least one sensor device (14) of the autonomous mobile robot (12) detecting sensor data in an area around the autonomous mobile robot (12), wherein the sensor data are indicative of a state;b) a robot controller (20) of the autonomous mobile robot (12) determining robot information that is indicative of the current kinematic state of the autonomous mobile robot (12);c) feeding the sensor data and the current kinematic state to a path planner (22) that includes a Markov decision model (24) according to claim 8 and, with the Markov decision model (24), determining a next action to be taken by the autonomous mobile robot (12);d) the robot controller (20) generating a control signal for a drive system (26) such that the control signal causes the drive system (26) to move the autonomous mobile robot (12) through the deployment environment in accordance with the control signal.

10. A computer program comprising instructions that, when executed by a computer cause the computer to perform a method according to any of the claims 1 to 7 or 9.

11. A machine readable storage medium having stored thereon a computer program according to claim 10.

12. An autonomous mobile robot (12) comprising at least one sensor device (14) configured for detecting sensor data in an area around the autonomous mobile robot (12), wherein the sensor data are indicative of a state; a robot controller (20) configured for determining robot information that is indicative of the current kinematic state of the autonomous mobile robot (12); a path planner (22) that includes a Markov decision model (24) according to claim 8 that is configured for determining a next action to be taken by the autonomous mobile robot (12) based on the sensor data and the kinematic state; a drive system (26) configured to move the autonomous mobile robot (12) in accordance with a control signal; wherein the robot controller (20) is further configured for generating a control signal based on the next action and transmitting the control signal to the drive system (26) to cause the drive system (26) to move the autonomous mobile robot (12) in accordance with the control signal.21

Images