Device and method for controlling a robot

Abstract

Method (500) for training a control strategy for a robot device (100), comprising the following: Acquiring (501) a reference state of an environment of the robot device (100) and a reference observation of the environment for the reference state; Generating (503) for each of several errors of an estimate of a posture of the robot device (100) of an observation that is perturbed with respect to the reference observation, according to the error of the posture estimate and a training data element that includes the generated observation as a training input; and Training (504) the control strategy using the generated training data elements.

Description

State of the art

[0001] The present disclosure relates to apparatus and methods for controlling a robot.

[0002] In recent years, there has been a growing need for automation on construction sites, which suffer from labor shortages for various reasons. First, automation can increase productivity, which has remained relatively stagnant over the past few decades, and reduce rising costs. Second, it can improve worker safety by allowing machines to perform high-risk tasks, thus removing workers from the danger. Third, it can help reduce the manual labor required by workers, thus helping to alleviate the labor shortage.

[0003] However, construction sites are inherently unpredictable and unstructured environments where multiple machines work simultaneously on various demanding tasks. Furthermore, construction projects are extremely diverse, as each project is tailored to a specific architectural design, specifications, etc. For these reasons, automation on construction sites is a challenging task.

[0004] Data collection for training, which forms the backbone of current machine learning methods for autonomous driving, proves extremely challenging in the unstructured environment of a construction site, where safety, time, and cost are the primary practical considerations. This problem can be partially solved using simulators, but these have their drawbacks. In addition, the unpredictable nature of the construction environment, where extreme and dangerous scenarios frequently occur, proves difficult to model and learn using standard procedures for autonomous vehicles.

[0005] Therefore, one object of the invention is to provide approaches for training a control strategy for a robot device in unstructured environments such as a construction site. This object is achieved according to the claims.

[0006] DE 10 2021103 272 A1 relates to devices, systems and techniques for facilitating robotic execution using neural networks to perform complex, multi-step tasks in situations for which a robot has not been trained.

[0007] DE 10 2020 214 177 A1 relates to a method for training a control strategy using reinforcement learning.

[0008] DE 10 2020 103 854 B4 relates to a method for determining a termination condition for a control program of a robot manipulator.

[0009] lDE 11 2020 004 135 T5 relates to a robot control device that controls a robot and: a machine learning model that is used to describe regions of environments.

[0010] DE 10 2019 131 385 B4 concerns a safety and performance stability of automation through uncertainty-driven learning and control. A control and learning module for controlling a robot arm includes at least one learning module that contains at least one neural network.

[0011] DE 11 2017 008 089 B4 relates to a device for monitoring a three-dimensional space, which has a coexistence space in which a first monitoring target and a second monitoring target are located. Disclosure of the invention

[0012] According to various embodiments, a method for training a control strategy for a robot device is provided, comprising acquiring a reference state of an environment of the robot device and a reference observation of the environment for the reference state, generating for each of a plurality of errors of an estimate of a posture of the robot device of an observation perturbed with respect to the reference observation according to the error of the posture estimate and a training data item comprising the generated observation as a training input, and training the control strategy using the generated training data items.

[0013] The method described above enables the training of a control strategy (or agent) of a robotic device such that the control strategy is robust against errors in pose estimation and can therefore achieve good results in a challenging (unstructured) environment, such as a construction site. It can be seen that the generated observations correspond to training poses, with each training pose differing from the correct pose by the respective error.

[0014] For example, one or more reference actions (generated for the reference observation, e.g., coming from expert knowledge, such as a demonstration) can be used as a target output for the generated observations, i.e., the observations that are incorrect in the sense that they correspond to incorrect pose estimates, as the actions to be performed, so that the robot device is trained to perform the "correct" actions even if it makes errors in its pose estimation. The target output of a training data item (i.e., the ground truth label) can also be generated by an expert, e.g., by an expert model acting as a "teacher."

[0015] The observation (e.g., a height map) is generated, for example, such that if the robot device's estimated pose was its true pose in a state of the environment as shown by the generated observation, it would have a position and orientation with respect to the state of the environment shown by the generated observation (e.g., relative to the piles of sand on a construction site) as its position and orientation (according to its reference pose) with respect to the reference state of the environment.

[0016] Various embodiments are described below.

[0017] Embodiment 1 is a method for training a control strategy for a robot device as described above.

[0018] Embodiment 2 is the method of Embodiment 1, wherein each of at least some of the errors is the error between a posture estimation result provided by a posture estimation function of the robot device provided in response to sensor measurement data and a reference posture, wherein the sensor measurement data is the sensor measurement data that the robot device would obtain when in the reference posture disturbed by respective noise.

[0019] Consequently, it can be achieved that the postures used for training are the postures corresponding to the posture estimates that the robot device can develop in practical application.

[0020] Embodiment 3 is the method of Embodiment 2, wherein the sensor measurement data includes the measurement data of an inertial measurement unit of the robot device and the image data from a camera that observes the environment, and wherein the posture estimation function performs sensor fusion to determine the posture estimation result.

[0021] The errors (and consequently the (training) postures) can then be generated by perturbing the inertial measurement unit's measurements and using the estimated postures resulting from sensor fusion as the training postures. This provides realistic training postures, because the inertial measurement unit's measurements, in particular, can be erroneous in practical applications.

[0022] Embodiment 4 is a method according to any one of Embodiments 1 to 3, comprising performing a pose estimation (e.g., from the training sensor measurement data) that outputs an uncertainty of the pose estimation result, and generating at least some of the errors by sampling from a distribution of the errors according to the uncertainty of the pose estimation result around the pose estimation result.

[0023] The pose estimation result can, for example, be the output of an (extended) Kalman filter (EKF). The extensions are then generated, for example, from the distribution of the EKF residual (e.g., a Gaussian distribution given by the pose estimation as a mean and the EKF covariance matrix estimation as a covariance). The pose estimation is, for example, the one implemented by the robot device. Consequently, training data items are generated that are realistically encountered during operation of the robot device.

[0024] Embodiment 5 is the method according to any one of embodiments 1 to 4, wherein the robot device is a construction vehicle and the environment is a construction site.

[0025] In particular, on construction sites, pose estimation is difficult due to the uneven ground and the very noisy environment, with sensors consequently providing noisy measurements, leading to noisy pose estimations. Consequently, training a control strategy robust to pose estimation errors is particularly advantageous in such an environment.

[0026] Embodiment 6 is a method for controlling a robot device, comprising training a control strategy according to any one of Embodiments 1 to 5, observing the environment to generate an observation, determining one or more actions from the observation using the trained control strategy, estimating a posture of the robot device by the robot device, and executing the one or more actions taking into account the estimated posture within the observed environment.

[0027] Embodiment 7 is a controller configured to perform a method according to any one of embodiments 1 to 6.

[0028] Embodiment 8 is a computer program comprising instructions that, when executed by a computer, cause the computer to perform a method according to any one of Embodiments 1 to 6.

[0029] Embodiment 9 is a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform a method according to any one of Embodiments 1 to 6.

[0030] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, in which: Fig. 1 a control scenario in a construction environment according to an embodiment; Fig. 2A the complete state information of an agent; Fig. 2B an observation that results from the true (correct) state after Fig. 2A; Fig. 2C selected actions for observation after Fig. 2B; Fig. 3 an extension of the strategy inputs for a training dataset; and Fig. 4 the training of an agent according to various embodiments.

[0031] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0032] Various examples are described in more detail below.

[0033] Fig. 1 shows a control scenario in a construction environment 101.

[0034] A robot 100 is located in this environment 101. The robot 100 has a starting position 102 and should, for example, clear the sand piles 103. The environment 101 may also contain the obstacles 104 that should be avoided by the robot 100. For example, they cannot be passed by the robot 100 (e.g., walls, trees, or rocks) or should be avoided because the robot would damage or injure them (e.g., workers).

[0035] The robot 100 has a controller 105 (which may also be located remotely from the robot 100, i.e., the robot 100 may be controlled by a remote control). It can be seen that the controller 105 implements an agent that controls the robot 100. The terms "controller" and "agent" are used synonymously below. In the exemplary scenario according to Fig. 1, the goal is for the controller 105 to control the robot 100 to navigate the environment 101 to clear the sand piles 103, i.e., to perform a bulldozing task. In this example, the robot 100 is an autonomous bulldozer, but it can also be a robot with legs or tracks or another type of propulsion system (such as a deep-sea or Mars rover).

[0036] The controller (or agent) 105 controls the robot 100 from the observations, ie, it receives an observation (e.g., a height map observed by a camera 105), selects one or more actions for the observation, and controls the robot 100 to perform the one or more actions (e.g., move a certain distance in a certain direction).

[0037] The following describes approaches to autonomous path planning for construction vehicles (generally robots in unstructured environments). As an example, Fig. 1, consider an autonomous grading task performed by a bulldozer 100 under localization uncertainty, where the estimated attitude of the vehicle (the bulldozer) is erroneous. This task presents several challenges common to all machine tools on any construction site. Therefore, the addressed task can be considered a representative example in the field. The main challenges include data collection, which is a central difficulty for all machine tools. Second, the partial observability of the environment is extremely challenging, as large portions of the environment are hidden due to sensor positioning.Third, the sensory noise, which translates into localization uncertainty, influences the way in which the agent 105 uses the observations of the environment 101 and consequently causes a significant performance degradation in the decision-making process.

[0038] To overcome the difficulty of data collection, according to various embodiments, domain adaptation techniques can be used to bridge the gap between simulation and reality when training the agent 105 in a simulated environment. The simulation can be extended to be as similar to real-world data as possible. According to various embodiments, a (control) strategy for an agent 105 is learned purely in the simulation (i.e., an agent is trained) and tested both in the simulation and in a scaled prototype environment.

[0039] Additionally, according to various embodiments, localization uncertainty is addressed using a training regime that considers the uncertainty (e.g., due to sensory noise) during training of the agent's strategy. This allows the agent 105 to learn a robust strategy with improved performance under uncertainty during inference compared to learning a strategy in a clean, noise-free environment.

[0040] Specifically, the training dataset used to train the agent 105 contains many variations, including scaled, rotated, and translated versions of (reference) observations, and consequently improves the agent's ability to cope with a more realistic scenario in which the observation is uncertain due to localization errors.

[0041] The control of the bulldozer 100 can be modeled as a partially observable Markov decision process (POMDP) ​​consisting of a tuple (S,O,A,P,R) exists: the condition s∈S contains all the information required to learn an optimal strategy. However, the agent 105 is often provided with only partial or noisy information regarding the environment 101, which is considered the observation o∈O In contrast to the states, the observations typically lack the statistics sufficient for optimality. In each state the agent 105 takes an action (ie, it controls the robot 100 according to a selected action) a∈A Then the system (the bulldozer 100 and the environment 101) goes to the next state based on the transition kernel P(s'|s, a) S′ Finally, the agent is provided with a reward r(s, a). The goal of an agent is to learn a behavioral strategy π that maximizes the cumulative reward, where the strategy maps the observations (or estimated states) to the actions.

[0042] According to various embodiments, the autonomous classification task is formalized as a POMDP / R (i.e., a POMDP without the reward term), i.e., a 4-tuple consisting of states, observations, actions, and the transition kernel is defined.

[0043] The state contains all the information necessary to obtain the optimal strategy and determines the outcome of each action. In this example, the state contains the exact attitude of the agent (i.e., the bulldozer) 100.

[0044] Fig. 2A illustrates the complete state information of the agent 200 according to one embodiment, including accurate (without error) knowledge of its posture and knowledge of the environment 202 including the location of the sand piles 201.

[0045] Fig. Figure 2B illustrates an observation that results from the true (correct) state after Fig. 2A has been derived without error, wherein the observation contains a portion of the state information (here a portion 204 of the environment 202).

[0046] In the context of errors, two aspects of actions can be considered: (i - control chain selection), where the strategy outputs a waypoint for the dozer 100 to reach. Here, posture estimation errors are represented as a suboptimal projection from the state to the observation. (ii - closed-loop control), where the posture estimation errors are fed back to the dozer's low-level controller for trajectory execution. Here, the errors propagate through the system, resulting in a deviation from the command path.

[0047] Fig. Figure 2C shows selected actions as points 205 (which define a path to be taken by the bulldozer 200 to clear the top sand pile 201).

[0048] The state transitions are determined by the dynamics of the bulldozer and the physical properties of the soil and the environment.

[0049] According to various embodiments, training observations (i.e., training inputs) are generated that include errors reflecting the uncertainty of the state estimation caused by measurement noise affecting the sensor data (in this example, a sensor measuring the attitude of the dozer). In the present example, the inaccurate state estimate (primarily attitude estimate) is translated into an erroneous bounding box view around the current location of the dozer 100.

[0050] For training (and testing) according to various embodiments, this behavior is mimicked in the simulation by applying an extension (e.g., rotation and translation) to the true and accurate observation, i.e., an observation corresponding to a (slightly) rotated and / or translated pose is generated.

[0051] Fig. Figure 3 illustrates the expansion of the strategy input for a training dataset.

[0052] A first graphical representation 301 illustrates a reference training data item in which the bulldozer 300 has a reference pose (i.e., a pose that in this example is assumed to be the true pose for the training data) in the environment 306. From this pose, the bulldozer has a specific observation (referred to as a reference observation) of the environment. The reference observation case therefore corresponds to the true state of the environment assumed for the training data item (which here specifically includes the location of the sand piles 307). The reference data item contains reference training labels, i.e., a specification of one or more (reference) actions (here two actions, represented by the points 308, in the form of waypoints defining a path to clear one of the sand piles 307, whereby the bulldozer should first drive to the more distant point and then reverse to the closer waypoint).

[0053] The further graphical representations 302, 303, 304, 305 each represent a training data element corresponding to an extended version of the reference observation. Each training data element corresponds to a perturbed pose (i.e., a specification of a pose that differs from the true pose of the graphical representation 301) and consequently to a perturbed observation (e.g., height map) that is perturbed with respect to the reference observation.

[0054] Each training data item contains as the training labels the reference training labels, that is, specified as the ground truth actions corresponding to the true state, so that the bulldozer is trained to perform an appropriate action even if its pose estimation is erroneous.

[0055] Fig. 4 illustrates the training of agent 105 according to various embodiments.

[0056] In this example, a perception block 406 is used to generate pose estimates as included in the bulldozer 100.

[0057] Specifically, the agent 105 comprises an inertial measurement unit 401 that measures speed and angular velocity increments (qte,qve) at high frequency.

[0058] An interface 402 to the camera 105 (located somewhere on the construction site) provides supporting position and attitude measurements (ie, orientation measurements) (pme,Ψme) at low frequency, i.e., it provides supporting information. The supporting information can also be provided by another device such as a GNSS (Global Navigation Satellite System) receiver.

[0059] An inertial navigation system (INS) 403 uses the inertial sensor measurements (from the IMU 401) and the initial conditions (IC) 404 to calculate the position, velocity, and orientation of the dozer. According to one embodiment, a sensor fusion approach using an extended Kalman filter (EKF) 405 is used to perform sensor fusion between the IMU 401 measurements and the camera measurements (received by the dozer via the camera interface 402).

[0060] The inertial integration system 403 therefore uses both the IC 404 and the measurements from the IMU 401 to provide position and attitude estimates at high frequency (p̃ INS , Ψ̃ INS), wherein, once a supporting measurement is available via the camera interface 402, the EKF 405 is activated to provide corrected pose, distortion, and drift estimates. These estimates are fed back to the inertial integration system 403 for inertial compensation. The output of the perception block 406, which includes these components, is a high-frequency pose estimate.

[0061] If the perception block represents a posture for a training state (i.e., a true state of a reference training data item, as shown in the first graphical representation 301 after Fig. 3), a corresponding training data item (as represented by one of the second to fifth graphical representations 302 to 305 according to Fig. 3). This is accomplished by feeding the estimated pose from the perception block 406 into a simulator to reproduce an observation 407 from the true state 408 and the estimated pose by perturbing the correct observation corresponding to the true state according to the estimated pose, i.e., generating an observation (height map) that would be observed by the camera 105 if the estimated pose of the bulldozer were its true pose. The training labels in the generated training data item are generated by giving the generated observation to an expert model.

[0062] Once the observation 407 is available, it is fed into the strategy that provides the actions 409 (i.e., the waypoint decisions). Then, a loss with respect to the training markers is calculated.

[0063] The simulator can then perform the provided actions, calculate the next true state (or set the simulation to another true state, e.g., by resetting after several iterations or by setting it to another training scenario in another way), and repeat the above process (i.e., performing multiple iterations) to calculate further losses (i.e., further components of a total training loss). The agent can then be trained to reduce the total training loss, e.g., the sum of the batch losses from multiple iterations.

[0064] It should be noted that from a single pose estimate provided by the perception block 406, multiple training data items can be generated by taking into account the uncertainty of the pose (from the EKF covariance matrix estimation), i.e., {x˜k}k=0K−1∼N(x˜,Σ)

[0065] Here K is the number of observations reproduced from the distribution over the estimated attitude X̃, while N(⋅,⋅) is the normal distribution. In the above example, X̃ (p̃ INS , Ψ̃ INS ), ie, posture and orientation.

[0066] In sensor fusion filtering, as implemented by the perception block 401, many noisy (perturbed) observations (i.e., extensions of the true observations) can therefore be generated. This can be done in particular by (i) adding synthetic noise to the inertial sensors of the inertial measurement unit (IMU) 401 and to the supporting sensor measurements (here, the image data received via the interface 402), (ii) applying the inertial navigation system 403 (INS) and the EKF 405, and (iii) reproducing the noisy observations from the distribution generated by the filter 405. In this way, uncertainties are introduced into the training pipeline. This improves training by resulting in a training dataset that now contains a much broader distribution across potential states.This in turn enables agent 105 to learn a strategy that is more robust to localization uncertainties.

[0067] In practice, the insertion of sensory noise into the measurements translates into small perturbations from the respective true observation.

[0068] Therefore, if the bulldozer estimates its attitude in global coordinates (i.e., coordinates of the height map) and makes an error in the attitude estimation (due to errors in the sensor measurements), this can be seen as a corresponding shift or rotation of the height map, as in Fig. 3 (because from the bulldozer's point of view, the environment is shifted and / or rotated, because it misjudges its attitude within the environment and therefore assumes that the environment is shifted and / or rotated, while it is itself that is shifted and / or rotated).

[0069] In summary, according to various embodiments, a method is provided which Fig. 5 is illustrated.

[0070] Fig. 5 shows a flowchart 500 illustrating a method for training a control strategy for a robotic device.

[0071] In 501, a reference state of an environment of the robot device and a reference observation of the environment for the reference state (ie, reflecting the reference state) are acquired.

[0072] In 502, one or more reference actions for the state are determined.

[0073] In 503, for each of a plurality of errors of an estimation of a pose of the robot device, an observation is generated that is perturbed with respect to the reference observation according to the error of the pose estimation, and a training data item is generated that includes the generated observation as a training input and the one or more reference actions as a target output.

[0074] In 504, the control strategy is trained using the generated training data items.

[0075] In other words, according to various embodiments, an agent is trained (or retrained) with an expanded dataset. Specifically, the agent is presented with additional training data items (from a distribution of training data items) that it has not encountered so far in training, for example. The additional training data items improve the scaling and rotation invariance of the control strategy implemented by the agent. As described above, for example, the location of the agent (the bulldozer) relative to the sand piles is perturbed to enable it to learn to make better inference decisions.

[0076] As explained above, the training data items (i.e., the training observations, i.e., the training inputs they contain) for different poses can be replayed from the "true" state (corresponding to the correct observation for which the action labels are given). Providing the agent with multiple observations for the same ("true") state derived from many poses improves the agent's robustness to scaling and rotation.

[0077] The approach according to Fig.5 can be used to train an agent to calculate a control signal for controlling a technical system that takes posture into account, such as a computer-controlled machine such as a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. According to various embodiments, a strategy for controlling the technical system can be learned and then the technical system can be operated accordingly.

[0078] Various embodiments may receive and use image data (i.e., digital images) from various visual sensors (cameras), such as video, radar, LiDAR, ultrasound, thermography, motion, sonar, etc., for example, to obtain the observations.

[0079] According to one embodiment, the method is computer-implemented.

Claims

[1] A method (500) for training a control strategy for a robot device (100), comprising: detecting (501) a reference state of an environment of the robot device (100) and a reference observation of the environment for the reference state; generating (503) for each of a plurality of errors of an estimate of a pose of the robot device (100) of an observation that is perturbed with respect to the reference observation, according to the error of the pose estimate and a training data item comprising the generated observation as a training input; and Training (504) the control strategy using the generated training data elements. [2] The method of claim 1, wherein each of at least some of the errors is the error between a pose estimation result provided by a pose estimation function of the robot device (100) provided in response to sensor measurement data and a reference pose, the sensor measurement data being the sensor measurement data that the robot device would obtain when in the reference pose disturbed by respective noise. [3] The method according to claim 2, wherein the sensor measurement data includes measurement data from an inertial measurement unit of the robot device and image data from a camera (105) observing the environment, and wherein the posture estimation function performs sensor fusion to determine the posture estimation result. [4] A method according to any one of claims 1 to 3, comprising performing a pose estimation outputting an uncertainty of the pose estimation result, and generating at least some of the errors by sampling from a distribution of the errors according to the uncertainty of the pose estimation result around the pose estimation result. [5] Method according to one of claims 1 to 4, wherein the robot device (100) is a construction vehicle and the environment is a construction site. [6] A method for controlling a robot device, comprising: Training a control strategy according to one of claims 1 to 5; Observing the environment to create an observation; Determining one or more actions from the observation using the trained control strategy; Estimating a posture of the robot device by the robot device; and Performing the one or more actions taking into account the estimated posture within the observed environment. [7] Controller configured to perform a method according to any one of claims 1 to 6. [8] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 6. [9] A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of claims 1 to 6.

Images