Method for controlling a robot device

Abstract

Method for controlling a robot device (101), comprising: for each control vector from a plurality of control vectors, wherein each control vector defines a respective control sequence for a given task of manipulating an object (113) by the robot device (101), Controlling the robot device (101) to perform a sequence of actions, wherein the control vector specifies which action is to be performed for each observed control situation; Determining the values ​​of several target metrics that evaluate the execution of the specified task through the sequence of actions in terms of how well the robot device (101) achieves a goal of the specified task in the respective control situation by performing the sequence of actions, Adjusting a probability distribution of the control vectors such that the probability of control vectors for which the given task was performed with high scores and which have met at least one target condition across multiple target metrics is increased; Random selection, to perform the task in a current control scenario, of a control vector according to the probability distribution; and Controlling the robot device (101) to perform a sequence of actions, wherein the control vector specifies which action is to be performed for each observed control situation.

Description

[0001] The present disclosure relates to methods for controlling a robotic device.

[0002] A robotic device, such as a robot arm, that is supposed to retrieve objects from a container should be able to adapt to different situations, such as initial states. For example, if an object is located at the edge of a container instead of in the center, the robot should also be able to pick it up. However, this may require the robot to perform an additional action, such as first pushing the object to the center.

[0003] Therefore, approaches to controlling a robotic device are required that enable successful and efficient control for different situations.

[0004] Publication DE 10 2021 201 918 A1 describes a device and a method for controlling a robot, in which a labeled Markov decision process is determined for each skill of one or more skills, wherein each labeled Markov decision process has a labeling function that indicates, for each state of the labeled Markov decision process, whether one or more control conditions of a plurality of predetermined control conditions are met. A control task defined by a task description is provided, wherein the task description has a temporal sequence across a subset of the plurality of predetermined control conditions, and the robot is controlled to execute the control task such that the control conditions contained in the task description are met by states of the robot.

[0005] Publication DE 10 2021 107 568 A1 describes a method for adaptive access planning for robots for grasping containers. A robot is to grasp workpieces in a container using random positions and poses and place them in a target position and pose. The shape of the workpiece is analyzed to determine a plurality of robust access options, each access option having a position and orientation. The workpiece shape is also analyzed to determine a plurality of stable intermediate poses. Each individual workpiece in the container is assessed to determine a set of feasible accesses, with the workpiece being moved to the target pose if such a direct movement is possible.

[0006] The publication DE 10 2020 211 648 A1 describes an apparatus and a method for controlling a robot device, the method comprising performing a sequence of actions by the robot device using a robot control model, determining an updated strategy using the performed sequence of actions, projecting the updated strategy onto a projected strategy such that for each state of the plurality of states of the projected strategy, a similarity value according to a similarity metric between the projected strategy and the updated strategy is maximized and a similarity value according to the similarity metric between the projected strategy and the initial strategy is greater than a predefined threshold,adapting the robot control model to implement the projected strategy and controlling the robot device using the adapted robot control model.

[0007] Publication DE 10 2019 122 790 B4 describes a system comprising an articulated robot with a manipulator, one or more processors, and a memory storing executable instructions which, when executed by the one or more processors, cause the system to obtain at least one image from a camera, the camera being positioned so that the view of the camera is directed along the axis of the manipulator towards an object to be grasped, provide the image and a suggested action to a trained neural network to generate a Q value from a value function which is used to select an action most likely to produce a desired grasp, and move the manipulator based on the Q value to grasp the object with the manipulator when the Q value reaches a threshold.

[0008] The publication DE 11 2019 001 507 T5 describes techniques for controlling a robotic picking arm using estimated seal quality metrics.

[0009] Publication WO 2021 / 039 995 A1 describes a robot control device in which an environment and a situation in which control for activating a control routine for executing an operation to achieve an operational purpose is possible are described by an ML (machine learning) model.

[0010] The invention is based on the object of controlling a robot device in various control situations for manipulating an object in such a way that a predetermined control objective is achieved.

[0011] The object is achieved by a method having the features of claim 1, a robot device control device having the features of claim 5, a computer program having the features of claim 6 and a computer-readable medium having the features of claim 7.

[0012] According to various embodiments, a method for controlling a robot device is provided, which comprises, for each control vector from a plurality of control vectors, wherein each control vector defines a respective control sequence for a predetermined task of manipulating an object by the robot device, controlling the robot device to perform a sequence of actions, wherein the control vector indicates which action is to be performed for a respectively observed control situation, and determining the values ​​of a plurality of target metrics which evaluate the performance of the predetermined task by the sequence of actions in terms of how well the robot device achieves a goal of the predetermined task in the respective control situation by performing the sequence of actions.The method further comprises adapting a probability distribution of the control vectors such that the probability of control vectors for which the predetermined task was performed with high ratings and which have satisfied at least one target condition for a plurality of target metrics is increased, randomly selecting a control vector according to the probability distribution to perform the task in a current control scenario, and controlling the robot device to perform a sequence of actions, wherein the control vector indicates which action is to be performed for a respective observed control situation.

[0013] The method described above enables control of the robot device that adapts to the circumstances in which it is deployed. Each control vector can be viewed as representing a control strategy, and control vectors (or similar control vectors) that lead to successful control (in the sense of fulfilling at least one target condition and achieving good results with regard to the target metrics) are increasingly preferred over time, so that the control system increasingly adapts.

[0014] Various examples of implementation are given below.

[0015] Embodiment 1 is a method for controlling a robot device as described above.

[0016] Embodiment 2 is the method according to embodiment 1, wherein the control vector for a respectively observed control situation specifies parameters of the actions for at least some of the actions.

[0017] In this way, the machine learning model (represented by the fitted probability distribution) learns not only good sequences of actions (or decisions that, depending on the control scenario and control situation, lead to good sequences of actions in terms of the target metrics), but also how the actions should be parameterized (in a control situation that has been observed). This, in particular, increases the degrees of freedom in optimizing the control and thus enables a better optimization result.

[0018] Embodiment 3 is the method according to embodiment 1 or 2, wherein the task comprises picking up an object from a container.

[0019] Especially in such an application, the selection of actions performed is crucial for how well the task is performed, since, for example, a decision must be made as to whether an object can be grasped immediately or whether it must be moved first. By adapting the probability distribution based on successful control runs, the respective control device executing the procedure learns to adapt to different scenarios such as object types, grippers, and lighting and make appropriate decisions.

[0020] Embodiment 4 is the method according to one of the embodiments 1 to 3, wherein the adaptation of the probability distribution is carried out by means of a gradient-free optimization method.

[0021] An example of this is CMA-ES (Covariance Matrix Adaptation Evolution Strategy). This allows optimization with respect to multiple target metrics and taking into account the target constraints of the machine learning model's parameters (especially the probability distribution), since it does not require a single real-value evaluation (i.e., not a single real value of a loss function, for example), but can also be optimized with respect to a vector-valued objective (values ​​of multiple target metrics).

[0022] Embodiment 5 is a robot device control device configured to perform a method according to any one of embodiments 1 to 4.

[0023] Embodiment 6 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 4.

[0024] Embodiment 7 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 4.

[0025] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a robot. Fig. Figure 2 illustrates the control of a technical system by a control device. Fig. Figure 3 illustrates various sequences of primitives in the case of picking up an object from a container so that a barcode of the object can be inspected. Fig. 4 shows a flowchart illustrating a method for controlling a robot according to an embodiment.

[0026] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0027] Various examples are described in more detail below.

[0028] Fig. 1 shows a robot 100.

[0029] The robot 100 comprises a robot arm 101, for example, an industrial robot arm for handling or assembling a workpiece (or one or more other objects). The robot arm 101 comprises manipulators 102, 103, 104 and a base (or support) 105 by which the manipulators 102, 103, 104 are supported. The term "manipulator" refers to the movable elements of the robot arm 101, the actuation of which enables physical interaction with the environment, e.g., to perform a task. For control, the robot 100 comprises a (robot) controller 106 configured to implement the interaction with the environment according to a control program. The last element 104 (furthest from the support 105) of the manipulators 102, 103, 104 is also referred to as end effector 104 and may comprise one or more tools such as a welding torch, a gripping instrument, a painting tool or the like.

[0030] The other manipulators 102, 103 (closer to the support 105) can form a positioning device, so that, together with the end effector 104, the robot arm 101 is provided with the end effector 104 at its end. The robot arm 101 is a mechanical arm that can perform functions similar to a human arm (possibly with a tool at its end).

[0031] The robot arm 101 may include joint elements 107, 108, 109 that connect the manipulators 102, 103, 104 to each other and to the support 105. A joint element 107, 108, 109 may have one or more joints, each of which can provide rotational movement (i.e., rotary movement) and / or translational movement (i.e., displacement) of associated manipulators relative to each other. The movement of the manipulators 102, 103, 104 may be initiated by means of actuators controlled by the controller 106.

[0032] The term "actuator" can be understood as a component designed to influence a mechanism or process in response to being driven. The actuator can implement commands issued by the controller 106 (the so-called activation) into mechanical movements. The actuator, e.g., an electromechanical transducer, can be configured to convert electrical energy into mechanical energy in response to its activation.

[0033] The term "controller" can be understood as any type of logic implementing an entity, which may, for example, include a circuit and / or a processor capable of executing software stored in a storage medium, firmware, or a combination thereof, and which can issue commands, e.g., to an actuator in the present example. The controller may, for example, be configured by program code (e.g., software) to control the operation of a robotic device.

[0034] In the present example, the controller 106 comprises one or more processors 110 and a memory 111 that stores code and data, based on which the processor 110 controls the robot arm 101. According to various embodiments, the controller 106 controls the robot arm 101 based on a machine learning model 112 stored in the memory 111. For example, the robot 100 is intended to manipulate an object 113.

[0035] At any given time, the overall system comprising the robot arm 101 and the object 113 to be manipulated (or other objects) has a specific state with regard to position, orientation, end effector state (gripper open / closed), etc. This state of the system, robot, or object is referred to below as the control configuration.

[0036] A manipulation task performed by, for example, a robot arm 101 can typically be broken down into a sequence of skills. Such skills are also referred to as primitives or, more generally, as actions. Primitives can be viewed as the elementary skills of the respective robot device.

[0037] It is assumed that the robot 100 has a control function for each primitive that allows the controller 106 to control the robot arm to execute the primitive.

[0038] For example, for a primitive corresponding to a movement skill, a statistical model (e.g., a HSMM (Hidden Semi-Markov Model)) can be trained through learning from demonstrations (LfD). A model can also be learned using machine learning for a primitive such as determining a grasping pose for picking up an object. For simpler primitives, such as capturing an image of the object with a camera 114, such a control function can also be programmed directly without training a model.

[0039] Each primitive is a configurable function that has an effect on the controlled system (in the example of Fig. 1 robot 100 and object 103). A primitive can have one or more preconditions that must be met for the function to be executed.

[0040] In the following, it is assumed that the manipulation task is picking up an object 113, e.g., from a container 115 (so-called bin picking).

[0041] Examples of primitives related to such a manipulation task are • Recording the scene: ◯ Prerequisite: Camera is active ◯ Parameters: Camera mode (RGB or RGB-D, aperture settings, ...) • Image editing: ◯ Prerequisite: A scene recording is available ◯ Parameters: Modifications applied to the image and their internal parameters, e.g. filter size of erosion / dilation operations, changes to the image histogram • Simple gripping point calculation (fast, but does not work for complex shapes): ◯ A scene recording is available ◯ Error threshold, minimum size, ... • Advanced gripping point calculation (slower computationally, but works for complex shapes) ◯ Prerequisite: A scene recording is available ◯ Parameters: Maximum calculation time, modeled surface properties, error thresholds, ... • Make object tangible: Pushing an object out of a corner ◯ Precondition: Object is too far into a corner. ◯ Parameters: Corner of the container, way to approach the object. • Reorient the object (so that the object’s barcode is visible after picking it up): ◯ Prerequisite: The barcode is on the top of the object. ◯ Parameter: How the object should be reoriented so that the barcode is not on top.

[0042] A sequence of primitives for sorting multiple objects is then, for example: 1. Taking a picture of the scene (e.g. the working area of ​​the robot 100 with object 113) 2. Calculating a list of possible gripping points with ratings 3. Select the grasping point with a rating greater than a threshold that is closest to the camera 4. Checking the feasibility of gripping at the selected gripping point with the motion planner 5. Not feasible, continue with 3. 6. Feasible, continue with 7. 7. Grasping the object 8. Reading the object's barcode 9. Sorting the object 10. Continue with 1 (if not all objects are sorted yet)

[0043] Given a set of primitives, even for the same type of task (e.g., transporting objects from one container to another), there may be multiple sequences of primitives required for different scenes. For example, if the object is too close to a corner, it must be pushed out of the corner. Such sequences can be manually defined with fine-grained rules regarding which sequence to choose in which scenario.

[0044] For example, the above sequence can be hard-coded into the control device 106. However, for more complex manipulation tasks, the resulting control is often not optimal. According to various embodiments, a control method (and a corresponding control device) is therefore provided that optimizes the sequence of primitives and their parameters (particularly during runtime). This enables the controlled technical system (robot 100 in the bin-picking example) to adapt to different application cases, e.g., different object types, gripper types, etc., as well as different environmental conditions such as lighting (which affects primitives such as gripping point detection).

[0045] Such online optimization (optimization at runtime, i.e., during operation) can be realized using black-box optimization approaches. These include Bayesian optimization (BO) and CMA-ES (Covariance Matrix Adaptation Evolution Strategy). According to various embodiments, CMA-ES is used because it performs better with a large number of samples (e.g., results from previous control runs), which is an important aspect in the continuous optimization of parameters during operation.

[0046] Fig. 2 illustrates the control of a technical system 206 by a control device 201.

[0047] The control device 201 (e.g., control device 106) accesses a set of primitives 202 for which control functions are available (e.g., in the control device 106) according to an ML model 201 (e.g., ML model 112) trained for a target manipulation task. It selects a primitive 202 for each phase of the control in 203 and sets parameters for the selected primitive in 204. The control device 106 selects a primitive 202 only if all of its preconditions are met (if any). It can check this before selecting a primitive. In 205, the control device 106 applies the selected primitive, configured according to the set parameters, to the respective controlled system 206 (e.g., the robot arm 101 and, if applicable, other controlled elements, such as the camera 114). The controlled system 206 delivers sensor data to the control device (in particular to the ML model 112) in 207, e.g. image data or measurement data from a force sensor.

[0048] Each primitive 202 has an effect on the controlled system 206, e.g., it provides an image, the robot arm 101 moves, the object 113 is grasped, etc. This effect can be detected by the system's sensors (camera, robot sensor measurements, detection of a successful grasp, etc.) and is fed to the ML model 201 as feedback via the sensor data in 207. By collecting this feedback (i.e., the results a primitive has with certain parameter values) and training accordingly based on this feedback, the ML model 201 can learn the effect of primitives (with associated parameters) and, if trained in this way for a sufficiently long time, select the primitive and its parameter values ​​with the highest benefit for the task goal in each control situation (which it recognizes based on the returned sensor data).

[0049] For example, the ML model can learn 201 • Avoid capturing an image of a scene when unnecessary: ​​Each sampling of the scene (e.g., by capturing one or more images) takes time. If a grasp is performed only in one corner of a container, other objects may not be affected, and there is no need to resample the scene. The ML model 201 can learn under which circumstances resampling is necessary and avoid resampling in other cases. For example, a target metric that leads the ML model 201 to capture only one image when possible during training is the number of captured images, which should be as small as possible. • Select a gripping method: The controller may have multiple options (in particular, primitives 202) for calculating a grasping point (i.e., a location at which an object is grasped). The ML model 201 can learn which of these grasping methods is best applied in the current use case and the current scene (target metric for training: successful grasping, e.g., with as little force as possible) and select this method (and configure it if necessary). • Making objects graspable: In applications for picking up objects from a container, there are cases where an object is not graspable, for example, because the gripper cannot reach the object in a corner. Corresponding actions to make the object graspable, such as pushing the object out of the corner or shaking the container, can be implemented in the form of primitives 202. It is difficult to hard-code when a specific action to make the object graspable is successful. The ML model 201 can learn this based on feedback, i.e., results from previous controls (in particular, executions of such primitives) (target metric for training: successful grasping, e.g., with as few collisions with the container as possible).

[0050] According to various embodiments, a controller does not use a hard-coded sequence of primitives, but instead trains a model that decides which primitive should be executed next (and how it should be configured, i.e., how its parameters should be set). The model receives the current state of the controlled system (including the environment such as manipulated objects) as input (e.g., as one or more images, robot state data, meta-information regarding the task) and outputs the primitive to be executed next and its parameters. In this way, the model can adapt the sequence of primitives to the state of the controlled system and execute the task more efficiently.

[0051] Fig. 3 illustrates various sequences of primitives in the case of picking up an object 113 from a container 115 so that a barcode of the object 113 can be inspected.

[0052] Between a start state 301 and an end state 302, a picking up 303, a moving 304, a reorientation (including placing at the current location) 305, a scanning (of the barcode) 306 and a placing at the destination location 307 can be executed as primitives. • In the nominal case, object 113 is near the center of container 115 and its barcode is not on its upper surface. In this case, the sequence of primitives and their parameters is given by: Pick up object (parameter: object pose) → Scan barcode (parameter: scanner location) → Drop object (parameter: object destination) • Special case 1: The object 113 is close to an edge of the container 115 and the barcode is not on the upper surface. The sequence and selection of parameters are the same as in the nominal case, but the controller 105 uses different trajectories of the robot arm 101 (in particular, the gripper orientation). The selection of the appropriate parameters is done automatically based on the control function (e.g., the learned motion model, e.g., HSMMs) for the corresponding motion skill. • Special case 2: Object 113 is close to a corner of container 115 and the barcode is not on the top surface. The sequence and parameter selection are as follows: Move the object from the corner to the center of the container (parameter: object pose, center of the container) → Pick up object (parameter: object pose) → Scan barcode (parameter: scanner location) → Drop object (parameter: object destination) • Special case 3: Object 113 is near the center of container 115, but its barcode is not on the upper surface. Then, a reorientation is required compared to the nominal case: Pick up object (parameter: object pose) → Reorient object (parameter: object pose) → Pick up object (parameter: object pose) - → Scan barcode (parameter: scanner location) → Drop object (parameter: object destination) • Special case 4: The object 113 is close to an edge or corner of the container 115 and the barcode is not on the top surface. Then, compared to special case 2, a reorientation is required: Move the object from the corner or edge to the center of the container (parameter: object pose, center of the container) → Pick up object (parameter: object pose) → Reorient object (parameter: object pose) → Pick up object → Scan barcode (parameter: scanner location) → Drop object (parameter: object destination)

[0053] As described above, the ML model 201 receives feedback on configured primitives (i.e., primitives with associated parameter values), which enables it, in particular, to evaluate how well a primitive or a sequence of primitives is or was suited to achieving the respective goal. According to various embodiments, a configured primitive or a sequence of primitives is evaluated using a plurality of goal metrics.

[0054] For example, a sequence of configured primitives is executed, and as part of the feedback, the force exerted by the robot on object 113 is measured. A target metric could then be the measured force (which should be as low as possible), and a condition regarding this target metric could be that the force does not exceed a specified limit. Another target metric could be the deviation from the reached target location. This can also provide an evaluation (as close to the target as possible), and a condition can also be specified for this, e.g., the object may only be within a certain maximum distance from the target location.

[0055] For a control sequence, i.e., a sequence of (configured) primitives from start state to end state, the controller can determine values ​​of target metrics accordingly. Each target metric has an associated optimization objective (as low a force as possible, as close to the target as possible) and can have a condition (force must remain below a limit). For a control sequence, such a condition can be met or not.

[0056] Each sequence of primitives executed results from which primitives the ML model 201 selects for a particular situation (i.e., a state of the controlled system). The ML model 201 makes these decisions according to a vector of values ​​for a set of control parameters. This vector is also referred to as a control vector. The control vector specifies decision parameter values ​​for decision conditions, for example, that the object is moved if it is closer than x mm to the edge of the container 115. That is, the ML model 201 makes a particular decision based on the values ​​of the vector's bits specified for that decision.

[0057] Thus, the control vector defines a specific control sequence (ie a sequence of primitives with configuration parameters of the primitives) for an instance of the task, ie for the task in a specific control scenario (e.g. start configuration and target configuration), since it defines the decisions which primitives are selected (in response to feedback) and their configuration.

[0058] The ML model samples the control vector from a space of control vectors. According to various embodiments, the machine learning model 201 learns a probability distribution of the control vectors from control sequences (for which it receives ratings or determines from feedback), according to which it subsequently samples from the space of control vectors. This is done, for example, according to CMA-ES.

[0059] The probability distribution is adjusted over time (across multiple control processes) in such a way that control vectors that have led to control processes for which the target conditions were met and for which the target metrics have assumed good (i.e., high or low depending on the target metric) values ​​are sampled with a higher probability.

[0060] The probability distribution is adjusted iteratively, with a set of control vectors being evaluated and then sorted for each iteration. The sorting can take the target metrics into account (e.g., through a weighted combination (e.g., sum) of the target metrics, so that a value can be determined for each target vector and the control vectors can be sorted based on these values), as well as lower weights or exclude from the sorting control vectors that do not meet the target conditions (e.g., lie outside a permissible range for at least one target metric). Based on the sorting, a subset of the control vectors is then selected (e.g., the n best ones), and the probability distribution is adjusted in favor of these vectors.In the case of normally distributed sampling, this involves shifting the expected value of the distribution toward the mean of the vectors and adjusting the covariance matrix according to the variance. Furthermore, prior parameters of the probability distribution can be taken into account to ensure a more stable distribution fit.

[0061] In other words, the optimization of the target metrics is achieved by learning the probability distribution, through which control vectors that lead to good results (in terms of the target metrics) become increasingly likely over the course of operation.

[0062] This adaptation of the probability distribution can be performed during operation via control runs. After a certain number of control runs (or another criterion, e.g., reliable control over a certain number of control runs), training can be terminated, meaning the probability distribution can be left the same or adjusted less frequently.

[0063] Over time, the ML model increasingly selects control vectors that result in sequences of primitives that the machine learning model (based on its previous experience) expects to satisfy the constraints of the target metrics and to well meet the optimization goals of the target metrics.

[0064] In summary, according to various embodiments, a method is provided as described in Fig. 4 shown.

[0065] Fig. 4 shows a flowchart 400 illustrating a method for controlling a robot according to one embodiment.

[0066] In 401, for each control vector from a plurality of control vectors, • in 402, the robot device is controlled to perform a sequence of actions, wherein the control vector indicates which action is to be performed for a respective observed control situation; • in 403 the values ​​of several target metrics that evaluate the execution of a given task through the sequence of actions are determined.

[0067] In 403, a probability distribution of the control vectors is adjusted so that the probability of control vectors for which the given task was performed with high ratings and which have satisfied at least one target condition for several target metrics is increased.

[0068] In 404, to perform the task in a current control scenario, a control vector is randomly selected according to the probability distribution.

[0069] In 405, the robot device is controlled to perform a sequence of actions, wherein the control vector indicates which action is to be performed for a respective observed control situation.

[0070] The procedure of Fig. 4 may be performed by one or more computers having one or more data processing units. The term “data processing unit” may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be handled according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0071] The approach of Fig.4 is used to generate a control signal for a robotic device. The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule for the technical system is learned, and the technical system is then controlled accordingly.

[0072] Various embodiments may receive and use sensor signals from various sensors, such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc., for example, to obtain sensor data regarding demonstrations or states of the system (robot and object or objects), configurations, and scenarios. The sensor data may be processed. This may include classifying the sensor data or performing semantic segmentation on the sensor data, for example, to detect the presence of objects (in the environment in which the sensor data was obtained). Embodiments may be used to train a machine learning system and control a robot, e.g., autonomously from robot manipulators, to achieve various manipulation tasks under different scenarios. In particular, embodiments are applicable to controlling and monitoring the execution of manipulation tasks, e.g.,in assembly lines.

[0073] The robotic device is, for example, a robot arm that is controlled to pick up and, if necessary, inspect an object. In this case, the sensor data on the basis of which the robot arm is controlled comprises, for example, digital color images (RGB images) and depth images (possibly in combination, i.e., RGB+D images).

Claims

[1] A method for controlling a robot device (101), comprising: for each control vector from a plurality of control vectors, wherein each control vector defines a respective control sequence for a given task of manipulating an object (113) by the robot device (101), Controlling the robot device (101) to perform a sequence of actions, wherein the control vector indicates which action is to be performed for a respective observed control situation; Determining the values ​​of several target metrics that evaluate the execution of the specified task by the sequence of actions in terms of how well the robot device (101) achieves a goal of the specified task in the respective control situation by performing the sequence of actions, Adjusting a probability distribution of the control vectors such that the probability of control vectors for which the given task was performed with high scores and which satisfied at least one target condition for several target metrics is increased; Randomly selecting, to perform the task in a current control scenario, a control vector according to the probability distribution; and Controlling the robot device (101) to perform a sequence of actions, wherein the control vector indicates which action is to be performed for a respective observed control situation. [2] The method of claim 1, wherein the control vector for a respective observed control situation indicates parameters of the actions for at least some of the actions. [3] The method of claim 1 or 2, wherein the task comprises picking up the object (113) from a container (115). [4] Method according to one of claims 1 to 3, wherein the adaptation of the probability distribution is carried out by means of a gradient-free optimization method. [5] Robot device control device (106, 201) arranged to carry out a method according to one of claims 1 to 4. [6] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 4. [7] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 4.

Images