Robot device, computer-implemented method for training robot control model, and method for controlling robot device

Abstract

To provide a method for controlling a robot device.SOLUTION: First pickup prediction includes first success probability of allocating a first pick-up robot configuration vector to each image point of an image, the method further includes supplying the image to a second prediction model so as to generate second pick-up prediction, the second pick-up prediction includes second success probability of allocating a second pick-up robot configuration vector to each image point of an image, and the method further includes supplying the first pick-up prediction and the second pick-up prediction to a mixed model of a robot control model so as to generate third pick-up prediction.SELECTED DRAWING: Figure 2A

Description

Technical Field

[0001] Various embodiments generally relate to robotic devices, computer-implemented methods for training robot control models, and methods for controlling robotic devices.

Background Art

[0002] Robotic devices can be used, for example, in manufacturing, production, maintenance, housework, medical technology, etc. Here, the robotic device can, for example, pick up and move an object. Here, the robot control model can determine which object among a plurality of objects the robotic device should pick up, and further how the robotic device should pick up and move this object (for example, using one or more robot arms).

[0003] In this regard, various models for determining an object to be picked up and a related configuration of a robotic device based on an image showing a plurality of objects are known.

[0004] In the publication "Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics" (arXiv:1703.09312, 2017) by J. Mahler et al. (hereinafter referred to as Reference [1]), a model (Dexterity Network, Dex-Net) for determining a grip configuration of a robot arm to grasp one object among a plurality of objects based on a depth image showing the plurality of objects is described.

[0005] The publication "Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning" by A. Zeng et al. (arXiv:1803.09956, 2018) (hereinafter referred to as Reference [2]) describes a model for obtaining the grip configuration of a robotic arm and the associated success probabilities based on depth images showing multiple objects.

[0006] The publication "Robot Learning of Shifting Objects for Grasping in Cluttered Environments" by L. Berscheid et al. (arXiv:1907.11035, 2019) (hereinafter referred to as Reference [3]) describes a model for obtaining the grip configuration of a robotic arm based on depth images showing multiple objects.

[0007] The publication "Learning robust, real-time, reactive robotic grasping" by D. Morrison et al. (The International Journal of Robotics Research, 2020) (hereinafter referred to as Reference [4]) describes a model for obtaining the grip configuration of a robotic arm based on depth images showing multiple objects.

[0008] The publication "On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks" by V. Satish et al. (IEEE Robotics and Automation Letters, 2019) (hereinafter referred to as reference [5]) describes a model for determining the grip configuration of a robot arm using a convolutional neural network based on depth images showing multiple objects. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Publication “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics” by J.Mahler et al. (arXiv:1703.09312, 2017) [Non-Patent Document 2] Publication “Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning” by A. Zeng et al. (arXiv:1803.09956, 2018) [Non-Patent Document 3] Publication “Robot Learning of Shifting Objects for Grasping in Cluttered Environments” by L. Berscheid et al. (arXiv:1907.11035, 2019) [Non-Patent Document 4] Publication: "Learning robust, real-time, reactive robotic grasping," by D. Morrison et al. (The International Journal of Robotics Research, 2020) [Non-Patent Document 5] Publication “On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks” by V. Satish et al. (IEEE Robotics and Automation Letters, 2019) [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] However, the quality of individual models is related to various factors. These factors include, for example, the similarity between the input data and the training data, the suitability of each training method for the objects to be picked for various object classes, and the sensitivity to input noise and / or artifacts in the processed images. Each model provides the best results for the conditions assumed in the model. Each model is related to each hyperparameter search and training data preparation. For example, reference [4] is based on highly densely labeled ground truth images. For example, reference [5] is based on synthetic data generation that approximates real-world physics. In short, these models have advantages and disadvantages depending on the application. Since the images recorded during robot operation are also different from the training data and are unpredictable, it is not possible to predict which model will provide the best results. [Means for solving the problem]

[0011] This method (Examples 1 and 11) and the apparatus (Examples 14 and 15) can improve the successful pickup of objects using a robotic apparatus. Here, the probability of successfully picking up an object using a robotic apparatus (for example, without dropping it) is increased. Specifically, an improved robotic control model is provided.

[0012] Robotic devices may include all kinds of computer-controlled devices, such as robots (e.g., manufacturing robots, maintenance robots, household robots, medical robots, etc.), vehicles (e.g., autonomous vehicles), household appliances, production machinery, personal assistants, access control systems, etc.

[0013] A method having the features of independent claim 1 constitutes a first example. Specifically, a robot control model trained according to the first example selects the most reliable robot configuration from a plurality of predictive models. The robot control model described herein includes a weighting coefficient which weights the robot configurations predicted by various predictive models, thus taking into account the advantages and disadvantages of each predictive model.

[0014] The first weighting coefficient may include weighting coefficients for all first pickup robot configurations and weighting coefficients for all second pickup robot configurations. The features described in this paragraph, combined with the first example, constitute the second example. The weighting coefficients for all image points in the image can reduce the computational cost of adapting the first weighting coefficients.

[0015] The first weighting coefficient may include each weighting coefficient for each first pickup robot configuration vector and each weighting coefficient for each second pickup robot configuration vector. The features described in this paragraph, combined with the first example, constitute a third example. The first weighting matrix of image point modes can improve the accuracy and flexibility of the trained mixture model.

[0016] The second weighting coefficient may include a weighting coefficient for all predicted first success probabilities and a weighting coefficient for all predicted second success probabilities. The features described in this paragraph can be combined with one or more of the first to third examples to form a fourth example. A weighting coefficient for all image points in an image can reduce the computational cost of adapting the second weighting coefficient.

[0017] The second weighting coefficient may include each weighting coefficient for each predicted first success probability, and each weighting coefficient for each predicted second success probability. The features described in this paragraph can be combined with one or more of the first to third examples to form a fifth example. The second weighting matrix for the image point modes can improve the accuracy and flexibility of the trained mixture model.

[0018] The robot control model may include a first convolutional neural network with a first weighting coefficient and a second convolutional neural network with a second weighting coefficient. The generation of the third pickup prediction is as follows: In order to generate a third pickup robot configuration, the first pickup robot configuration and the second pickup robot configuration are supplied to the first convolutional neural network, In order to generate the predicted third success probability, the predicted first success probability and the predicted second success probability are supplied to the second convolutional neural network, This may include the following. The features described in this paragraph, when combined with the first example, constitute the sixth example.

[0019] Convolutional neural networks may have lower complexity compared to the same weighting coefficients and / or weighting matrices described above. This can improve the efficiency of the robot control model. Furthermore, convolutional neural networks are scalable, thereby increasing flexibility. In addition, parallelization of modern graphics processor (GPU) architectures can be utilized. This is done, for example, by parallelizing a first convolutional neural network with a second convolutional neural network. Specifically, this can reduce training time and therefore increase efficiency.

[0020] This method further, This may include generating a segmented image using image segmentation, where multiple image points in the segmented image are assigned to one or more objects. This may include supplying the generated segmentation images to a first weighting model of the robot control model in order to generate a first weighting coefficient (which optionally includes a weighting coefficient for each image point in the image), This may include supplying the generated segmentation images to a second weighting model of the robot control model in order to generate a second weighting coefficient (which optionally includes a weighting coefficient for each image point in the image), Training a robot control model using the adaptation of the first and second weighting coefficients involves the adaptation of the first and second weighting models. The features described in this paragraph are combined with one or more of the first to fifth examples to form a seventh example. Here, the first and second weighting models can use segmentation images to consider additional criteria for object pickup, such as areas unsuitable for object pickup, independently of the robot's task.

[0021] This method further, This may include determining the normal vector and associated standard deviation of each surface shown in the image. This may include supplying the obtained normal vector and associated standard deviation to a third weighting model of the robot control model in order to generate a first weighting coefficient (which optionally includes a weighting coefficient for each image point in the image), This may include supplying the obtained normal vector and associated standard deviation to a fourth weighting model of the robot control model in order to generate a second weighting coefficient (which optionally includes a weighting coefficient for each image point in the image), Training a robot control model using the adaptation of the first and second weighting coefficients includes the adaptation of the third and fourth weighting models. The features described in this paragraph are combined with one or more of the first through fifth examples to form an eighth example. Here, the first and second weighting models can consider additional criteria for object pickup, such as areas unsuitable for object pickup, independently of the task of the robot device, using normal vectors with associated standard deviations.

[0022] Each pickup robot configuration vector may be a tuple of values ​​describing the configuration of the robot device. At least one image point in the target data may be assigned one or more values ​​from a target tuple describing the target pickup robot configuration. The robot control model may be trained to generate a third pickup robot configuration vector for at least one image point, having a tuple of values ​​that have one or more values ​​from the target tuple (and optionally, to increase the predicted third success probability assigned to the third pickup robot configuration vector). The features described in this paragraph can be combined with one or more of the first to eighth examples to form a ninth example. Specifically, the robot control model can be trained with respect to the probability of a successful pickup, as well as to the configuration of the robot device that can successfully pick up the object.

[0023] In the target data, it is often assumed that a failed pickup is assigned to at least one other image point in the image, where the robot control model is trained to further reduce the predicted third success probability generated for at least one other image point using adaptations of the first and second weighting coefficients. The features described in this paragraph are combined with one or more of the first to ninth examples to form a tenth example. Specifically, the robot control model can learn not only which image points are suitable for picking up objects, but also which image points are suitable.

[0024] The method for controlling the robotic device is: This may include detecting images showing one or more objects, To generate a third pickup prediction, the image may be fed into a robot control model trained according to one of the examples from the first to the tenth example. This may include controlling the robotic device to pick up, according to the third pickup robot configuration vector, one or more objects, that are assigned to the image points of the third pickup robot configuration vector, with respect to the third pickup robot configuration vector of the third pickup prediction to which the predicted best third success probability is assigned. The method described in this paragraph constitutes an eleventh example.

[0025] A computer program may include instructions that cause the processor to perform a method according to one or more of the first to eleventh examples when executed by the processor. A computer program having the characteristics described in this paragraph constitutes the twelfth example.

[0026] Computer-readable media (e.g., computer program products, non-temporary storage media, non-volatile storage media) can store instructions that cause the processor to perform a method according to one or more of the first to eleventh examples when executed by the processor. Computer-readable media having the characteristics described in this paragraph constitute a thirteenth example.

[0027] The training apparatus may include a computer configured to carry out the method according to one or more of the first to tenth examples. A training apparatus having the features described in this paragraph constitutes a fourteenth example.

[0028] The robotic device is A robotic arm configured to pick up an object, Control device and The control device may include, The system is configured to obtain a third pickup prediction for an image showing one or more objects, using a robot control model trained according to any one of the examples from the first to the tenth example. The system is configured to control at least one robotic arm to perform the pickup of one or more objects, which are assigned to the image points of the third pickup robotic configuration vector, according to the third pickup robotic configuration vector, to which the best predicted third success probability is assigned. A robotic device having the features described in this paragraph constitutes a 15th example. Specifically, the robotic device can successfully pick up objects with an increased probability.

[0029] Embodiments of the present invention are shown in the drawings and will be described in more detail below. [Brief explanation of the drawing]

[0030] [Figure 1] This figure shows exemplary robotic apparatus assemblies according to various embodiments. [Figure 2A] This is a flowchart for training robot control models according to various embodiments. [Figure 2B] This is a flowchart for training robot control models according to various embodiments. [Figure 3A] This is a diagram illustrating an example image. [Figure 3B] This is a diagram illustrating an example of a requested pickup robot configuration map. [Figure 3C] This is a diagram illustrating the pickup success probability map obtained as an example. [Figure 4] This is a flowchart for controlling robotic devices according to various embodiments. [Figure 5]This figure shows an exemplary robotic device assembly having multiple robotic arms according to various embodiments. [Figure 6] This is a flowchart for computer-implemented training of robot control models according to various embodiments. [Modes for carrying out the invention]

[0031] In one embodiment, “computer” may be understood as any kind of logic implementation entity that can be hardware, software, firmware, or a combination thereof. Thus, in one embodiment, “computer” may be a hardwired logic circuit or a programmable logic circuit, such as a programmable processor, such as a microprocessor (e.g., CISC (a processor with a large instruction stock) or RISC (a processor with a reduced instruction stock)). “computer” may include one or more processors. “computer” may also be software implemented or run by a processor, such as any kind of computer program, such as a computer program that uses virtual machine code, such as Java. Each implementation of other aspects of each function described in detail below may be understood as “computer” in agreement with selective embodiments.

[0032] When controlling a robotic device to pick up and move an object, a robot control model can be used to determine the configuration of the robotic device. In this case, various predictive models may have advantages and disadvantages depending on the application. Various embodiments relate to a robot control model that can determine the configuration of a robotic device, taking into account the advantages and disadvantages of individual predictive models, thereby enabling the robotic device to successfully pick up an object with an increased probability. Specifically, an improved robot control model for picking up an object is provided.

[0033] Figure 1 shows a robotic apparatus assembly 100. The robotic apparatus assembly 100 may include a robotic apparatus 101. The robotic apparatus 101 shown in Figure 1 and described hereafter as illustrative is, for illustrative purposes, an exemplary robotic apparatus and may include, for example, an industrial robot in the form of a robotic arm for moving, assembling, or processing a workpiece. It should be noted that the robotic apparatus may be any kind of computer-controlled device, such as a robot (e.g., a manufacturing robot, a maintenance robot, a household robot, a medical robot, etc.), a vehicle (e.g., an autonomous vehicle), a household appliance, a production machine, a personal assistant, an access control system, etc.

[0034] The robotic device 101 includes robotic limbs 102, 103, and 104 and a base (or generally a support) 105, the base by which these robotic limbs 102, 103, and 104 are supported. The term "robotic limb" refers to the movable part of the robotic device 101, whose operation enables physical interaction with the surroundings, for example, to perform a task, or to perform or execute one or more capabilities.

[0035] For control, the robotic apparatus assembly 100 includes a control device 106 configured to enable interaction with the surroundings according to a control program. The last element 104 (as seen from the base 105) of the robotic limbs 102, 103, and 104 is also referred to as the end effector 104 and may include, for example, a gripping tool or suction device (e.g., a suction head) or one or more similar tools.

[0036] Other robotic limbs 102 and 103 (closer to the base 105) can constitute a positioning device, and therefore, a robotic arm 120 (or articulated arm) is provided at its end, along with the end effector 104. The robotic arm 120 is a mechanical arm (possibly having a tool at its end) that can perform functions similar to a human arm.

[0037] The robotic device 101 may include coupling elements 107, 108, and 109, which connect the robotic limbs 102, 103, and 104 to each other and to the base 105. The coupling elements 107, 108, and 109 may include one or more joints, each of which can provide rotational and / or translational motion (i.e., movement) relative to each other for the associated robotic limbs. The movement of the robotic limbs 102, 103, and 104 may be initiated by a moving member controlled by a control device 106.

[0038] The term “operating member” can be understood as a component suitable for responding to being driven in a manner that affects the mechanism. The operating member can convert instructions (so-called activations) output by the control device 106 into mechanical motion. An operating member, such as an electromechanical transducer, may be configured to convert electrical energy into mechanical energy in response to its drive control.

[0039] The term “control device” (also referred to as “control mechanism”) can be understood as any kind of logical implementation unit that may include, for example, a circuit and / or processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and further capable of giving instructions to, for example, the moving parts in this example. The control device may be configured, for example, by program code (e.g., software), to control the operation of a system, in this example, a robot.

[0040] In this example, the control device 106 includes a computer 110 and a memory 111, the memory 111 which stores code and data, and the computer 110 controls the robot device 101 based on that code and data. According to various embodiments, the control device 106 controls the robot device 101 based on a robot control model 112 stored in the memory 111.

[0041] According to various embodiments, the robotic device 101 (for example, the robotic arm 120) may be configured to pick up one or more objects 114. According to various embodiments, the robotic device 101 (for example, the robotic arm 120) may be configured to move the picked-up objects.

[0042] According to various embodiments, the robotic apparatus assembly 100 may include one or more sensors. One or more sensors may be configured to provide sensor data characterizing the state of the robotic apparatus. For example, one or more sensors may include imaging sensors, such as cameras (e.g., standard cameras, digital cameras, infrared cameras, stereo cameras, etc.), radar sensors, LiDAR sensors, position sensors, velocity sensors, ultrasonic sensors, acceleration sensors, pressure sensors, etc. According to various embodiments, imaging sensors may be configured to provide (e.g., detect and transmit to a control device 106) an image showing one or more objects 114. The image may be an RGB image, an RGB-D image, or a depth image (also referred to as a D image). A depth image as described herein may be any type of image having depth information. Specifically, a depth image may include three-dimensional information relating to one or more objects. A depth image as described herein may include, for example, a point cloud provided by a LiDAR sensor and / or a radar sensor. The depth image may be, for example, an image containing depth information provided by a LIDAR sensor.

[0043] The robot control model 112 may be configured to determine a robot configuration for picking up (and optionally for moving) one of the objects 114 from an image showing one or more objects 114. The control device 106 may be configured to use the robot control model 112 to control the robot device 101 to perform object pick-up using the determined robot configuration. Determining control parameters based on an image according to one robot control model (e.g., robot control model 112) is illustrated with reference to Figure 4. Exemplary training of such a robot control model is illustrated with reference to Figures 2A and 2B.

[0044] As used herein, the term "object pickup" may mean contact with an object that allows the object to be moved as a result of the movement of the robotic device 101 (e.g., robotic arm 120). Pickup may relate to the type of end effector 104. For example, the end effector 104 may include a gripping tool, in which case object pickup may be gripping of the object. For example, the end effector 104 may include a suction device (e.g., a suction head), in which case object pickup may be suction of the object.

[0045] According to various embodiments, the robot control model 112 may be generated (e.g., learned or trained) while the robot device 101 is not operating. According to various embodiments, the generated robot control model 112 may be used to determine the capabilities that the robot device 101 should perform while the robot device 101 is operating. According to various embodiments, the generated robot control model 112 may be further trained while the robot device 101 is operating.

[0046] Figure 2A shows flowchart 200A for training robot control models according to various embodiments. For illustrative purposes, various embodiments of the control device 106 and robot device 101 are described below. It should be noted that other robot device assemblies are available.

[0047] According to various embodiments, an image 202 showing one or more objects may be provided. The image 202 can be provided, for example, by one or more sensors 113 and may show one or more objects 114.

[0048] Computer 110 may be configured to implement a robot control model. The robot control model may include multiple prediction models 204 (2 ≤ i ≤ N), where "N" may be any integer greater than or equal to 2. Specifically, the robot control model is:

number

[0049] Specifically, ensemble methods are provided to improve the results of individual prediction models. According to various embodiments described herein, pickup predictions 206 (2≦i≦N) generated by multiple prediction models 204 (2≦i≦N) may be stacked by linear or nonlinear mixing. This mixing can be performed using a convolutional neural network (for example, in the form of image points). Furthermore, regions unsuitable for picking up objects may be considered. These regions can be determined independently of each task of the robotic device.

[0050] For illustrative purposes, we will describe below a set of prediction models 204(2≦i≦N) configured to generate their respective pick-up predictions for images I and 202. Please understand that images I and 202 for each prediction model 204(i) can be various images or multiple images, as described above. One of the prediction models 204(2≦i≦N) may be, for example, the model described in one of the references [1] to [5].

[0051] Each of the multiple prediction models 204 (2≦i≦N) may be configured to generate a pickup prediction 206(i) in response to the supply of image 202. Each pickup prediction 206(i) may include a pickup robot configuration map 208(i) and a pickup success probability map 210(i). The pickup robot configuration map 208(i) may include a pickup robot configuration vector for each image point in image 202 that describes the configuration of the robot device 101. Here, each image point can be considered a pickup center. The pickup success probability map 210(i) may include a predicted success probability for picking up an object for each image point in image 202, based on the pickup robot configuration vector assigned to each image point.

[0052] According to various embodiments, multiple prediction models 204 (2 ≤ i ≤ N) can output mutually distinct data having a pickup robot configuration vector and an assigned predicted success probability. Here, the computer 110 may be configured to structure the output data according to a pickup robot configuration map 208(i) and a pickup success probability map 210(i).

[0053] Specifically, in this way, each prediction model g for images I and 202 is generated. i , 204(i) is,

number

[0054] The probabilities used herein have assigned probability values. Where herein a probability is compared to another probability or threshold (e.g., greater than, less than, higher than, lower than, or greater than), this relates to the probability value to which the probability is assigned. The success probability assigned to a pickup robot configuration vector can indicate the probability with which a successful pickup operation is expected using the configuration described by this pickup robot configuration vector. A successful pickup operation may be, for example, that the end effector is able to grasp, lift, and / or move the object assigned to the robot configuration. This is similarly applicable to other capabilities (e.g., the motor capabilities of the robot arm 120). Determining the success probability for a pickup robot configuration may be, for example, a classification of the pickup robot configuration (e.g., class "success").

[0055] Figures 3A to 3C illustrate this in an illustrative way. The illustrative image 300 may have a height H and a width W, and may be formed by multiple image points p(h,w) in the height direction h (here, 0 ≤ h ≤ H) and the width direction w (here, 0 ≤ w ≤ W) (see Figure 3A). It should be understood that the image points p(h,w) for image 300 are not shown to scale, or do not need to be shown to scale. Pickup robot configuration map A generated for image 300. iThis may include each pickup robot configuration vector a(h,w) for each image point p(h,w) (see Figure 3B). Pickup success probability map Q generated for image 300. i This can include the predicted success probability q(h,w) for each image point p(h,w) (see Figure 3C).

[0056] A pickup robot configuration vector as used herein may include a tuple of values ​​describing the configuration of the robot device 101. Each value in the tuple may describe a parameter of the configuration of the robot device 101. The pickup robot configuration vector may be related to the robot device (e.g., robot device 101) to which a robot control model should or is being trained. The pickup robot configuration vector may be related, for example, to the type of end effector 104 (e.g., configured as a gripping tool or suction head), the degrees of freedom of the end effector (e.g., defined by robot limbs 102, 103), etc. For example, the end effector 104 may include a gripping tool, and one or more values ​​in the tuple forming the pickup robot configuration vector may represent the gripper width, yawing, yaw rate, tilt, and / or rotation of the robot device 101. A robot configuration as used herein may be the posture of the robot (e.g., the posture of the end effector). The robot configuration may include, for example, the rotation of the end effector (e.g., the orientation of the end effector). The robot configuration may include, for example, the position of the end effector (e.g., the translation of the end effector). The position t of the end effector is a 3D position.

number

[0057] According to various embodiments, each pick-up robot configuration vector may be a tuple having a plurality of D values. Specifically, the pick-up robot configuration vector may have a magnitude or dimension of D. According to various embodiments, the plurality of prediction models 204 (2≦i≦N) can output tuples of different magnitudes of configuration values. Here, the computer 110 may be configured to structure tuples of different magnitudes of configuration values according to a unified tuple having a dimension D that forms a pick-up robot configuration vector. According to various embodiments, the pick-up robot configuration map A i may have a size of H*W*D.

[0058] The predicted success probability, as used herein, may have a value, for example, between "0" and "1". The predicted success probability may be, for example, a percentage success probability, where "0" may represent a 0% success probability and "1" may indicate a 100% success probability. According to various embodiments, the pick-up success probability map Q i may have a size of H*W*1.

[0059] Figure 2A shows, for illustrative purposes, a first prediction model 204(i=1) and a second prediction model 204(i=2) for the case N=2. The first prediction model 204(1) can generate a first pickup prediction 206(1) in response to the supply of images I and 202, accompanied by a first pickup robot configuration map A1 (having a first pickup robot configuration vector for each image point of images I and 202) and a first pickup success probability map Q1 (having a predicted first success probability for each image point of images I and 202). The second prediction model 204(2) can, in response to the supply of images I and 202, generate a second pickup prediction 206(2) which includes a second pickup robot configuration map A2 (having a second pickup robot configuration vector for each image point of images I and 202) and a second pickup success probability map Q2 (having a predicted second success probability for each image point of images I and 202).

[0060] The robot control model may include a blended model 212. The blended model 212 may be configured to generate a third pickup prediction 218 in response to the supply of all pickup predictions 206 (2≦i≦N) generated by multiple prediction models 204 (2≦i≦N) (for example, if N=2, the first pickup prediction 206(1) and the second pickup prediction 206(2)). Specifically, the blended model 212 can blend the pickup predictions 206 (2≦i≦N) generated by multiple prediction models 204 (2≦i≦N). Linear blending is described below for illustrative purposes.

[0061] The third pickup prediction 218 may include a third pickup robot configuration map A, 220, which has a third pickup robot configuration vector for each image point in image I, 202. The third pickup prediction 218 may also include a third pickup success probability map Q, 222, which has a predicted third success probability for each image point in image I, 202.

[0062] The third pickup robot configuration map A, 220 is for each image point in image I, 202, and all pickup robot configuration maps A i The third pickup robot configuration vector a(h,w,) may include the predicted best success probability q(h,w,). The third pickup success probability map Q, 222 may include the predicted best success probability q(h,w,) for each image point in image I, 202. The computer 110 may be configured to select a third pickup robot configuration vector with the assigned predicted best success probability and generate control information, which controls the robot device 101 according to the third pickup robot configuration vector. The prediction model 204(i) with the predicted best success probability and the associated selection of image points are as follows:

number

number

[0063] Pickup robot configuration map A generated by multiple prediction models 204 (2 ≤ i ≤ N) i , 208(i) and Pickup Success Rate Map Q i , 210(i) can be expressed in the third pickup robot configuration maps A, 220 and the third pickup success probability maps Q, 222 using the function f as follows.

number

[0064] According to various embodiments, the mixed model 212 may include a first weighting coefficient 214 and a second weighting coefficient 216. The mixed model 212 is a pickup robot configuration map A, 220 generated by multiple prediction models 204 (2 ≤ i ≤ N) weighted using the first weighting coefficient 214. i , may be configured to be generated by combinations of 208(i). The mixed model 212 is a third pickup success probability map Q, 222, which is weighted by a second weighting coefficient 216, and is generated by multiple prediction models 204(2≦i≦N) Q i It may be configured to be generated by a combination of 210(i).

[0065] A weighted combination may be, for example, a weighted sum or a weighted product. For illustrative purposes, various possibilities for weighted combinations are described in this specification as weighted sums.

[0066] In the following examples, we will use Pickup Robot Configuration Map A. i , 208(i) weighted combinations and pickup success probability map Q i Various weighted combination methods are described in 210(i).

[0067] The first weighting coefficient 214 is the (first) weighting coefficient θ for each prediction model 204(i). i This may include the following: Each first weighting coefficient θ i This can be assumed to be a scalar value. Specifically, the first weighting coefficient θ of the prediction model 204(i) for all image points in image I, 202. i These may be considered identical. The third pickup robot configuration map A, 220 is, for example, the first weighting coefficient θ i (Here, using 2≦i≦N),

number

[0068] The second weighting coefficient 216 is the (second) weighting coefficient w for each prediction model 204(i). i This may include each second weighting coefficient w i w can be assumed to be a scalar value. Specifically, w is the second weighting coefficient of the prediction model 204(i) for all image points in image I, 202. i These can be considered identical. The third pickup success probability map Q, 222, is given, for example, the second weighting coefficient w i (Here, using 2≦i≦N),

number

[0069] The mixed model 212 can be trained by adapting the first weighting coefficient 214 and the second weighting coefficient 216. In this case, the same first weighting coefficient θ is applied to all image points in images I, 202. i , and / or the same second weighting coefficient w for all image points of image I,202 i This can reduce the computational and technical costs of training the mixed model 212.

[0070] However, the first weighting coefficient 214 may also include the first weighting coefficient for each prediction model 204(i) and each image point in images I and 202. Accordingly, the first weighting coefficient 214 is the first weighting matrix Θ for each prediction model 204(i). i This may include the following. Each image point in images I and 202 has a first weighting matrix Θ i It may be assumed that exactly one element is assigned to each first weighting matrix Θ. i This is the pickup robot configuration map A i , which may include a size of 208(i), that is, it may include H*W (optionally H*W*D). The third pickup robot configuration map A, 220 is, for example, the first weighting matrix Θ i (Here, using 2≦i≦N),

number

number

[0071] The second weighting coefficient 216 may include each second weighting coefficient for each prediction model 204(i) and each image point in images I and 202. Accordingly, the second weighting coefficient 216 includes each second weighting matrix W for each prediction model 204(i). i This may include the following. Each image point in images I and 202 has a second weighting matrix W i It is acceptable to assume that exactly one element is assigned to each second weighting matrix W. i This is the pickup success rate map Q i , which may include a size of 210(i), i.e., H*W. A third pickup success probability map Q, 222, is used, for example, in the second weighting matrix W. i (Here, using 2≦i≦N)

number

[0072] For all 202 image points in Image I, the same first weighting coefficient θ i , 202 and / or the same second weighting coefficient w i Unlike the first weighting matrix Θ of the image point characteristics i or a second weighting matrix W i This can improve the accuracy and flexibility of the trained mixed model 212.

[0073] According to various embodiments, the first weighting coefficient 214 of the mixed model 212 is the same as the first weighting coefficient θ i This can be pre-trained (i.e., the first weight matrix Θ of each predictive model 204(i)) i Each element has a value θ i (having), then the first weighting coefficients 214 of the mixed model 212 can be trained individually for each image point of the image. Specifically, this can ensure relatively rapid training progress at first, and at the same time, ensure high accuracy of the trained mixed model 221 (i.e., the trained first and second weighting coefficients).

[0074] Figure 2B shows flowchart 200B for training robot control models according to various embodiments.

[0075] According to various embodiments, the mixed model 212 may include a first weighting model 244. The first weighting model 244 is represented in images I, 202 and all of the pickup robot configuration maps A. i The system may be configured to output a first weighting coefficient 214 in response to the supply of . Thus, the first weighting coefficient 214 may be related to images I, 202.

[0076] The first weighting coefficient 214 is, for example, the first weighting coefficient θ for each prediction model 204(i). i This may include, and the first weighting coefficient may be related to image I, 202 in this case (i.e., θ i (I)). Third pickup robot configuration map A, 220,

number

[0077] According to various embodiments, the mixed model 212 may include a second weighted model 246. The second weighted model 246 includes images I, 202 and all pickup success probability maps Q. i It may be configured to output a second weighting coefficient 216 in response to the supply of 210(i). Thus, the second weighting coefficient 216 may be related to images I, 202. The second weighting coefficient 216 is, for example, each second weighting coefficient w for each prediction model 204(i) i This may include, and the second weighting coefficient may be related to images I, 202 in this case (i.e., w i (I)). The third pickup success probability map Q, 222,

number

[0078] The first weighting model 244 and / or the second weighting model 246 may be configured to generate a segmentation image using the segmentation (i.e., image segmentation) of image I, 202. As described herein, image I, 202 may represent one or more objects 114. The segmentation image may represent the segmentation (e.g., classification) of one or more objects 114. Each weighting model may be configured to generate weighting coefficients based on the segmentation image. Each weighting model may, for example, determine, based on the segmentation image, which regions of image I, 202 do not contain any of the one or more objects 214. As an example for illustrative purposes, the weighting coefficients associated with image points in these regions may be set to zero, and as a result, the third pickup success probability map Q, 222 for this image point has a predicted success probability q(h,w) equal to, for example, zero.

[0079] The first weighting model 244 and / or the second weighting model 246 may be configured to determine each normal vector and associated standard deviation for each surface shown in image I, 202. Specifically, the normal vector and associated standard deviation may represent the flatness of each surface. Each weighting model may be configured to generate weighting coefficients based on the determined normal vector and associated standard deviation. Each weighting model can, for example, determine which surfaces shown in image I, 202 are unsuitable for object pickup based on the normal vector and standard deviation. For example, in the case of an end-effector gripping or suction device, a strongly curved surface may be unsuitable to ensure the necessary adhesion (e.g., gripping or suction) of the end-effector to the object. As an example for illustration, the weighting coefficients associated with image surfaces having a standard deviation above a threshold can be set to zero, and as a result, the third pickup success probability map Q, 222 for this image point has a predicted success probability q(h,w) equal to, for example, zero.

[0080] Specifically, each weighting model may use the segmentation image and / or a normal vector with the associated standard deviation to consider additional criteria for object pickup, independently of the task of the robotic device 101. For example, areas unsuitable for object pickup may be considered.

[0081] Similar to the linear blending described above, the blending model 212 can, according to various embodiments, nonlinearly blend pickup predictions 206 (2≦i≦N) generated by multiple prediction models 204 (2≦i≦N). The blending model 212, like the first weighting model 244, may include a first nonlinear weighting model, which is shown in images I, 202 and all pickup robot configuration maps A. i In response to the supply of 208(i), the first weighting coefficient 214 is output as the first nonlinear weighting coefficient. The third pickup robot configuration map A, 220,

number

number

[0082] According to various embodiments, the robot control model may include a first convolutional neural network (CNN).

[0083] The convolutional neural network described herein may contain any number of layers and may be characterized by the format of the input data. Furthermore, the convolutional neural network may include multiple weighting coefficients in the hidden layers of the convolutional neural network that weight the data. Here, the convolutional neural network can be trained by adapting these weighting coefficients. The convolutional neural network may use batch normalization, L1 normalization, or L2 normalization, etc.

[0084] The first convolutional neural network may include a first weighting coefficient of 214. The first convolutional neural network includes all pickup robot configuration maps A generated by multiple prediction models 204 (2 ≤ i ≤ N). i , in response to the supply of 208(i), it may be configured to output a third pickup robot configuration map A, 220. For example, computer 110 may output pickup robot configuration map A generated by multiple prediction models 204(2≦i≦N). iThe data may be combined into a tensor of size H*W*N*D, and this tensor may be supplied to the first convolutional neural network. Here, the first convolutional neural network may be a 3-dimensional convolutional neural network having a kernel of k*k*l (here, k*k in H*W dimensions and l in D dimensions).

[0085] According to various embodiments, the robot control model may include a second convolutional neural network. The second convolutional neural network may include a second weighting coefficient 216. The second convolutional neural network generates all pickup success probability maps Q generated by multiple prediction models 204 (2 ≤ i ≤ N). i , in response to the supply of 210(i), it may be configured to output a third pickup success probability map Q, 222. For example, computer 110 may output pickup success probability maps Q generated by multiple prediction models 204(2≦i≦N). i The 210(i) can be stacked in an N-channel tensor of size H*W*N, and this tensor can be supplied to the first convolutional neural network. Here, the second convolutional neural network can be assumed to be a 2-dimensional convolutional neural network having k*k kernels (in H*W dimensions).

[0086] Convolutional neural networks use the same weighting coefficients θ as described above. i Or w i , weighting matrix Θ i Or W i and / or nonlinear weighted model f i , g iCompared to others, it can have lower complexity. This can improve the efficiency of robot control models. Furthermore, convolutional neural networks are scalable, which increases flexibility. In addition, by parallelizing, for example, the first and second convolutional neural networks, the parallelization capabilities of modern graphics processor (GPU) architectures can be utilized. Convolutional neural networks can implement both linear and nonlinear mixtures. As an example, the pickup robot configuration map A of all prediction models 204 (2≦i≦N) i , 208(i) and Pickup Success Probability Map Q i ,210(i) may be obtained in parallel by another computer and supplied to computer 110, and the computer uses the first and second convolutional neural networks to obtain the third pickup robot configuration map A,220 and the third pickup success probability map Q,222 in parallel. Specifically, this reduces the training time and therefore increases efficiency.

[0087] Pickup robot configuration map A i , 208(i) and Pickup Success Rate Map Q i It should be understood that 210(i) and 210(i) can be combined with each other by different types of combinations. As an example, see Pickup Robot Configuration Map A. i , 208(i) is the first weighting matrix Θ i It can be combined using the pickup success probability map Q i, 210(i) can be combined with each other using a second convolutional neural network. In one example, one convolutional neural network can perform linear mixing, and the other convolutional neural network can perform nonlinear mixing. According to various embodiments, the first convolutional neural network and / or the second convolutional neural network may include a portion that generates a segmentation image or normal vector with a corresponding standard deviation for image 202, as illustrated with reference to Figure 2B. In this case, it should be understood that image 202 can also be supplied to each convolutional neural network.

[0088] According to various embodiments, a robot control model can be trained by adapting a first weighting coefficient 214 and a second weighting coefficient 216 (see Figure 2A). For this purpose, target data 224 (also referred to as training data) can be supplied. The target data 224 may be stored, for example, in memory 111.

[0089] In the target data 224, a successful pickup may be assigned to at least one image point p(h*,w*) in images I and 202. For at least one image point p(h*,w*) to which a successful pickup is assigned, a second weighting coefficient 216 may be adapted so as to increase the generated, predicted third success probability q(h*,w*).

[0090] In the target data 224, a failed pickup may be assigned to at least one other image point p(h\h*,w\w*) in images I, 202. For at least one other image point p(h\h*,w\w*) to which a failed pickup is assigned, a second weighting coefficient 216 may be adapted such that the generated, predicted third success probability q(h\h*,w\w*) is reduced.

[0091] In the target data 224, one or more values ​​of the target tuple may be assigned to at least one image point p(h,w) of images I, 202. The target tuple may represent a target pickup robot configuration. The first weighting coefficient 214 may be adapted so that the third pickup robot configuration vector a(h,w) is generated such that for at least one image point p(h,w), a tuple representing the configuration of the third pickup robot configuration vector a(h,w) has one or more of these values ​​of the target tuple. For example, the target tuple may be a target pickup robot configuration vector, and the generated third pickup robot configuration vector a(h,w) may correspond to the target pickup robot configuration vector.

[0092] In the case of the first weighting model 244 and / or the second weighting model 246, the mixed model 212 can be trained by adapting the first weighting model 244 or the second weighting model 246. Specifically, the weighting coefficients generated by each weighting model are also adapted in this way. The first weighting model 244 and / or the second weighting model 246 may be machine learning-based models.

[0093] In the case of a first convolutional neural network and / or a second convolutional neural network, the weighting coefficients of each convolutional neural network can be adapted. Here, any kind of training principle can be used, such as backpropagation. The convolutional neural network can be trained, for example, using supervised learning (an exemplary training is described in reference [4]). The convolutional neural network can also be trained, for example, using reinforcement learning (an exemplary training is described in references [2] and [5]).

[0094] According to various embodiments, a robot control model can be iteratively trained on multiple images and each associated target data, as illustrated above (as iterations) for image I, 202 and target data 224.

[0095] Multiple predictive models 204(2≦i≦N) may be pre-trained, trained individually on this target data and / or other target data, or trained together with a mixed model 212. When trained together, the mixed model 212 and each predictive model 204(i) of the multiple predictive models 204(2≦i≦N) can be adapted in each iteration of training.

[0096] Figure 4 shows a flowchart 400 for controlling a robotic device (for example, robotic device 101) according to various embodiments. A robotic control model can be used here. The robotic control model may include multiple predictive models 204 (2 ≤ i ≤ N) (for example, when N = 2, a first predictive model 204(1) and a second predictive model 204(2)) and a trained mixed model 412.

[0097] The trained mixed model 412 may be configured to generate a third pickup prediction 418 in response to a supply of all pickup predictions 206 (2≦i≦N) generated for an image 402 from multiple prediction models 204 (2≦i≦N). The third pickup prediction 418 may include a third pickup robot configuration map A, 420 and a third pickup success probability map Q, 222.

[0098] The trained mixed model 412 may include a first weighting coefficient 414 and a second weighting coefficient 416. The first weighting coefficient 414 and the second weighting coefficient 416 may be configured according to the first weighting coefficient 214 or the second weighting coefficient 216 which have been trained on multiple images. For example, the trained mixed model 412 may be the mixed model 212 which has been trained on multiple images.

[0099] The computer 110 may be configured to select a third pickup robot configuration vector a(h,w) to which the predicted best third success probability q(h,w) is assigned. The computer 110 may be configured to generate control data 424 for controlling the robot device 101 according to the selected third pickup robot configuration vector. For example, the control device 106 can control the robot device 101 according to the selected third pickup robot configuration vector.

[0100] Optionally, a robot control model can be additionally trained during the operation of the robot device 101. According to various embodiments, the control device 106 can control the robot device 101 to perform object pickup using a selected third pickup robot configuration vector, and can determine whether the object pickup was successfully performed. In an example of reinforcement learning training, the control device 106 (e.g., computer 110) can determine successful or unsuccessful pickups as rewards. For example, if the object pickup is successful, a predetermined maximum reward (e.g., a reward value equal to "1") may be determined. For example, if the object pickup is unsuccessful, a predetermined minimum reward (e.g., a reward value equal to "0") may be determined. The image from which the selected third pickup robot configuration vector has been determined can be used as additional target data (e.g., training data) in relation to the reward for adapting the first and / or second weighting coefficients.

[0101] Figure 5 shows exemplary robotic apparatus assembly 500 having multiple robotic arms according to various embodiments. The robotic apparatus assembly 500 shows two robotic arms as an example. It should be noted that the robotic apparatus assembly 500 may include more than two robotic arms.

[0102] According to various embodiments, the robot device assembly 500 may include the robot device assembly 100. The robot arm 120 of the robot device 101 may be the first robot arm 120.

[0103] The robotic apparatus assembly 500 may further include a second robotic arm 520. The robotic apparatus assembly 500 shown in Figure 5 and described exemplified below represents, for illustrative purposes, an exemplary robotic apparatus 101 comprising a first robotic arm 120 and a second robotic arm 520. It should be noted that a robotic apparatus may be any kind of computer-controlled apparatus that includes at least two operating devices for manipulating an object (e.g., moving, processing, etc.). These include, for example, robots (e.g., manufacturing robots, maintenance robots, household robots, medical robots, etc.), vehicles (e.g., autonomous vehicles), household appliances, production machinery, personal assistants, access control systems, etc.

[0104] The second robot arm 520 may include robot limbs 502, 503, and 504, and a base (or generally a holder) 505 that supports the robot limbs 502, 503, and 504. The control device 106 may be configured to enable interaction with the surroundings according to a control program. For example, the control device 106 may be configured to control the first robot arm 120 and the second robot arm 520. The last element 504 of the robot limbs 502, 503, and 504 (as seen from the base 505) is also referred to as the end effector 504 and may include one or more tools, such as a gripping tool or suction device (e.g., a suction head) or similar. The second robot arm 520 may include coupling elements 507, 508, and 509 that connect the robot limbs 502, 503, and 504 to each other and to the base 505.

[0105] The second robot arm 520 may similarly be configured to pick up one of the one or more objects 114. The robot control model 122 may include one trained robot control model 412 for both the first robot arm 120 and the second robot arm 520 (see Figure 4 and its related description). The two trained robot control models 412 can each generate one third pick-up prediction 418. The computer 110 may be configured to select a third pick-up robot configuration vector to which the robot arms 120, 520 and the predicted best third success probability are assigned, and can generate corresponding control data 424 to control the selected robot arm according to the selected third pick-up robot configuration vector.

[0106] Figure 6 shows a flowchart of a computer-implemented method 600 for training robot control models according to various embodiments.

[0107] Method 600 may include supplying an image showing one or more objects to a first predictive model of a robot control model in order to generate a first pickup prediction (in 602). The first pickup prediction may include, for each image point of the image, each first pickup robot configuration vector describing the configuration of the robot device, along with an assigned predicted first success probability.

[0108] Method 600 may include supplying an image to a second prediction model of a robot control model in order to generate a second pickup prediction (in 604). The second pickup prediction may include, for each image point of the image, each second pickup robot configuration vector describing the configuration of the robot device, along with an assigned predicted second success probability.

[0109] Method 600 may include supplying the first and second pickup predictions to a mixed model of the robot control model in order to generate a third pickup prediction (in 606). The third pickup prediction may include, for each image point of the image, a third pickup robot configuration vector which is a combination of the first pickup robot configuration vector and the second pickup robot configuration vector weighted by a first weighting coefficient, and a predicted third success probability which is a combination of the predicted first success probability and the predicted second success probability weighted by a second weighting coefficient.

[0110] Method 600 may include training a robot control model by adapting a first weighting coefficient and a second weighting coefficient to increase the predicted third success probability for at least one image point in the image, based on target data in which a successful pickup is assigned to at least one image point (in 608).

[0111] According to various embodiments, a method for controlling a robotic device may include detecting an image showing one or more objects. The method may further include supplying the image to a first prediction model of a robotic control model in order to generate a first pickup prediction. The first pickup prediction may include, for each image point of the image, a first pickup robot configuration vector describing the configuration of the robotic device, along with an assigned predicted first success probability. The method may further include supplying the image to a second prediction model of a robotic control model in order to generate a second pickup prediction. The second pickup prediction may include, for each image point of the image, a second pickup robot configuration vector describing the configuration of the robotic device, along with an assigned predicted second success probability. The method may further include supplying the first and second pickup predictions to a mixed model of a robotic control model in order to generate a third pickup prediction. The third pickup prediction may include, for each image point in the image, a third pickup robot configuration vector which is a combination of the first pickup robot configuration vector and the second pickup robot configuration vector weighted by a first weighting coefficient, and a predicted third success probability which is a combination of the predicted first success probability and the predicted second success probability weighted by a second weighting coefficient. The method may further include controlling the robotic device to pick up the object indicated by the image point assigned to the third pickup robot configuration vector, according to the third pickup robot configuration vector, to which the predicted best third success probability is assigned.

Claims

1. A computer-implemented method (200A, 200B) for training a robot control model, wherein the robot control model is configured to control a robot device (101) to pick up one of one or more objects (114), The method (200A, 200B) comprises: - Supplying an image (202) showing the one or more objects (114) to a first prediction model (204(1)) of the robot control model to generate a first pick-up prediction (206(1)), wherein the first pick-up prediction (206(1)) includes, for each image point of the image (202), each first pick-up robot configuration vector describing the configuration of the robot device (101), with an assigned predicted first success probability; - Supplying the image (202) to a second prediction model (204(2)) of the robot control model to generate a second pick-up prediction (206(2)), wherein the second pick-up prediction (206(2)) includes, for each image point of the image (202), each second pick-up robot configuration vector describing the configuration of the robot device (101), with an assigned predicted second success probability; - Supplying the first pick-up prediction (206(1)) and the second pick-up prediction (206(2)) to a mixed model (212) of the robot control model to generate a third pick-up prediction (218), wherein the third pick-up prediction (218) includes, for each image point of the image (202), - A third pick-up robot configuration vector that is a combination of the first pick-up robot configuration vector and the second pick-up robot configuration vector weighted by a first weighting factor (214), and - A predicted third success probability that is a combination of the predicted first success probability and the predicted second success probability weighted by a second weighting factor (216). Including The method (200A, 200B) further comprises Training the robot control model by adapting the first weighting coefficient (214) and the second weighting coefficient (216) so that the predicted third success probability increases, based on target data (224) to which at least one successful pickup of the image (202) is assigned, for the at least one image point. Method (200A, 200B). **Claim 2** The first weighting coefficient (214) includes weighting coefficients for all first pickup robot configurations and weighting coefficients for all second pickup robot configurations. The method (200A, 200B) according to claim 1. **Claim 3** The first weighting coefficient (214) includes each weighting coefficient for each first pickup robot configuration vector and each weighting coefficient for each second pickup robot configuration vector. The method (200A, 200B) according to claim 1. **Claim 4** The second weighting coefficient (216) includes weighting coefficients for all predicted first success probabilities and weighting coefficients for all predicted second success probabilities. The method (200A, 200B) according to claim 1. **Claim 5** The second weighting coefficient (216) includes each weighting coefficient for each predicted first success probability and each weighting coefficient for each predicted second success probability. The method (200A, 200B) according to claim 1. **Claim 6** The robot control model includes a first convolutional neural network with the first weighting coefficient (214) and a second convolutional neural network with the second weighting coefficient (216), and the generation of the third pickup prediction (218) includes: - Supplying the first pickup robot configuration and the second pickup robot configuration to the first convolutional neural network to generate the third pickup robot configuration. - Supplying the predicted first success probability and the predicted second success probability to the second convolutional neural network to generate the predicted third success probability. Including. The method (200A, 200B) according to claim 1.

7. The method (200A, 200B) further comprises: - generating a segmentation image using segmentation of the image (202), wherein a plurality of image points of the segmentation image are assigned to the one or more objects (114); - supplying the generated segmentation image to a first weighting model (244) of the robot control model to generate the first weighting coefficient (214); - supplying the generated segmentation image to a second weighting model (246) of the robot control model to generate the second weighting coefficient (216); - the training of the robot control model using adaptation of the first weighting coefficient (214) and the second weighting coefficient (216) includes adaptation of the first weighting model (244) and the second weighting model (246). The method (200A, 200B) according to claim 1.

8. The method (200A, 200B) further comprises: - for each surface shown in the image (202), determining a normal vector of the shown surface and a related standard deviation; - supplying the determined normal vector and the related standard deviation to a third weighting model (244) of the robot control model to generate the first weighting coefficient (214); - supplying the determined normal vector and the related standard deviation to a fourth weighting model (246) of the robot control model to generate the second weighting coefficient (216); - the training of the robot control model using adaptation of the first weighting coefficient (214) and the second weighting coefficient (216) includes adaptation of the third weighting model (244) and the fourth weighting model (246). The method (200A, 200B) according to claim 1.

9. Each pick-up robot configuration vector is a tuple of values describing the configuration of the robot device (101), assigning one or more values of a target tuple describing a target pick-up robot configuration to the at least one image point in the target data (224). The robot control model generates a third pick-up robot configuration vector having a tuple of values having the one or more values of the target tuple for the at least one image point, and the predicted third success probability assigned to the third pick-up robot configuration vector is high. Train the robot control model so that The method (200A, 200B) according to claim 1.

10. In the target data (224), assign a failed pick-up to at least one other image point of the image (202), and use the adaptation of the first weighting coefficient (214) and the second weighting coefficient (216) to Train the robot control model so that the predicted third success probability generated for the at least one other image point is reduced. The method (200A, 200B) according to claim 1.

11. A method (400) for controlling a robot device (101), the method (400) comprising: - Detecting an image (402) showing one or more objects (114); - Supplying the image (402) to a robot control model trained according to the method of claim 1 to generate a third pick-up prediction (418); - For the third pick-up robot configuration vector of the third pick-up prediction (418) to which the predicted highest third success probability is assigned, according to the third pick-up robot configuration vector, among the one or more objects (114), Controlling the robot device (101) to pick up the object assigned to the image point of the third pick-up robot configuration vector. Method (400).

12. A computer program comprising instructions for causing a processor to perform the method (200A, 200B, 400) according to any one of claims 1 to 11 when executed by the processor.

13. A computer-readable medium storing instructions for causing a processor to perform the method (200A, 200B, 400) according to any one of claims 1 to 11 when executed by the processor.

14. A training device including a computer configured to implement the method (200A, 200B) according to any one of claims 1 to 10.

15. A robot device (101), wherein the robot device (101) - at least one robot arm (120) configured to pick up an object, - a control device (106), and the control device (106) - using the robot control model trained according to the method according to any one of claims 1 to 10, to obtain a third pick-up prediction (418) for an image (402) showing one or more objects (114); - for the third pick-up robot configuration vector of the third pick-up prediction (418) to which the predicted highest third success probability is assigned, according to the third pick-up robot configuration vector, of the one or more objects (114), the object assigned to the image point of the third pick-up robot configuration vector is picked up, the at least one robot arm (120) is configured to be controlled. Robot device (101).

Images