Apparatus and method for training a machine learning model for determining a gripping of a multi-finder clamp for

Abstract

According to various embodiments, there is provided a method for training a machine learning model to determine a grip of a multi-finger gripper for manipulating an object, comprising: for each of a plurality of scenes, each scene comprising an object in a respective pose: determining a point cloud representation of the object included in the scene; determining a multi-finger grip for each of a plurality of surface points of the object through a machine learning model; in one embodiment, a method includes determining a grasp, determining a grasp loss of the grasp, where the loss includes at least one of a collision loss and a grasp stability loss, determining a total loss including the determined grasp loss, and adjusting a machine learning model to reduce the total loss.

Description

Technical Field

[0001] The present disclosure relates to apparatus and methods for training a machine learning model to determine grasps of a multi-fingered gripper for manipulating an object. Background Art

[0002] Robotic grasping is a fundamental skill required to manipulate objects in cluttered environments, for example in bin picking applications. Multi-fingered robotic hands, such as the "Shadow Hand", mimic the structure of the human hand, enabling complex object manipulation. Data-driven grasp planning for multi-fingered robotic hands aims to find a hand configuration that provides stable fixation of the target object within the hand. This involves predicting the 6D pose of the robotic gripper, together with determining the joint angles of the fingers of the multi-fingered hand. This increases the difficulty due to the increased number of degrees of freedom.

[0003] Therefore, efficient methods for training machine learning models to predict grasping, especially grasping by multi-finger grippers, are desirable.

[0004] The publication by C. Ferrari and JF Canny in "Planning optimal grasps" in ICRA, vol. 3, no. 4, 1992, p. 6 (hereinafter referred to as reference 1) describes in particular the Q1 grasping metric. Summary of the invention

[0005] According to various embodiments, there is provided a method for training a machine learning model to determine grasping of a multi-fingered gripper for manipulating an object, the method comprising: The following operations are performed for each of a plurality of scenes, each scene including (at least comprising) an object in a corresponding pose (i.e., each scene including one or more objects in various positions and orientations, wherein these objects may be different between different scenes, but may also be (at least partially) the same): Determine point cloud representations of objects included in the scene; determining a grasp for each of a plurality of surface points of the object using a machine learning model, wherein each grasp is a multi-finger grasp specifying a palm position and a finger joint configuration; determining a grasp loss for the grasp, wherein the loss comprises at least one of a (differentiable) collision loss and a (differentiable) grasp stability loss, determining an overall loss comprising the determined grasp losses (i.e. combining, e.g. adding or averaging, the grasp losses), and adjusting the machine learning model to reduce the overall loss (i.e. adjusting parameter values, typically weights, in a direction such that the overall loss decreases (i.e. the overall loss will be lower in another forward pass), i.e. adjusting according to the gradient of the loss, typically using backpropagation).

[0006] The above method allows machine learning models (e.g., neural networks) to be trained to predict multi-finger grasping in the form of efficient grasp representations (palm positions and joint configurations), which facilitates the acquisition of dexterous grasping skills for complex objects while achieving efficient training.

[0007] Especially for multi-finger grippers, collision loss is an important aspect to consider, since each finger may collide with other objects, or for example the walls of the box the object should be taken out of. This can be determined by determining the mesh of the gripper, determining the collision points from the mesh of the gripper, and calculating the distance to the mesh of the object, other objects, or other elements in the scene (such as the box walls).

[0008] Hereinafter, various embodiments are described.

[0009] Embodiment 1 is the method for training the machine learning model as described above.

[0010] Embodiment 2 is a method according to embodiment 1, wherein the grasp stability loss is an upper bound of the Q1 metric loss.

[0011] Since the Q1 metric and its lower bound are difficult to compute, using an upper bound on the Q1 metric allows for more efficient training.

[0012] Embodiment 3 is a method according to embodiment 1 or 2, comprising: for each scene, determining one or more ground truth grasps, and determining a total loss to include a supervised training loss between the determined grasp and the one or more ground truth grasps for each scene.

[0013] Therefore, expert knowledge can be included to enable the machine learning model to learn the “optimal” crawl.

[0014] Embodiment 4 is a method according to any one of embodiments 1 to 3, comprising: determining a surface mesh of an object from a point cloud representation, and determining a total loss to include a guidance loss for each determined grasp, which guide loss penalizes the distance between the contact point of the multi-finger gripper according to the determined grasp and the surface of the object as given by the surface mesh.

[0015] Thus, the machine learning model learns to determine grasps that actually touch the object.The grid may also (or may alternatively) be used to determine grasp stability loss.

[0016] Embodiment 5 is a method according to any one of Embodiments 1 to 4, further comprising: for each scene and each determined grasp, determining the confidence of the grasp (i.e., the confidence of the machine learning model in the grasp) through a machine learning model, and the higher the determined confidence of the grasp, the more the loss contribution of the determined grasp is reduced.

[0017] Thus, it is possible to avoid the machine learning model being “trained away” from grasping something in which it is very confident.

[0018] Embodiment 6 is a method for controlling a robot, comprising: training a machine learning model according to any one of Embodiments 1 to 5, obtaining a point cloud representation of an object to be manipulated, determining a grasp by feeding the point cloud representation of the object to be manipulated from the point cloud representation to the trained machine learning model, and controlling the robot to perform the determined grasp to manipulate the object.

[0019] Embodiment 7 is a data processing device (particularly a robot controller) configured to execute the method according to any one of embodiments 1 to 6.

[0020] Embodiment 8 is a computer program comprising instructions, which, when executed by a computer, cause the computer to perform the method according to any one of embodiments 1 to 6.

[0021] Embodiment 9 is a computer-readable medium comprising instructions, which, when executed by a computer, cause the computer to perform the method according to any one of embodiments 1 to 6. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In the drawings, like reference numerals generally refer to the same parts throughout the different views. The drawings are not necessarily drawn to scale, but instead generally focus on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, in which: Figure 1 A robot is shown.

[0023] Figure 2 Illustration of a neural network used to determine multi-finger grasping posture.

[0024] Figure 3 A flow chart illustrating a method for training a machine learning model to determine grasps of a multi-fingered gripper for manipulating an object is shown.

[0025] The following detailed description refers to the accompanying drawings, which illustrate the specific details and aspects of the present disclosure in which the present invention can be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects. DETAILED DESCRIPTION

[0026] Hereinafter, various examples will be described in more detail.

[0027] Figure 1 A robot 100 is shown.

[0028] The robot 100 comprises a robot arm 101, for example, an industrial robot arm for handling or assembling a workpiece (or one or more other objects 113). The robot arm 101 comprises manipulators 102, 103, 104 and a base (or support) 105, by which the manipulators 102 and 103, 104 are supported. The term "manipulator" refers to a movable member of the robot arm 101, which is actuated to enable physical interaction with the environment, for example to perform a task. For control, the robot 100 comprises a (robot) controller 106, which is configured to implement the interaction with the environment according to a control program. The last member 104 of the manipulators 102, 103, 104 (the farthest from the support 105) is also called the end effector 104 and comprises a gripping tool (which can also be a suction gripper).

[0029] Other manipulators 102, 103 (closer to the support 105) may form a positioning device such that together with the end effector 104 a robotic arm 101 is provided having the end effector 104 at the end. The robotic arm 101 is a robotic arm capable of providing similar functions to a human arm.

[0030] The robot arm 101 may include joint elements 107, 108, 109 interconnecting the manipulators 102, 103, 104 to each other and to the support 105. The joint elements 107, 108, 109 may have one or more joints, each of which may provide the associated manipulators with rotatable motion (i.e., rotational motion) and / or translational motion (i.e., displacement) relative to each other. The movement of the manipulators 102, 103, 104 may be initiated by means of actuators controlled by the controller 106.

[0031] The term "actuator" may be understood as a component adapted to affect a mechanism or process in response to being driven. The actuator may implement a command (so-called activation) issued by the controller 106 as a mechanical movement. An actuator, such as an electromechanical converter, may be configured to convert electrical energy into mechanical energy in response to being driven.

[0032] The term "controller" may be understood as any type of logic implementation entity, which may include, for example, a circuit and / or a processor capable of executing software stored in a storage medium, firmware, or a combination thereof, and may, for example, issue instructions to the actuators in this example. The controller may be configured, for example, by program code (e.g., software) to control the operation of the system (in this example, the robot).

[0033] In this example, the controller 106 includes one or more processors 110 and a memory 111 storing code and data based on which the processor 110 controls the robotic arm 101. According to various embodiments, the controller 106 controls the robotic arm 101 based on a machine learning model (e.g., including one or more neural networks) 112 stored in the memory 111.

[0034] The end effector 104 can be a multi-fingered (e.g., five-fingered) hand. Therefore, in addition to the characteristic degrees of its posture, the end effector 104 also has additional degrees of freedom (and is therefore highly dexterous). For example (according to the so-called "shadow hand"), it can have 24 degrees of freedom, including 3 positional degrees of freedom, 3 rotational degrees of freedom (i.e., there are 6 degrees of freedom for the posture of the end effector 104) and 18 finger joint angles (i.e., finger joint degrees of freedom (DoF)). For example, there are 24 joints in total, some fingers have four to five joints, and some fingers have three joints. However, all 24 joints are controlled by only 18 degrees of freedom (the values ​​can be changed to see the parameters of the joint movement) because some joints use the same degree of freedom, so there are fewer degrees of freedom than the number of joints.

[0035] The increased number of degrees of freedom increases the complexity of control. In particular, methods designed for controlling parallel grippers are generally not suitable for controlling an end effector 104 in the form of a multi-fingered hand.

[0036] According to various embodiments, a method for generating a dexterous high-DOF robotic hand grasp (i.e., grasp determination) with physically plausible and on-collision grasps from a cluttered scene consisting of unknown objects is provided, which determines grasps from an input point cloud densely (i.e., for a large number of input points, such as for each input point of a downsampled input point cloud), and includes a differentiable grasp planning that uses differentiable optimization as an inductive bias to integrate a generalized Q1 grasp metric and a collision loss. The proposed method allows for the prediction of a dense grasp distribution for a cluttered scene, and the differentiable grasp planning is formulated as a continuous optimization. It enables efficient prediction of a set of diverse grasps for a cluttered scene of objects, which are densely projected to less ambiguous contact points of these objects on the input point cloud. According to various embodiments, a machine learning model 112 (in particular a neural network) is provided, which can predict a multimodal grasp distribution, such as dense (point-by-point) grasp proposals, so it achieves better generalization. The machine learning models are: i) geometry-aware, e.g. taking into account collision losses; and ii) physics-aware, e.g. taking into account the Q1 grasp metric (grasp quality based on contact between gripper points and object points).

[0037] Figure 2 A neural network 200 for determining a multi-finger grasping gesture is illustrated.

[0038] The input 201 of the neural network 200 is a point cloud P of a scene (the scene is registered in the workspace of the robot 100, e.g., from multi-views from one or more cameras 114). The point cloud P specifically represents an object to be grasped (e.g., object 113) (e.g., at least a portion of the points of the point cloud P are surface points of the object to be grasped). The neural network 200 predicts grasps (i.e., points of the point cloud corresponding to surface points of the object) that are densely projected on the object point cloud. Each grasp is represented as a 6D reference palm point p (the origin of the palm) and a joint angle θ of the fingers of the hand, i.e., g = {p, θ}.

[0039] According to one embodiment, the end effector 104 has the form of a 5-fingered hand and has (6+18) degrees of freedom (DoF), where: The parameter p includes the three-dimensional translation vector t={x, y, z} and the three-dimensional rotation vector {α, β, γ} of the palm (i.e., the overall posture of the end effector 104 ), The 18-dimensional parameters θ include 18 finger joint angles.

[0040] The neural network 200 uses a single inference to predict a diverse set of collision-free grasps for objects in the scene (or each object in the case where the point cloud includes representations of multiple objects). This is achieved by processing the raw visual 3D data (i.e., point cloud) and directly predicting the 6D pose of the palm and the joint angles of the fingers associated with each object point on the input point cloud.

[0041] According to various embodiments, a contact grasp representation is used, where the palm translation of a successful ground truth grasp (given by ) are mapped to their corresponding contact points (on the corresponding objects), which are represented by p obj express.

[0042] To simplify training, a 6D continuous rotation representation is used to describe the orientation of the robot hand, i.e., two 3D vectors {a, b}, and the orientation matrix is ​​reconstructed by the Gram-Schmidt procedure. The 6D continuous representation can be transformed into a more conventional representation, such as a quaternion. The advantage of this representation is that it eliminates any discontinuities, resulting in more stable training. For the representation of the palm position, instead of calculating it directly, the object point (given by ) to represent the translation of the palm, as shown below: t palm =p obj +offset x，y，z (1) in is the offset from the reference point to the palm. Since only the offset needs to be predicted around the object point, this method reduces the prediction interval.

[0043] About the joint angle of fingers The neural network 200 directly predicts their values. The result of this representation is a 27-dimensional state vector describing the grasping posture of the end effector 104, namely: Each contact point p obj g = {offset x，y，z , {a, b}, I}.

[0044] To train (including validation) and test the neural network 200, various workspace (e.g., table-top) scenes with ground truth grasps (labels) can be generated. The scenes can contain different object selections and different poses (e.g., upright, lying, stacked, etc.). For each object in the scene, multiple grasps (e.g., hundreds) can be included as labels in the training data.

[0045] As training input data, depth images of the generated scene may be presented (eg, from multiple viewpoints, such as from all sides of the workspace), and a training input point cloud generated from the depth images.

[0046] The generated poses of objects from the scene can be used to transform ground truth grasps in a canonical state to objects in the created scene. For example, grasps are matched to object points in the corresponding (training) input point cloud to generate dense annotations of the point cloud (i.e., ground truth grasp labels). The matching of grasps is done, for example, by using reference points on the palm of the hand as follows: Determine the distance d between the ground truth reference point pp at the origin of the palm and the i-th object point po,i t , and in addition, the normal direction n along the object's mesh is determined o，i The signed distance d n , as shown below: d t =||p p -p o，i ||2,d n =n o，i *(p p -p o，i ). (2) For example, if the translation of the palm is less than 5 mm from an object point in the point cloud and has a positive distance in the normal direction, then the ground truth grasp matches the object point. Specifically, for object point i, there exists the following ground truth grasp set whose reference point p satisfies the matching condition: g i = {g p |d t <=5mm&d n >0}. (3) For each point i of each object o, the matching capture set g i Is a non-unique match. Has a non-empty set g i The object points with are set with positive labels, otherwise they are set with negative labels.

[0047] For each input point cloud 201, the neural network 200 predicts dense grasp proposals for points on the input point cloud PCL, i.e., achieving Functions that map input point clouds to output elements (including grip configurations for multiple grip points) Next, where m is the number of predicted grasp points, and 27 is the dimensionality of each grasp representation, as described above. Since the prediction should be "dense", the number m is high, e.g. m = 512 points.

[0048] Figure 2 The graph shows the result of a dense grasp distribution on the input point cloud (of a cylindrical object). Each line represents a grasp point, and the normal direction of the object surface is specified in the grasp point.

[0049] The neural network 200 includes, for example, a feature extraction network 202, and uses, for example, a segmentation-style structure. This allows the feature network 202 to extract embeddings for individual points. For example, a U-shaped network is used as the feature extraction network 202 (which can be considered as a backbone). The layers of the feature extraction network are, for example, based on a collection abstraction module from PointNet++, such as four downsampling modules followed by three upsampling modules. As an example, let the input 201 of the network be a point cloud of N=2048 points (which can be a downsampled point cloud of the original point cloud generated from one or more images). The first downsampling module of the feature extraction network 202 reduces the number of points to 512. This is the level used for the prediction of the network (by the head network 203 containing prediction heads 204, 205, 206, 207). The output of the feature extraction network 202 is a point-by-point geometric embedding 208 for the input point cloud 201, which includes a 128-dimensional feature vector for each of the 512 points.

[0050] The head network 203 transforms the feature representation of each of the 512 points to a corresponding grasp representation. It includes a corresponding prediction head 204, 205, 206 for each of the position offset, orientation, and finger joint angle predicted for each grasp. The motivation for using different heads for these three different parts is that each part has a different scale and unit. Separating each prediction allows each head network 204, 205, 206 to learn a specific part. The head networks 204, 205, 206 can be, for example, simple three-layer MLP (multilayer perceptron) networks: the first two layers of each head network 204, 205, 206 are linear layers with ReLu activation functions, which also use batch normalization. The next last linear layer predicts the output (and does not have an activation function). The output dimensions of the head networks 204, 205, 206 correspond to the dimensions of the grasp representation (multiplied by the number of predicted grasps m for each scene, in this example, m = 512).

[0051] In order to enable the selection of the best grasp from the predicted m grasps, according to one embodiment, an additional head network 207 is provided to predict a one-dimensional confidence value. The additional head network 207 has the same structure as the other head networks 204, 205, 206, but has an additional Sigmoid activation function at the output.

[0052] In addition, according to various embodiments, the model of the hand (i.e., the hand-shaped end effector) is transformed to the predicted grasp in the scene. To this end, the pose of each joint of the hand is calculated. Using the predicted joint state of the predicted grasp as input, a forward kinematics layer can be applied to transform the hand mesh model into its corresponding world coordinate state, i.e., all points on the gripper sampled from the hand model mesh are transformed to the predicted grasp. Specifically, the predicted joint angles can be used to calculate the homogeneous transformation from the parent joint to each child joint. In order to enforce joint angle restrictions in the prediction, the joint angles are clamped to a predefined range Θ=max(min(Θ,Θ max ),Θ min ). The position of each link can be calculated by applying all transformations from the chain (a series of links (finger segments) connected by joints) to that link (finger segment). This allows the loss function to be calculated based on the predicted collision with the object.

[0053] To train the neural network 200 to predict the grasp representation, according to various embodiments, a task loss function is used, which is a combination of different loss functions. For the prediction at point i on the input point cloud 201, the task loss can be expressed as follows: Where w1, w2, w3, w4 are weighted coefficients between different loss terms. The loss terms are described below: a) Supervision (training) loss: Chamfer loss The distance between the ground truth grasp and the prediction of the network 200 is calculated. To calculate the loss, the closest ground truth that satisfies the matching condition as described in equation (3) is determined. This added condition enables object points to be matched only locally to the ground truth grasp. Essentially, the chamfer loss is a loss at each predicted point i using the predicted pose To calculate, as follows: where g i is the set of ground truth grasp poses at point i as defined in Eq. (3). Note that we only compute

[0054] b) Differentiable Collision Loss: This collision loss is calculated based on the mesh model of the hand. For this loss function, collision points (e.g., 2000 collision points) are sampled from the hand model mesh. These points are used to calculate the distance between the hand model and the mesh of the scene, such as the table, objects, and the linkage of the hand (to capture self-collisions). Then, the collision loss is calculated as follows: in is the signed distance from any object mesh's enclosing point to each collision point i, and L is the number of meshes in the scene.

[0055] c) Guidance loss: Guidance loss is used to guide the inside of the hand towards the closest mesh. The guidance loss should minimize the distance between the hand and the object surface. The guidance loss uses the distance calculation from the loss function to guide the hand towards the next mesh surface. To calculate the guidance loss, a set of, for example, V = 45 hand points are marked that are located inside the hand. These points are used to calculate the distance between the hand model and the mesh of the scene. The distance is calculated for each mesh in the scene, including the table, the object, and the hand link. The loss is then calculated by the following equation: where p j is the position of the hand point, and p mesh，j is the closest to p among all the grids j The positions of these collision points are calculated by the collision point layer. The interior of the hand points defines a subset of these points. For multi-object input scenes, all object meshes are treated as one mesh. The hand is guided towards the closed face of the mesh, regardless of which mesh the face belongs to.

[0056] d) Differentiable Q1 metric loss: The last term of the loss function of equation (4) is an upper bound on the Q1 metric (see Ref. 1). This upper bound describes the quality of the (force-enclosed) grasp with a minimum wrench to break the grasp. Using a lower bound on the Q1 metric can lead to higher grasp quality. However, it is computationally expensive. This upper bound is given by: where M is the metric tensor used to weight the torque components, w is the twist of the grasp, and s j It is the support of the grabbing point.

[0057] e) Confidence loss: In the task loss formula, confidence is not considered in the prediction loss function.

[0058] The joint loss function including confidence can be calculated using the following formula: where c i is the predicted confidence score at point i, B is the batch size, and N is the number of points in the prediction. The neural network 200 may, for example, use a loss function L confidence To train (i.e., use back propagation to adjust the neural network 200 to reduce the loss function L confidence ), where for example w5=1.

[0059] In general, according to various embodiments, there is provided a Figure 3 The method shown in the figure.

[0060] Figure 3 A flow chart 300 illustrating a method for training a machine learning model to determine grasps of a multi-fingered gripper for manipulating an object is shown.

[0061] In 301, for each of a plurality of (training) scenes, the following operations are performed, each scene including (at least including) an object in a corresponding pose (i.e., each scene includes one or more objects in various positions and orientations, wherein these objects may be different between different scenes, but may also be (at least partially) the same): At 302, a point cloud representation of an object included in the scene is determined; At 303, determining, by a machine learning model, a grasp for each of a plurality of surface points of the object, wherein each grasp is a multi-finger grasp specifying a palm position (e.g., a position with the aid of a palm reference point) and a finger joint configuration (e.g., joint angles of all joints); In 304, a grasping loss for the grasp is determined, wherein the loss comprises at least one (or both) of: a (differentiable) collision loss (e.g., computed from the object mesh, which may be determined from the point cloud and / or the mesh of the gripper, see item b) above); and a (differentiable) grasping stability loss (based on the object mass and the friction between the gripper and the object, see item d) above; this loss is analytically difficult to compute, so an approximation is used according to various embodiments).

[0062] At 305 , a total loss is determined that includes the determined grab losses (ie, the grab losses are combined, such as added or averaged).

[0063] At 306, the machine learning model is adjusted to reduce the total loss (i.e., parameter values, typically weights, are adjusted in a direction such that the total loss decreases (i.e., the total loss will be lower in another forward pass), i.e., adjustment is made according to the gradient of the loss, typically using backpropagation).

[0064] Various embodiments may receive and use image data (ie, digital images) from various vision sensors (cameras) such as video, radar, lidar, ultrasound, thermal imaging, motion, sonar, etc., for example as the basis for a point cloud representation of an object.

[0065] Figure 3 The method can be performed by one or more data processing devices (e.g., computers or microcontrollers) having one or more data processing units. The term "data processing unit" can be understood to mean any type of entity that enables processing of data or signals. For example, data or signals can be handled according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any combination thereof, or be formed by them. Any other device for implementing the corresponding functions described in more detail herein may also be understood to include a data processing unit or a logic circuit. One or more of the method steps described in more detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0066] Thus, according to one embodiment, the method is computer-implemented.

Claims

1. A method for training a machine learning model (112, 202) to determine grasping of a multi-finger gripper (104) for manipulating an object, comprising: For each of a plurality of scenes, the following operations are performed, each scene including an object in a corresponding pose: Determining a point cloud representation of an object included in a scene (201); determining, by a machine learning model (112, 202), a grasp for each of a plurality of surface points of an object, wherein each grasp is a multi-finger grasp specifying a palm position and a finger joint configuration; determining a grasp loss of the grasp, wherein the loss comprises at least one of a collision loss and a grasp stability loss, Determine the total loss including the determined crawl loss, and The machine learning model (112, 202) is adjusted to reduce the total loss.

2. The method of claim 1, wherein the grasping stability loss is an upper bound of the Q1 metric loss.

3. The method according to claim 1 or 2, comprising: For each scene, one or more ground truth grasps are determined, and the total loss is determined to include, for each scene, a supervised training loss between the determined grasps and the one or more ground truth grasps.

4. The method according to any one of claims 1 to 3, comprising: A surface mesh of the object is determined from the point cloud representation (201), and the total loss is determined to include, for each determined grasp, a guidance loss that penalizes the distance between a contact point of the multi-finger gripper (104) according to the determined grasp and the surface of the object as given by the surface mesh.

5. The method according to any one of claims 1 to 4, further comprising: For each scenario and each determined grasp, the confidence of the grasp is determined by the machine learning model (112, 202), and the higher the determined confidence of the grasp, the more the loss contribution of the determined grasp is reduced.

6. A method for controlling a robot 100, comprising: Training a machine learning model (112, 202) according to any one of claims 1 to 5; Obtaining a point cloud representation of the object to be manipulated (201); determining grasping by feeding the point cloud representation (201) of the object to be manipulated from the point cloud representation to a trained machine learning model (112, 202); as well as The robot 100 is controlled to perform the determined grasp to manipulate the object.

7. A data processing device (102) configured to execute the method of any one of claims 1 to 6.

8. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

Images