Apparatus and method for controlling a robot to pick up objects at different orientations

Abstract

According to various embodiments, a method for controlling a robot to pick up an object in a different pose is described, the method having: defining a plurality of reference points on the object; mapping a first camera image of the object in a known pose onto a first descriptor image; identifying descriptors of the reference points from the first descriptor image; mapping a second camera image of the object in an unknown pose onto a second descriptor image; searching the identified descriptors of the reference points in the second descriptor image; determining positions of the reference points in three-dimensional space in the unknown pose from the found positions; and determining a pick-up pose of the object for the unknown pose from the determined positions of the reference points.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an apparatus and a method for controlling a robot to pick up an object in different orientations. BACKGROUND

[0002] In order to enable flexible production or processing of objects by a robot, it is desirable that the robot is able to operate on an object irrespective of the pose in which the object is placed in the workspace of the robot. Thus, the robot should have the ability to recognize in which orientation an object is placed, at least in respect to what is important for picking up (e.g. grasping) the object, that is, the robot should have the ability to determine the pick-up pose (e.g. grasping pose) of the object so that the end effector (e.g. with gripper) of the robot can be correspondingly oriented accurately and moved to the accurate position. A corresponding control method for a robot apparatus for picking up objects in different orientations is desirable. SUMMARY

[0003] According to various embodiments, a method for controlling a robot to pick up an object in different orientations is provided, the method having: prescribing a plurality of reference points on the object such that the orientations of the reference points in three-dimensional space define a pick-up pose of the object; receiving a first camera image of the object in a known orientation of the object such that the positions of the reference points in the first camera image are known; mapping the first camera image onto a first descriptor image by means of a machine learning model which is trained to assign object points visible in a camera image to descriptors independently of the positions of the object points in the camera image; identifying the descriptors of the reference points in the first descriptor image by reading the first descriptor image at the known positions of the reference points; receiving a second camera image of the object in an unknown orientation of the object in which the object is to be picked up; mapping the second camera image onto a second descriptor image by means of the machine learning model; searching for the identified descriptors of the reference points in the second descriptor image; determining the positions of the reference points in three-dimensional space from the found positions of the descriptors of the reference points in the descriptor image in the unknown orientation; determining the pick-up pose of the object for the unknown orientation from the determined positions of the reference points in three-dimensional space; and controlling a robot apparatus to pick up the object in accordance with the determined pick-up pose.

[0004] The above method enables reliably picking up (e.g. grasping) an object in any orientation of the object. A camera image of the object in an unknown orientation is sufficient to determine the pick-up pose of the object. In particular, there is no need for an arrangement of multiple cameras covering multiple angles. The machine learning model can be flexibly trained so that the machine learning model can be used for various objects. Then, only the reference points have to be prescribed for each object and the descriptors of the reference points have to be determined.

[0005] Various examples are explained in the following.

[0006] Example 1 is a method for controlling a robot to pick up an object in a different orientation as described above.

[0007] Example 2 is the method of example 1, wherein the positions of the reference points in three-dimensional space in the unknown orientation are determined from the found positions of the descriptors of these reference points in the second descriptor image in such a way that the depth information of the positions of these reference points in the second camera image that correspond to the positions of the reference points in the second descriptor image are projected into three-dimensional space.

[0008] Thereby, the pick-up pose can be determined with the aid of a single camera image and the associated depth information, e.g. an RGBD image.

[0009] Example 3 is the method of example 1 or 2, wherein the reference points have at least two reference points that are specified such that the at least two reference points are arranged along an elongated portion of the object, and wherein the pick-up pose of the object for the unknown orientation is determined by determining an axis that extends through the reference points.

[0010] In this way, a pick-up pose can be determined that enables the robot to be controlled to grasp around the elongated portion. For example, grippers at the end effector of the robot are controlled such that they are located in a plane that is perpendicular to the axis that extends through the reference points, and then the grippers are closed.

[0011] Example 4 is the method of any one of examples 1 to 3, wherein the reference points have at least three reference points that are specified such that the at least three reference points are arranged on a flat surface of the object, and wherein the pick-up pose of the object for the unknown orientation is determined by determining a plane that extends through the reference points.

[0012] In this way, a pick-up pose can be determined that enables the robot to be controlled to grasp the surface, e.g. to suck up or lift the surface. For example, a suction device at the end effector of the robot is controlled such that a suction port or nozzle of the suction device is perpendicular to the plane that extends through the reference points, and then the suction device is activated.

[0013] Example 5 is a method of any one of Examples 1 to 4, wherein training the machine learning model comprises: obtaining a 3D model of an object, wherein the 3D model includes a vertex mesh; determining a descriptor for each vertex of the mesh; generating training data image pairs, wherein each training data image pair includes a training input image displaying the object and a target image, and wherein generating the target image comprises: determining the vertex positions of the vertices of the object model of the object in the training input image, and for each determined vertex position in the training input image, assigning a descriptor determined for the vertex at that vertex position to a position in the target image; and training the machine learning model by supervised learning using the training data image pairs as training data.

[0014] This training method enables the training of machine learning models (such as neural networks) to perform more accurate predictions (i.e., descriptor determination) compared to using self-supervised learning (meaning it enables a wider range of network applications). Furthermore, this training method provides greater flexibility in adapting the machine learning model, allowing it to be applied to various problems and reducing training data requirements (e.g., the amount of training data needed).

[0015] Example 6 is the method of Example 5, wherein the generation of training data image pairs includes: obtaining multiple images of an object with different poses; and generating training data image pairs for each obtained image by generating target images for the obtained images.

[0016] This enables the training of the machine learning model (e.g., a robot with a robot controller that implements the machine learning model) to identify the topology of an object regardless of its pose, such as in the robot's workspace.

[0017] Example 7 is a method of Example 5 or 6, which includes: determining the vertex positions of the vertices of the object model of the object in the training input image, based on the corresponding pose of the object in the training input image (e.g., in camera coordinates).

[0018] This enables precise determination of vertex positions, and in return, it allows for the production of accurate target images for supervised training.

[0019] Example 8 is a method of any of Examples 5 through 7, wherein the vertices of the 3D model are connected by edges, each edge having a weight that specifies the proximity of two vertices in the object connected by that edge, and wherein the descriptor of each vertex of the mesh is determined by searching for the vertices whose distance between the descriptors of the connected vertex pairs is minimized by weighted sum of the edges between the vertex pairs.

[0020] It is clear that these descriptors are determined in an optimal manner. Thereby, a high consistency of these descriptors is achieved. Also, the descriptor mapping is easily interpretable.

[0021] Example 9 is a robot control device set up to perform the method according to any one of examples 1 to 8.

[0022] Example 10 is a computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of examples 1 to 8.

[0023] Example 11 is a computer readable medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of examples 1 to 8. BRIEF DESCRIPTION OF DRAWINGS

[0024] In the drawings, like reference numerals refer to like parts throughout the various views unless otherwise indicated. These drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. In the following description, various aspects of the application are described with reference to the following drawings, in which:

[0025] Figure 1 A robot is shown;

[0026] Figure 2 Training of a neural network according to an embodiment is illustrated;

[0027] Figure 3 An exemplary embedding of a 4-node graph for descriptor determination is shown;

[0028] Figure 4 Definition of angles at vertices of a 3D model for edge weight determination according to a Laplace-Beltrami-Operator is illustrated;

[0029] Figure 5 A method for training a machine learning model to recognize an object topology of an object from an image of the object according to an embodiment is shown;

[0030] Figure 6 A method for controlling a robot to pick up an object in different orientations is shown. DETAILED DESCRIPTION

[0031] The following detailed description relates to the accompanying drawings that illustrate particular details of and aspects of the disclosure, which can be implemented in conjunction with the present disclosure. Other aspects can be utilized and structural, logical, and electrical changes can be made without departing from the scope of the present disclosure. The various aspects of the disclosure are not necessarily mutually exclusive, as some aspects of the disclosure can be combined with one or more other aspects of the disclosure to form new aspects.

[0032] Various examples are described in greater detail below.

[0033] Figure 1 A robot 100 is shown.

[0034] The robot 100 comprises a robot arm 101, e.g. an industrial robot arm for manipulating or mounting a workpiece (or one or more other objects). The robot arm 101 comprises manipulators 102, 103, 104 and a base (or pedestal) 105 by means of which the manipulators 102, 103, 104 are supported. The term “manipulator” relates to movable parts of the robot arm 101, the manipulation of which enables physical interaction with the environment, e.g. in order to perform a task. For control, the robot 100 comprises a (robot) control device 106 which is designed to implement the interaction with the environment in accordance with a control program. The last member 104 of the manipulators 102, 103, 104 (which is furthest from the pedestal 105) is also referred to as an end effector 104 and can comprise one or more tools, such as a welding torch, a gripper, a paint spraying tool, etc.

[0035] The other manipulators 102, 103 (which are closer to the pedestal 105) can form a positioning device, so that the robot arm 101 with the end effector 104 at its end is provided together with the end effector 104. The robot arm 101 is a robotic arm which can provide similar functionality as a human arm (possibly with a tool at its end).

[0036] The robot arm 101 can comprise link elements 107, 108, 109 which connect the manipulators 102, 103, 104 to each other and to the pedestal 105. The link elements 107, 108, 109 can have one or more links which can provide a rotatable motion (i.e. turning motion) and / or a translational motion (i.e. displacement) of the respective manipulators relative to each other, respectively. The motion of the manipulators 102, 103, 104 can be initiated by means of actuators which are controlled by the control device 106.

[0037] The term "actuator" can be understood as a component which is configured to influence a mechanism or a process as a reaction to its actuation. The actuator can implement an instruction created by the control device 106 (so-called activation) into a mechanical movement. The actuator, for example an electromechanical transducer, can be designed to convert electrical energy into mechanical energy as a reaction to its actuation.

[0038] The term "controller" can be understood as any type of logic-implementing entity, for example, which can comprise a circuit and / or a processor, which is capable of executing software, firmware or a combination thereof stored in a memory medium, and which can, for example, issue instructions to the actuator in the present example. The controller can be configured, for example, by program code, for example software, in order to control the operation of the system, namely the robot in the present example.

[0039] In the present example, the control device 106 comprises a memory 111 storing code and data and one or more processors 110 controlling the robot arm 101 based on these code and data. According to various embodiments, the control device 106 controls the robot arm 101 based on a machine learning model 112 stored in the memory 111.

[0040] According to various embodiments, the machine learning model 112 is designed and trained to enable the robot 100 to recognize a pick-up pose of an object 113, for example placed into the working space of the robot arm 101. This means that the robot 100 recognizes how it can pick up the object 113, that is to say how it has to orient its end effector 104 and where it has to move it in order to pick up, for example grasp, the object 113. The pick-up pose is understood such that it contains sufficient information for the pick-up, that is to say information about the orientation and position 113 of the object, from which it can be determined how the object 113 can be grasped. The pick-up pose does not necessarily need to contain complete orientation information about the object 113, since in the case of an object 113 having a rotationally symmetrical part for grasping, it can for example be irrelevant how this rotationally symmetrical part is rotated around its axis of rotation.

[0041] The robot 100 can for example be equipped with one or more cameras 114 which enable the robot to take images of its working space. The cameras 114 are for example fixed on the robot arm 101 such that the robot can take images of the object 113 from different angles by moving the robot arm 101 around.

[0042] According to various embodiments, the control device 106 implements a machine learning model 112 for object recognition, which the control device uses the output of for determining a pick-up pose of an object 113.

[0043] An example of such a machine learning model 112 for object recognition is a dense object network. The dense object network maps an image, for example an RGB image provided by a camera 114, onto a descriptor space image of arbitrary dimensionality (dimension D).

[0044] The dense object network is a neural network which is trained to output a descriptor space image for an input image of an image using self-supervised learning. However, the effectiveness of this approach depends to a large extent on the quality of the collected training data and the participating sensors, for example the camera 114. Furthermore, the interpretation of the network predictions can be difficult.

[0045] According to various embodiments, a method for recognizing objects and their poses is used, in which a 3D model of the object is assumed to be known, for example a CAD (Computer Aided Design) model, which is often the case for industrial assembly or machining tasks. According to various embodiments, a non-linear dimensionality reduction technique is used in order to calculate an optimal target image for training an input image of a neural network. Instead of using self-supervised training of the neural network as such, according to various embodiments, supervised training of the neural network is used. It is also possible to take RGBD images (RGB + depth information) of the object and to determine a 3D model of the object therefrom.

[0046] According to one embodiment, in order to generate training data for training the machine learning model 112, a data collection is first performed. In particular, for example, registered RGB (red-green-blue) images are collected. Here, a registered image means an RGB image with known intrinsic and extrinsic camera values. In a real-world scenario, for example, a camera 114 fixed on a robot, for example a camera fixed at a joint of a robot hand, is used in order to scan an object during movement of the robot, for example a robot arm 101, around it. Other extrinsic estimation techniques can be used, for example ChArUco marker calibration, that is to say the object can be placed into different positions and poses relative to a ChArUco board and images of the arrangement (ChArUco board and object) are taken. In a simulated scenario, photo-realistically generated RGB images are used using known object poses.

[0047] In the case of the collection of RGB images, for supervised training of the neural network, target images of the RGB images are rendered.

[0048] It is assumed that the pose of each object in world coordinates is known in each collected RGB image. This is not complex for a simulated scene, but requires manual adjustments for real-world scenes, e.g. placing objects at predefined positions. It is also possible to use RGBD images in order to determine the position of objects.

[0049] With this information and in case of using vertex descriptor computation techniques, such as described below, a descriptor image (i.e. a training output image, also called target image or Ground-Truth-Bild) is rendered for each RGB image (i.e. a training input image).

[0050] If a target image is generated for each RGB image, i.e. a pair of an RGB image and a target image is formed, these pairs of training input images and associated target images can be used as training data for training a neural network, as shown in Figure 2 .

[0051] Figure 2 The training of the neural network 200 according to the embodiment is elucidated.

[0052] The neural network 200 is a fully convolutional network that maps a tensor (input image) onto a tensor (output image).

[0053] The fully convolutional network comprises a plurality of convolutional layers of levels 204, followed by pooling layers, up-sampling layers 205 and skip connections 206 in order to combine the outputs of different layers.

[0054] For training, the neural network 200 receives a training input image 201 and outputs an output image 202 with pixel values in the descriptor space (e.g. in color components according to descriptor vector components). A training loss is computed between the output image 202 and a target image 203 associated with the training input image. This can be done for a batch of training input images, and the training loss can be averaged over these training input images, and the weights of the neural network 200 are trained using this training loss using stochastic gradient descent. The training loss computed between the output image 202 and the target image 203 is e.g. an L2 loss function (in order to minimize the least square error pixel-wise between the target image 203 and the output image 202).

[0055] The training input image 201 shows the object and target images, as well as the output image containing vectors in the descriptor space. The vectors in the descriptor space can be mapped to colors, making the output image 202 (and the target image 203) resemble a heatmap of the object.

[0056] Vectors in the descriptor space (also called (dense) descriptors) are d dimensional vector ( d These vectors (e.g., 1, 2, or 3) are assigned to each pixel in the corresponding image (e.g., each pixel of input image 201, assuming input image 201 and output image 202 have the same dimensions). Dense descriptors implicitly encode the surface topology of the object shown in input image 201, invariant to the object's pose or camera position.

[0057] Given a 3D model of an object, it is possible to analytically determine the optimal (in Riemannian terms) and unique descriptor vector for each vertex of the object's 3D model. According to various implementations, a target image for the registered RGB image is generated using these optimal descriptors (or estimates of these descriptors determined through optimization), which leads to fully supervised training of a neural network 200. Additionally, the descriptor space ignores the chosen descriptor dimension. d In this case, it becomes interpretable and optimal.

[0058] In the following text, the 3D model is considered to be embedded in Riemannian manifolds This involves calculating geodesics (shortest paths between vertices) while also considering geometry. Optimal surface topology can be explicitly encoded by embedding the 3D model into... d In the Euclidean descriptor space, the geodesic distance between adjacent vertices is preserved as well as possible. This Euclidean space is considered a descriptor space, and the optimal mapping is performed. The search. In one implementation, the optimal embedding of vertices in the descriptor space is determined (or at least estimated) using the Laplacian operator for the mesh and its eigenvalue decomposition. Alternatively, the geodesic computation is separated from the mapping optimization, and the descriptors in the unique frame are extracted by computing the Laplacian operator of the 3D model.

[0059] The embedding of a 3D object model in Euclidean space into a descriptor space is determined using the method described below, in order to preserve the distances between vertices (e.g., geodesic distances).

[0060] To perform dimensionality reduction using the Laplace operator, point groups This should correspond to a node in an undirected graph. Two nodes should be calculated. The connection strength or proximity between .

[0061] The goal is to find d dimensional embedding (typically d D ) such that if and are close, their embeddings should also be close:

[0062] (1)

[0063] where .

[0064] The optimization problem (1) is equivalent to

[0065] (2)

[0066] where is a semi-positive definite Laplacian matrix. A is the adjacency matrix with elements and . It should be noted that the optimal solution can have arbitrary scale and trend. To remove this randomness, one can normalize the weighted second moments by which forces unit variance in different dimensions. Then, the resulting optimization problem becomes

[0067] (3)

[0068] In the case of finite optimization with Lagrange parameters

[0069] (4)

[0070] This is a generalized eigenvalue problem, which can be solved by standard linear algebra libraries. Since L and D are (semi-) positive definite matrices, the eigenvalues can be written as .

[0071] Moreover, the first eigenvector (first column of ) is equal to 1 (vector of all ones), which is the trivial solution mapping every vertex to a single point. Additionally, any two eigenvectors are orthogonal to each other. From the solution of the eigenvalue problem, one gets N eigenvalues and a dimension of N ​​​corresponding eigenvector. In practice, however, only the first d few eigenvectors corresponding to the lowest eigenvalues are used (in addition to the trivial solution).

[0072] Thus, Y the first i column is the embedding of the node i in R d , while each row represents the embedding of each point in a different orthogonal dimension.

[0073] Figure 3 An exemplary embedding of a 4-node graph is shown.

[0074] The eigenvalues are of importance with respect to the optimality aspect of the embedding. In the case of an optimal embedding , the constraint is satisfied, and thus applies:

[0075] (5)

[0076] That is, these eigenvalues correspond to the embedding error in different dimensions. For simplicity, one should d = 1, in which case each x is mapped to the point y = 1. In this case, (5) is simplified to:

[0077] (6)

[0078] Since , that is, if all vertices of the object are mapped to one point, the embedding error is 0, since the distance between all points y is 0. This is not useful for practical purposes, and thus the first eigenvalue and eigenvector are ignored. Using d = 2 corresponds to mapping each point x to a line, and Lambda 1 is the corresponding embedding error, and so on. Since the eigenvectors are orthogonal to each other, the increase of d will add a new dimension to the embedding, which aims at minimizing the error in the new orthogonal dimension. The same effect can be seen in (3): since , the original objective can be transformed so as to minimize the embedding error in each dimension. Thus, regardless of the chosen d , the resulting descriptor vector is optimal.

[0079] In some cases, the subsequent eigenvalues are identical, that is (see Figure 3Example, where the eigenvalues for d = 2 and d = 3 are the same eigenvalue). This carries some information about symmetry, where there are multiple orthogonal dimensions with the same embedding error. In fact, in the 4-node graph example above, if the graph is fully connected, then the embedding into each dimension is symmetric and all eigenvalues are the same, except for the trivial solution. Figure 3

[0080] The graph embedding method above can be directly applied to meshes, point clouds, etc. For example, a K-Nearest Neighbor (KNN) algorithm can be used to form local connections between vertices and create an adjacency matrix. This method is sufficient to create a graph-Laplacian and compute the embedding of each vertex. Nonetheless, this method is inherently based on Euclidean distance metric and heuristics that do not necessarily respect the underlying Riemannian geometry of 3D object models. For example, some edges can go through the object, or can connect non-adjacent vertices of a mesh. Even a few incorrect entries in the adjacency matrix can lead to poor embedding performance. According to one embodiment, if working with a model, then the geodesic distance between two arbitrary vertices is guaranteed to be accurate or have minimal approximation error.

[0081] In general, an object model, such as a mesh, or a point cloud, can be represented as an embedding in a Riemannian manifold with a metric g that varies uniformly. A Riemannian manifold M with a metric that varies uniformly can be considered "locally Euclidean", which captures the locally smooth nature of real-world objects. The generalization of the Laplacian operator to Riemannian manifolds is the Laplace-Beltrami (LB) operator Δ. Similar to the Laplacian in Euclidean space, the LB operator applied on a function is the divergence of the gradient of that function. The Laplacian for graphs and in Euclidean space is easy to compute (from adjacency information or finite differences), while the LB operator in differential geometry is based on Exterior Calculus and is generally not easy to use for manifolds.

[0082] However, for known discrete manifolds, such as meshes, the LB operator can be approximated. This provides an efficient and simple computational framework when working with meshes, point clouds, etc. Since the Riemannian equivalent of the Laplacian is the Laplace-Beltrami, the embedding method above can be directly applied to Δ. The eigenvectors of Δ Y will represent the optimal d dimensional Euclidean embedding of the vertices of a mesh.

[0083] For a mesh, Δ can be efficiently computed as follows. Assume that a mesh with N N vertices V , area F and edges​E of the grid. In this case, the size of Δ is N N The first i row of Δ describes the adjacency information of the first i vertex with respect to the vertices it is connected to. φ should be an arbitrary function on the grid. Then, the application of the discrete LB operator is mapped onto Δφ in the case of this function. The first i element of this function can be described by

[0084] (7)

[0085] Figure 4 The limits for the angles and are elucidated.

[0086] The sum of the cotangent expressions serves as the connection weight w ij According to one embodiment, these weights that occur in (7), that is, the weights of the LB operator when applied to the grid, are used as weights for determining D and A of equation (2).

[0087] It should be noted that due to negative connection weights w ij may occur, in particular in the case of angles that are significantly larger than the other angles (bad surfaces). In order to overcome this problem, the connection weights can be approximated by Edge Flipping.

[0088] The descriptor generation method described above treats each vertex uniquely. That is, each vertex is assigned to a unique descriptor. However, an object can be symmetric and thus assigning unique descriptors to seemingly identical vertices would lead to an asymmetric embedding.

[0089] In order to solve this problem, according to various embodiments, the intrinsic symmetry of a shape is detected and a symmetric embedding is compressed such that symmetric vertices are mapped onto the same descriptor. A shape can be said to have an intrinsic symmetry if the eigenfunctions of the Laplace-Beltrami operator appear symmetrically in Euclidean space. In other words, if the Euclidean embedding (descriptor space) of the symmetric features of a mesh, point cloud, etc. that preserve the geodesic distances between each vertex of a compact manifold exhibits Euclidean symmetry, then these symmetric features are detected. A manifold has an intrinsic symmetry if there exists a homeomorphism T that preserves the geodesic distances between each vertex of the compact manifold.

[0090] For the compression of symmetric descriptors, so-called globally intrinsic symmetry invariant functions (GISIF) can be used. In the assumption of a globally intrinsic symmetry homeomorphism where g denotes the geodesic distance and a function on the manifold f holds for each point on the manifold p if:

[0091] (8)

[0092] then f is a GISIF. For example, on a torus, the homeomorphism should be an arbitrary rotation around the z-axis. This means: if f is a GISIF, then this f must remain invariant with respect to the rotation.

[0093] It can also be shown that such a GISIF is the sum of the squared components of the eigenvector of a point in case of identical eigenvalues , that is

[0094] 。

[0095] This is in line with the analysis above for identical eigenvalues, which is a necessary condition for a symmetric embedding. Since in practice identical eigenvalues rarely exist due to numerical limitations, a heuristic can be used in case the eigenvalues are considered identical if they lie in the same ε-range (with small ε), that is, if the difference of the eigenvalues is less than a predefined threshold, e.g. 0.1% or 0.01%. Since only once for a given object the symmetric dimension has to be found, this can be performed manually.

[0096] For example, the first 7 eigenvalues for the eigenvalue decomposition of a torus should be as follows:

[0097]

[0098] The GISIF embedding in without considering trivial solutions, is then given by

[0099]

[0100] ​In the case of multiple objects, this can be represented as multiple separate connected graphs. In this case, the adjacency matrix is block-diagonal. The symmetric positive definite Laplacian will again have orthogonal eigenvectors. The result of the eigenvalue decomposition differs from the single graph embedding case in two ways: First, the non-decreasing eigenvalues will be the unsorted embedding errors of all objects. Second, the eigenvectors will have zero entries, since the corresponding eigenvalues remain orthogonal. This means that each dimension of the descriptor space will only correspond to one object embedding. Moreover, these dimensions are ordered with respect to the embedding error of the corresponding object. Thus, if a 3-dimensional embedding of two objects should be generated, d = 8 is used, since there are two trivial solutions corresponding to λ = 0.

[0101] This simple approach handles multiple objects independently, while there can be suboptimal approaches that still provide quite good embeddings with lower d

[0102] In case the pose of the object is considered, a target image can be generated by projecting the descriptors onto the image plane. A single descriptor mapped onto the descriptor space or descriptor space image random noise can be used as non-object (background).

[0103] To improve the robustness of the trained network 200, image augmentation methods such as domain randomization or perturbations like Gaussian soft focus, cropping or dropout can be applied.

[0104] If now the machine learning model 112, e.g. the neural network 200, is trained for mapping camera images of the object 113 onto descriptor images, the following approach can be taken in order to determine the pick-up pose of the object 113 in an unknown orientation.

[0105] First, a plurality of reference points on the object 113 is selected p i , i = 1, …, N and the descriptors of these reference points are determined. This can be achieved by taking camera images of the object 113, selecting reference pixels on the object u i , v i and mapping the camera images onto descriptor images by the neural network 200. Then, the descriptors in the descriptor images at the positions given by the positions of the reference pixels are the descriptors of the reference points, i.e. the descriptors of these reference points are d i ​​I d ( u i , v i ), wherein I d = f ( I; theta ) is a descriptor image, wherein f is a mapping (from camera image to descriptor image) realized by a neural network, I is a camera image and Theta are weights of the machine learning model 200.

[0106] If now the object 113 is in an unknown position, a camera image I neu is taken again and the belonging descriptor image is determined by means of the machine learning model I d neu = f ( I neu ; theta ) is searched for in this new descriptor image now as close as possible to the descriptor of the reference image d i , for example by

[0107] for all i = 1, …, N , ( u i , v i )* = argmin ui , vi || I d neu ( u i ,v i ) - d i ||2 2 .

[0108] From these reference points in the descriptor image I d neu and by this correspondingly in the new camera image I neu , the position u i , v i of these reference points in the three-dimensional space is determined. For example, with the camera imageI neu a depth image (or camera image I neu with a depth channel, e.g. the camera image is an RGBD image), so that the three-dimensional position of the i-th reference point u i , v i ) can be determined from p i the three-dimensional position of the i-th reference point u i , v i by projecting the depth value at position

[0109] If the positions of a plurality of reference points in space are known, it is possible to determine the pick-up pose therefrom, as shown in Figure 5

[0110] For example, two reference points p 1 and p2 on the object 500 are determined in space, and both are linearly combined, e.g. their average is taken, in order to specify the work point 504. In order to define the grasp direction, a first axis 501 with direction p 1 and p 2 is specified, a second axis 502 through the work point 504 is arbitrarily specified, e.g. in the z-axis direction of the camera 114 or in the direction of an axis of the work cell coordinate system. A third axis 503 through the anchor point 504 can be calculated by the vector product of the direction vector of the first axis 501 and the direction vector of the second axis 502. These three axes 501 to 503 and the anchor point 504 define the pick-up pose of the object 500. Then, the robot can be controlled so that it holds a handle of the object 500 which extends in the direction of the first axis. The reference points p 1 and p2 are specified, for example, so that they extend along the handle, i.e. along the elongated portion of the object which is suitable for grasping, as shown.

[0111] Similarly, three or more reference points can be arranged on the grasp face of the object so that from the positions of the reference points the full 6D pick-up pose of the object can be determined or also the direction of the grasp face on which the object is grasped.

[0112] ​It should be noted here that the gripper does not necessarily have to have a tongs shape, but for example can also have a suction device in order to suck onto the object on a suitable surface and thereby pick up the object. In order for the suction device to reach the correct pose, it can for example in this case be desirable to determine a pick-up pose which indicates the direction and position of the surface of the object which is suitable for sucking onto. This can for example be achieved by determining an anchor point and a plane normal vector in the anchor point.

[0113] More than three reference points can also be used in order to determine the pick-up pose, for example in order to reduce errors by averaging.

[0114] In summary, according to various embodiments, a method is provided as Figure 6 is set out in the detailed description.

[0115] Figure 6 A method for controlling a robot to pick up an object in different orientations is shown.

[0116] In 601, a plurality of reference points is prescribed on the object such that the positions of the reference points in three-dimensional space define a pick-up pose of the object.

[0117] In 602, a first camera image of the object in a known position of the object is received such that the positions of the reference points in the first camera image are known.

[0118] In 603, the first camera image is mapped onto a first descriptor image by means of a machine learning model which is trained to assign object points visible in a camera image to a descriptor independently of the position of the object points in the camera image.

[0119] In 604, the descriptors of the reference points from the first descriptor image are identified by reading the first descriptor image at the known positions of the reference points.

[0120] In 605, a second camera image of the object in an unknown position in which the object is to be picked up is received.

[0121] In 606, the second camera image is mapped onto a second descriptor image by means of the machine learning model.

[0122] In 607, the identified descriptors of the reference points are searched for in the second descriptor image.

[0123] In 608, the positions of the reference points in three-dimensional space in the unknown position are determined from the found positions of the descriptors of the reference points in the descriptor image.

[0124] In 609, a pick-up pose of the object is determined for the unknown orientation in dependence on the determined positions of the reference points in the three-dimensional space.

[0125] In 610, the robotic device is controlled to pick up the object in dependence on the determined pick-up pose.

[0126] According to various embodiments, in other words, the positions of the plurality of reference points are identified in a camera image showing the object in the unknown orientation in dependence on the descriptors of the plurality of reference points, and the positions of the plurality of reference points in the three-dimensional space are determined in dependence on the positions of the plurality of reference points in the camera image. Then, the pick-up pose (e.g. the direction of one or more axes for grasping the object) is determined in dependence on the positions of the plurality of reference points in the three-dimensional space.

[0127] “Pick-up” is to be understood as grasping with a gripper, for example. However, other types of fastening mechanisms can also be used, e.g. a suction device for sucking the object. Moreover, “pick-up” is not necessarily to be understood such that only the object is moved, e.g. a component on a larger structure can also be picked up and bent without being detached from the larger structure.

[0128] The machine learning model is a neural network, for example. However, other machine learning models which are correspondingly trained can also be used.

[0129] According to various embodiments, the machine learning model assigns pixels of the object (in the image plane of the respective camera image) to descriptors. This can be seen as an indirect encoding of the surface topology of the object. This connection between descriptors and surface topology can be made explicit by rendering in order to map the descriptors onto the image plane. It should be noted that the descriptor values on the surface of the object model (that is to say, points other than the vertices) can be determined by means of interpolation. If, for example, a surface is given by 3 vertices of the object model with corresponding descriptor values y1, y2, y3, then at an arbitrary point of the surface the descriptor value y can be calculated as a weighted sum of these values . In other words, the descriptor values are interpolated at the vertices.

[0130] In order to generate image pairs for training data for a machine learning model, e.g. map an image (e.g. an RGB image) of an object (or multiple objects) including an object with a known 3D (e.g. CAD) model and a known pose (in a global, that is to say world, coordinate system) onto a (dense) descriptor image, the descriptor image is optimal in that it is generated by searching for descriptors that minimize the deviation in the geometric properties (in particular the proximity of the points of the object) between the object model and its representation (embedding) in descriptor space. In practical use, due to the search being restricted to a certain search space, the theoretical optimal solution for minimization is usually not found. However, an estimate of the minimum value is determined within the constraints of practical application (available computational precision, maximum number of iterations, etc.).

[0131] Thus, by performing a minimization process on the sum of the distances between the descriptors of a connected pair of vertices, weighted by the weight of the edge between the pair of vertices, the descriptors of the vertices are searched, wherein each descriptor for a respective one of the vertices is searched.

[0132] Each training data image pair comprises a training input image of an object and a target image, wherein the target image is generated by projecting the descriptors of the vertices that are visible in the training input image onto the training input image plane in the pose that the object has in the training input image.

[0133] The images are used together with their associated target images for supervised training of a machine learning model.

[0134] Thus, the machine learning model is trained to recognize explicit features of an object (or multiple objects). By means of evaluation of the machine learning model, this information can be used in real time for various applications in the field of robot control, for example to predict object grasping poses for assembly. It should be noted that the supervised training method enables explicit encoding of symmetrical information.

[0135] Figure 6The methods described herein can be performed by one or more computers comprising one or more data processing units. The term "data processing unit" can be understood as any kind of entity capable of processing data or signals. For example, these data or signals can be processed in accordance with at least one, that is, one or more than one, specific function performed by the data processing unit. The data processing unit can comprise or be formed by analog circuits, digital circuits, mixed signal circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphical processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), or any combination or sub-combination thereof. Any other means for implementing the respective functions described in more detail below can also be understood as a data processing unit or logic circuit means. It is readily appreciated that one or more of the method steps described in detail herein can be implemented by (for example, realized by) the data processing unit by one or more specific functions performed by the data processing unit.

[0136] Various embodiments can receive and use sensor signals of various sensors, such as e.g. RGB cameras, video, radar, LiDAR, ultrasound, thermal imaging technology, etc., in order to obtain sensor data of a display object, for example. Embodiments can be used to generate training data and train a machine learning system, e.g. for autonomous control of a robot, e.g. a robot manipulator, in order to implement different manipulation tasks in different scenarios. In particular, embodiments can be applied to control and monitor the implementation of manipulation tasks, e.g. in assembly lines.

[0137] While specific embodiments are illustrated and described herein, it will be appreciated that various modifications and / or equivalent arrangements can be made therein without departing from the spirit and scope of the disclosure. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this application be limited only by the claims and the equivalents thereof.

Claims

1. A method for controlling a robot to pick up an object in a different pose, the method having: specifying a plurality of reference points on the object such that the poses of the reference points in three-dimensional space define a pick-up pose of the object; receiving a first camera image of the object in a known pose of the object such that the positions of the reference points in the first camera image are known; mapping the first camera image onto a first descriptor image by means of a machine learning model trained to assign object points visible in a camera image to descriptors independently of the positions of the object points in the camera image; identifying descriptors of the reference points in the first descriptor image by reading the first descriptor image at the known positions of the reference points; receiving a second camera image of the object in an unknown pose in which the object should be picked up; mapping the second camera image onto a second descriptor image by means of the machine learning model; searching for the identified descriptors of the reference points in the second descriptor image; determining the positions of the reference points in three-dimensional space in the unknown pose from the found positions of the descriptors of the reference points in the descriptor image; determining the pick-up pose of the object for the unknown pose from the determined positions of the reference points in three-dimensional space; and controlling a robot device to pick up the object in accordance with the determined pick-up pose.

2. The method according to claim 1, wherein the positions of the reference points in three-dimensional space in the unknown pose are determined from the found positions of the descriptors of the reference points in the second descriptor image in such a way that depth information of the positions of the reference points in the second camera image corresponding to the positions of the reference points in the second descriptor image is projected into the three-dimensional space.

3. The method according to claim 1 or 2, wherein the reference points have at least two reference points specified such that the at least two reference points are arranged along an elongated portion of the object, and wherein the pick-up pose of the object for the unknown pose is determined by determining an axis extending through the reference points.

4. The method according to claim 1 or 2, wherein the reference points have at least three reference points specified such that the at least three reference points are arranged on a planar surface of the object, and wherein the pick-up pose of the object for the unknown pose is determined by determining a plane extending through the reference points.

5. The method according to claim 1 or 2, wherein the training of the machine learning model has: obtaining a 3D model of the object, wherein the 3D model comprises a mesh of vertices; determining descriptors for each vertex of the mesh; generating training data image pairs, wherein each training data image pair comprises a training input image showing the object and a target image, and wherein the generation of the target image comprises: determining a vertex position of the vertex of the object model of the object in the training input image, and for each determined vertex position in the training input image, assigning the determined descriptor for the vertex at the vertex position to a position in the target image; and training the machine learning model by supervised learning using the training data image pairs as training data.

6. The method of claim 5, wherein the generation of the training data image pairs comprises: obtaining a plurality of images of the object in different poses; and generating training data image pairs from each obtained image by generating a target image for the obtained image.

7. The method of claim 5, comprising: determining a vertex position of the vertex of the object model of the object in the training input image according to the respective pose of the object in the training input image.

8. The method of claim 5, wherein, the vertices of the 3D model are connected by edges, wherein each edge has a weight, the weight specifying a closeness of two vertices in the object connected by the edge, and wherein the descriptor of each vertex of the mesh is determined by searching for a vertex whose descriptor minimizes a sum of distances between vertex pair descriptors of connected vertex pairs, weighted by the weight of the edge between the vertex pair.

9. A robot control device, the robot control device being set up to perform the method according to any one of claims 1 to 8.

10. A computer program product having a computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images