Apparatus and method for controlling a robot that picks up objects in various postures, as well as a computer program and a computer-readable storage medium.

Abstract

To provide a method for controlling a robot for picking up an object in various pose situations.SOLUTION: A method includes the steps of: defining a plurality of reference points on an object; mapping a first camera image of the object in a known pose situation onto a first descriptor image; identifying descriptors of the reference points from the first descriptor image; mapping a second camera image of the object in an unknown pose situation onto a second descriptor image; searching for the identified descriptors of the reference points in the second descriptor image; acquiring the positions of the reference points in a three-dimensional space of the unknown pose situation from the found positions; and acquiring a pickup pose of the object for the unknown pose situation from the acquired positions of the reference points.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an apparatus and a method for controlling a robot that picks up objects in various posture situations.

Background Art

[0002] In order to enable flexible manufacturing or processing of an object by a robot, it is desirable that the robot be able to handle the object regardless of the situation within the working space of the robot where the object is placed. Therefore, the robot needs to be able to recognize the posture situation of the object at least as long as it is related to the method of picking up (for example, gripping) the object. That is, the robot needs to be able to determine the pickup pose (for example, gripping pose) of the object in order to properly orient its end effector (for example, with a gripper) according to the posture situation and move it to an appropriate position. Correspondingly, a control method for a robot that picks up objects in various posture situations is desired. to

Summary of the Invention

Means for Solving the Problems

[0003] ​According to various embodiments, a method for controlling a robot that picks up objects in various pose situations, comprising: determining a plurality of reference points on an object, whereby the pose situation of the reference points in three-dimensional space determines the pick-up pose of the object; receiving a first camera image of the object at a known position of the object, whereby the positions of the reference points in the first camera image are made known; mapping the first camera image to a first descriptor image using a machine learning model trained to assign object points visible in the camera image to descriptors without depending on their positions in the camera image; identifying the descriptors of the reference points from the first descriptor image by reading the first descriptor image at the known positions of the reference points; receiving a second camera image of the object in an unknown pose situation in which the object is to be picked up; mapping the second camera image to a second descriptor image using the machine learning model; searching for the identified descriptors of the reference points within the second descriptor image; determining the positions of the reference points in three-dimensional space in the unknown pose situation from the positions of the descriptors of the reference points detected within the descriptor image; determining the pick-up pose of the object for the unknown pose situation from the positions of the reference points determined in three-dimensional space; and controlling the robot that picks up the object to in accordance with the determined pick-up pose. A method is provided that includes these steps.

[0004] The method described above enables safe pick-up (e.g., grasping) of an object for any pose situation of the object. A camera image of an object in an unknown pose situation is sufficient to determine the pick-up pose of the object. In particular, an arrangement configuration with multiple cameras encompassing multiple fields of view is not necessary. The machine learning model can be flexibly trained to be usable for various objects. In that case, all that is required is to determine a plurality of reference points for each object and obtain the descriptors of those reference points.

[0005] In the following, various examples are presented.

[0006] Example 1 is a method for controlling a robot that picks up objects at various locations as described above.

[0007] Example 2 is the method of Example 1, wherein the position of a reference point in three-dimensional space in an unknown posture situation is determined from the detected position of the reference point descriptor in the second descriptor image by projecting depth information about the position of the reference point in the second camera image, which corresponds to the position of the reference point in the second descriptor image, into three-dimensional space.

[0008] This makes it possible to determine the pickup pose using individual camera images and associated depth information (e.g., RGBD images).

[0009] Example 3 is the method of Example 1 or 2, wherein the reference points have at least two reference points determined to be positioned along an elongated sub-section of the object, and the pickup pose of the object for an unknown orientation is determined by determining the axis extending through the reference points.

[0010] In this way, a pickup pose can be determined that enables robot control for grasping an elongated sub-section. For example, the gripper in the robot's end effector is controlled so that the gripper claws lie in a plane perpendicular to the axis extending through a reference point, and then the gripper is closed.

[0011] Example 4 is one of the methods of Examples 1 to 3, wherein the reference points have at least three reference points determined to be positioned on the plane of the object, and the pickup pose of the object for an unknown orientation is determined by determining the plane extending through the reference points.

[0012] In this way, a pickup pose can be determined that enables control of a robot for grasping a surface (e.g., for suction or lifting). For example, a suction device in the robot's end effector is controlled so that the suction opening or nozzle of the suction device is perpendicular to a plane extending through a reference point, and then activated.

[0013] Example 5 is one of the methods of Examples 1 to 4, wherein the steps of training a machine learning model include: obtaining a 3D model of an object including a grid of vertices; determining a descriptor for each vertex of the grid; generating a pair of training data images, each pair of training data images including a training input image and a target image representing an object, wherein generating the target image includes a substep of determining the vertex positions of the vertices of the object model of the object having vertices in the training input image; and for each specific vertex position in the training input image, assigning the descriptor determined for the vertex at that vertex position to the position in the target image; and training the machine learning model by supervised learning using the pair of training data images as training data.

[0014] This training method allows machine learning models (such as neural networks) to be trained to make more accurate predictions (i.e., descriptor decisions) than when using self-supervised learning (i.e., enabling a wider range of network applications). Furthermore, it provides greater flexibility for fitting machine learning models, thereby enabling application to various problems and reducing the requirements for training data (e.g., the amount of training data required).

[0015] Example 6 is the method of Example 5, wherein the step of generating training data image pairs includes the steps of acquiring multiple images of an object in different poses, and generating training data image pairs from each acquired image by generating a target image for each acquired image.

[0016] This makes it possible to train a machine learning model (for example, in a robot with a robot control device that implements a machine learning model) to recognize the topology of an object regardless of its orientation within the robot's workspace.

[0017] Example 7 is a method of Example 5 or 6, comprising the step of determining the vertex positions of the vertices of an object model of an object having vertices in a training input image, from the respective poses of the object in the training input image (e.g., in the camera coordinate system).

[0018] This allows for precise determination of vertex positions, which in turn enables the creation of accurate target images for supervised training.

[0019] Example 8 is one of the methods of Examples 5 to 7, wherein the vertices of a 3D model are joined by edges, each edge having a weight that identifies the proximity of two vertices joined by the edge in the object, and the step of determining a descriptor for each vertex of the grid is performed by searching for a descriptor for a vertex, which minimizes the sum of the distances between descriptors of a vertex pair across the joined vertex pair, weighted by the edge weight between the vertex pair.

[0020] Simply put, descriptors are determined using the most optimal method. This ensures high consistency in the descriptors. In addition, good expressiveness of the descriptor mapping is achieved.

[0021] Example 9 is a robot control device configured to implement the method according to any one of Examples 1 to 8.

[0022] Example 10 is a computer program that includes instructions causing the processor to perform the method according to any one of Examples 1 to 8 when the instructions are executed by the processor.

[0023] Example 11 is a computer-readable storage medium that stores instructions causing a processor to perform any one of the methods of Examples 1 to 8 when the instructions are executed by the processor.

[0024] In the drawings, generally, the same reference numerals represent the same parts throughout the various drawings. These drawings are not necessarily to scale, but rather the focus is on generally illustrating the principles of the present invention. Various embodiments will be described in the following specification with reference to the following drawings. [Brief explanation of the drawing]

[0025] [Figure 1] A diagram showing a robot. [Figure 2] This figure shows the training of a neural network according to one embodiment. [Figure 3] This figure shows an exemplary embedding of a 4-node graph for descriptor determination. [Figure 4] This figure shows the definition of angles at the vertices of a 3D model for edge weighting determination using the Laplace-Belt-Lammy operator. [Figure 5] This figure shows a method for training a machine learning model to recognize the object topology of an object from an image of the object, according to one embodiment. [Figure 6] This figure shows a method for controlling a robot that picks up objects in various poses. [Modes for carrying out the invention]

[0026] The following detailed description refers to specific details for illustrative purposes and accompanying drawings illustrating embodiments of the present disclosure that can carry out the invention. Other embodiments may be used, and structural, logical, and electrical modifications may be made without departing from the scope of protection of the invention. Since some embodiments of the present disclosure can be combined with one or more other embodiments of the present disclosure to constitute new embodiments, the various embodiments of the present disclosure are not necessarily mutually exclusive.

[0027] The following sections will provide more detailed explanations of various examples.

[0028] Figure 1 shows robot 100.

[0029] Robot 100 includes a robotic arm 101, for example, an industrial robotic arm for manipulating or mounting a workpiece (or one or more other objects). This robotic arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 on which these manipulators 102, 103, 104 are supported. The term “manipulator” refers to the movable parts of the robotic arm 101 whose manipulation enables physical interaction with the surrounding environment to perform tasks, for example. For control, robot 100 includes a (robot) control device 106 designed to carry out interactions with the surrounding environment according to a control program. The last element 104 of the manipulators 102, 103, 104 (which is furthest from the support 105) is also referred to as the end effector 104 and may include one or more tools, for example, a welding torch, a gripper, a painter, etc.

[0030] Other manipulators 102 and 103 (located near the support base 105) can constitute a positioning device, and together with the end effector 104, the robot arm 101 is equipped with the end effector 104 at its end. The robot arm 101 is a mechanical arm that can provide functions similar to a human arm (and may be equipped with a tool at its end).

[0031] The robot arm 101 may include joint elements 107, 108, and 109 that connect the manipulators 102, 103, and 104 to each other and further connect to the support base 105. The joint elements 107, 108, and 109 may include one or more joints, each of which can provide rotatable motion (i.e., rotational motion) and / or translational motion (i.e., moving motion) of the associated manipulators relative to each other. The motion of the manipulators 102, 103, and 104 can be initiated using actuators controlled by the control device 106.

[0032] The term "actuator" can be understood as a component designed to act on a mechanism or process in response to its drive. An actuator can translate commands (so-called initiations) created by the control device 106 into mechanical motion. For example, an actuator such as an electromechanical transducer may be configured to convert electrical energy into mechanical energy in response to its drive.

[0033] The term "control unit" can be understood as any type of logically implemented item, which may include, for example, a circuit and / or processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and, in this embodiment, capable of outputting commands to an actuator. The control unit may consist, for example, of program code (e.g., software) to control the operation of the system (in this example, a robot).

[0034] In this example, the control device 106 includes one or more processors 110 and a memory 111 that stores code and data, and the processors 110 control the robot arm 101 based on this code and data. According to various embodiments, the control device 106 controls the robot arm 101 based on a machine learning model 112 stored in the memory 111.

[0035] According to various embodiments, the machine learning model 112 is designed and trained so that the robot 100 can recognize the pickup pose of an object 113 placed, for example, within the workspace of the robot arm 101. This means that the robot 100 recognizes how it can pick up the object 113, i.e., how its end effector 104 must be oriented, and where it must be moved in order to pick up (e.g., grasp) the object 113. The pickup pose is understood to contain sufficient information for pickup, i.e., information about the orientation and position of the object 113 that is sufficient to determine how the object 113 can be grasped. The pickup pose does not necessarily have to contain complete orientation information about the object 113, because, for an object 113 with a rotationally symmetric portion, how the rotationally symmetric portion rotates around its axis of rotation may be irrelevant to grasping, for example.

[0036] The robot 100 may be equipped with, for example, one or more cameras 114 that enable it to capture images of its workspace. These cameras 114 may be attached to, for example, a robot arm 101, so that the robot can take images of an object 113 from various viewpoints by moving the robot arm 101 around.

[0037] According to various embodiments, the control device 106 implements a machine learning model 112 for object recognition, and its output is used to determine the pickup pose of the object 113.

[0038] An example of such a machine learning model 112 for object recognition is a dense object network. This dense object network maps an image (e.g., an RGB image provided by camera 114) to an arbitrary-dimension (dimension D) descriptor space image.

[0039] A dense object network is a neural network trained using self-supervised learning to output a descriptor space image for each image input. However, the effectiveness of this approach depends heavily on the quality of the collected training data and the sensors involved (e.g., camera 114). Furthermore, interpreting network predictions can sometimes be difficult.

[0040] In various embodiments, an approach is used to recognize an object and its position, assuming that a 3D model of the object (e.g., a CAD (Computer-Aided Design) model) is known, which is a typical case for industrial assembly and machining tasks. In various embodiments, techniques for reducing nonlinear dimensions are used to compute the optimal target image for training the input image for the neural network. Thus, instead of using self-supervised training of the neural network, supervised training of the neural network is used in various embodiments. It is also possible to capture an RGBD image (RGB + depth information) of the object, from which the 3D model of the object can also be determined.

[0041] According to one embodiment, data collection is first performed to generate training data for training a machine learning model 112. In particular, for example, registered RGB (red-green-blue) images are collected. Registered images, as used herein, mean RGB images with known intrinsic and extrinsic camera values. For example, in a real-world scenario, a camera 114 attached to a robot (e.g., a camera attached to the robot's wrist) is used to scan objects while a robot (e.g., a robotic arm 101) moves around. Other extrinsic estimation methods, such as ChArUco markers, can also be used. That is, objects can be placed in various positions and orientations relative to the ChArUco board, and images of the arrangement configuration (of the ChArUco board and objects) can be captured. In a simulated scenario, a photographically realistic RGB image is used using known object orientations.

[0042] After collecting RGB images, a target image of the RGB images is rendered for supervised training of the neural network.

[0043] It is assumed that the orientation of each object in world coordinates is known for each collected RGB image. This is straightforward in simulated scenarios, but in real-world scenarios, manual adjustments are necessary, such as positioning objects in predetermined locations. RGBD images can also be used to determine the object's position.

[0044] Along with this information, a descriptor image (i.e., a training output image, also called a target image or ground-truth image) is rendered for each RGB image (i.e., a training input image) using, for example, a vertex descriptor calculation method as described below.

[0045] When a target image is generated for each RGB image, that is, when pairs of RGB images and target images are formed, these training input images and their associated target image pairs can be used as training data to train a neural network, as shown in Figure 2.

[0046] Figure 2 shows the training of a neural network 200 according to one embodiment.

[0047] Neural Network 200 is a fully convolutional network that maps an h×w×3 tensor (input image) to an h×w×D tensor (output image).

[0048] This includes multiple stages 204 of a convolutional layer, followed by a pooling layer, an upsampling layer 205, and a skip coupling layer 206 that combines the outputs of different layers.

[0049] For training, the neural network 200 receives a training input image 201 and outputs an output image 202 having pixel values ​​in descriptor space (e.g., color components by descriptor vector components). The training loss is calculated between the output image 202 and the target image 203 associated with the training input images. This can be done over a batch of training input images, and the training loss can be averaged over the training input images, and the weights of the neural network 200 are trained using stochastic gradient descent with the training loss. The training loss calculated between the output image 202 and the target image 203 is, for example, an L2 loss function (to minimize the pixel-wise least squares error between the target image 203 and the output image 202).

[0050] The training input image 201 shows vectors in descriptor space, including object and target images, as well as multiple output images. These vectors in descriptor space can be mapped to colors such that the output image 202 (and target image 203) resembles a heatmap of the object.

[0051] A vector in descriptor space (also called a dense descriptor) is a d-dimensional vector assigned to each pixel in each image (for example, each pixel in input image 201, assuming that input image 201 and output image 202 have the same dimensions) (where d is, for example, 1, 2, or 3). The dense descriptor implicitly encodes the surface topology of an object shown in input image 201 invariant to its orientation or camera position.

[0052] Given a 3D model of an object, the optimal and unique descriptor vector (in the Riemannian sense) for each vertex of the 3D model of the object can be analytically determined. According to various embodiments, these optimal descriptors (or estimates of these descriptors determined by optimization) are used to generate a target image for a registered RGB image, which leads to the complete supervised training of the neural network 200. Additionally, the descriptor space becomes explainable and optimal regardless of the dimension d of the selected descriptors.

[0053] In the following, the 3D model will be used as follows:

number

number

number

[0054] According to the approach described below, the embedding of a 3D object model in Euclidean space into descriptor space is determined in order to preserve the distances between vertices (e.g., geodesic distances).

[0055] For dimensionality reduction via the Laplace operator, the set of points

number

[0056] This goal is to find an embedding in d dimensions

Number

Number

[0057] The optimization problem (1) is

Number

Number

Number

Number

[0058] Furthermore, the first eigenvector (Y T The first column of the vector is equal to 1 (a vector of only 1s), which is a trivial solution that maps each vertex to a point. Additionally, any two eigenvectors are orthogonal to each other. From the solution to the eigenvalue problem, we obtain N eigenvalues ​​and their corresponding eigenvectors of dimension N. However, in practice, only the first d eigenvectors corresponding to the lowest eigenvalues ​​are used (except for the trivial solution).

[0059] Therefore, the i-th column of Y is R d This is an embedding of node i to , where each row represents the embedding of each point in various orthogonal dimensions.

[0060] Figure 3 shows an exemplary embedding of a 4-node graph.

[0061] Eigenvalues ​​are of significant importance regarding the optimality of the embedding. Optimal embedding Y * In this case, the following constraints apply:

number

number

number

[0062] Since λ0=0, that is, if all vertices of an object are mapped to a single point, the embedding error becomes 0. This is because the distance y between all points is 0. This is not useful for practical purposes, and therefore the first eigenvalues ​​and eigenvectors are omitted. Using d=2 corresponds to mapping each point x to a line, where λ1 is the corresponding embedding error, and so on. Since the eigenvectors are orthogonal to each other, increasing d adds a new dimension to the embedding. This is intended to minimize the error of the new orthogonal dimension. A similar effect can be seen in equation (3). Here,

number

[0063] In some cases, the subsequent eigenvalues ​​will be the same. That is, λ i =λ i+1 =λ i+2 =…(See the example in Figure 3 where the eigenvalues ​​for d=2 and d=3 are the same). This leads to some information about symmetry, where there exist multiple orthogonal dimensions with the same embedding error. In fact, in the example of the 4-node graph in Figure 3, if the graph is fully joined, the embeddings for each dimension are symmetric, and all eigenvalues ​​are identical except for the trivial solution.

[0064] The graph embedding approach described above can be directly applied to grids, point clouds, and the like. For example, the K-Nearest Neighbors (KNN) algorithm can be used to form local connections between vertices and create an adjacency matrix. This approach is sufficient to construct a graph Laplacian and compute the embedding for each vertex. Nevertheless, this approach is intrinsically based on the Euclidean distance function and discovery methods, which does not necessarily take into account the Riemannian geometry that underlies 3D object models. For example, some edges may extend through an object or connect vertices of a grid that are not adjacent. Even a few improper entries in the adjacency matrix can lead to a decrease in embedding ability. Therefore, according to one embodiment, when processing a model, it is guaranteed that the geodesic distance between any two vertices is appropriate or has the smallest possible approximation error.

[0065] Generally, object models such as grids and point clouds are

number

[0066] However, the LB operator can be approximated for known discrete manifolds such as grids. This provides a very efficient and simple computational framework when dealing with grids, point clouds, etc. Since the Riemann equivalent of the Laplace operator is Laplace-Tramy, the above embedding approach can also be directly applied to Δ. The eigenvector Y of Δ represents the optimal d-dimensional Euclidean embedding of the grid vertices.

[0067] Δ can be efficiently calculated in the case of a grid as follows. Here, we assume that a grid with N vertices V, a plane F, and edges E is given. In this case, the size of Δ is N × N. The i-th row of Δ describes the adjacency information of the i-th vertex to its joined vertices. Let φ be an arbitrary function on the grid, then the application of the discrete LB operator on this function is mapped to Δφ. The i-th element of this function can be written as follows: that is,

number

[0068] Figure 4 shows the angle

number

number

[0069] Total of cotangents

number

[0070] When α > π / 2 ⇒ cot(α) < 0, the bond weight is negative wij This can occur, and caution is needed, especially when one angle is significantly larger than the others (i.e., when the plane is not good). To overcome this problem, the bond weights can be approximated by edge flipping.

[0071] The descriptor generation approach described above treats each vertex uniquely; that is, each vertex is assigned one unique descriptor. However, the object may be symmetric, in which case assigning unique descriptors to seemingly identical vertices results in an asymmetric embedding.

[0072] To address this problem, according to various embodiments, the intrinsic symmetry of a shape is detected, and when the symmetry embedding is compressed, symmetric vertices are mapped to the same descriptor. If the eigenfunctions of the Laplace-Belt-Lamy operator appear symmetric in Euclidean space, it is shown that the shape has intrinsic symmetry. In other words, symmetric features such as grids and point clouds are detected if their Euclidean embedding (descriptor space) that preserves their geodesics exhibits Euclidean symmetry. A compact manifold exhibits intrinsic symmetry if there is a homeomorphism T that preserves the geodesic distances between each vertex of the manifold.

[0073] For the compression of symmetry descriptors, the so-called global intrinsic symmetry invariant function (GISIF) can be used. Assuming a global intrinsic symmetry homeomorphism as follows,

number

[0074] Furthermore, the same eigenvalue λ i =λ i+1 =…=λ i+N In this case, it can be shown that this type of GISIF is the sum of the squares of the eigenvector components of the point. That is,

number

[0075] This is consistent with the analysis of identical eigenvalues ​​described above, which is a requirement for symmetric embeddings. In practice, identical eigenvalues ​​rarely occur due to numerical constraints, so the discovery method can be used where eigenvalues ​​are considered identical if they lie within the same ε-sphere (when ε is small), i.e., if their difference is less than a predetermined threshold, e.g., 0.1% or 0.01%. Since symmetric dimensions only need to be detected once for a given object, this can be done manually.

[0076] For example, the first seven eigenvalues ​​of the eigenvalue decomposition of a torus are as follows: λ 0-6 =[0.0,0.0175,0.0175,0.069,0.069,0.1525,0.1525]

[0077] Without considering the obvious solution,

number

[0078] In the case of multiple objects, this can be represented as multiple independently connected graphs. In this case, the adjacency matrix is ​​block-diagonal. The symmetric positive definite Laplace operator again has orthogonal eigenvectors. There are two differences in the result of the eigenvalue decomposition compared to the single-graph embedding case. First, the non-decreasing eigenvalues ​​can be the unordered embedding errors for all objects, and second, since the corresponding eigenvalues ​​remain orthogonal, the eigenvectors have zero terms. This means that each dimension of the descriptor space corresponds to only one object embedding. Furthermore, the dimensions are ordered by reference to the corresponding object embedding errors. Thus, when we need to generate a three-dimensional embedding of two objects, d=8 is used because there are two trivial solutions corresponding to λ=0.

[0079] While this simple approach treats multiple objects independently, there may be suboptimal methods that utilize the correlations between objects to provide reasonably good embeddings at a lower d.

[0080] Considering the object's orientation, the target image can be generated by projecting the descriptors onto the image plane. As a non-object (background), random image noise in descriptor space or individual descriptors mapped to the furthest points in descriptor space can be used.

[0081] To improve the robustness of the trained network 200, image augmentation methods such as domain randomization, or disturbances such as Gaussian blurring, cropping, or dropout can be applied.

[0082] Here, if a machine learning model 112, for example a neural network 200, has been trained to map camera images of object 113 to descriptor images, the following steps can be taken to determine the pickup pose of object 113 in an unknown pose situation.

[0083] First, on the object, there are multiple reference points p on object 113. ii=1,...N are selected, and descriptors for these reference points are obtained. This means that a camera image of object 113 is captured, and a reference pixel (u i ,v i )(therefore, the corresponding reference point of the object) is selected, and the camera image is mapped to the descriptor image by the neural network 200. Then, the descriptor at the position in the descriptor image given by the position of the reference pixel can be considered as the descriptor of the reference point. That is, the descriptor of the reference point is d i =I d (u i ,v i ) and here, I d =f(I;θ) is the descriptor image, f is the mapping (from camera image to descriptor image) implemented by the neural network, I is the camera image, and θ are the weights of the machine learning model 200.

[0084] Here, if object 113 is in an unknown attitude state, camera image I neu The image is captured, and a machine learning model is used to obtain the associated descriptor image I d neu =f(I neu θ) is obtained. In this new descriptor image, here, the d of the reference image is i A descriptor that is as close as possible to the descriptor of the following expression, (u i, v i ) * =argmin ui,vi ||I d neu (u i, v i )-d i ||2 2 However, for all i=1, ...N It is searched by.

[0085] The position of the reference point in 3D space is shown in descriptor image I. d neu (Therefore, the corresponding new camera image I) neu The position of the reference point (inside) that was thus determined or estimated (ui ,v i ) * It can be calculated from the camera image I. neu Along with this, a depth image is captured (or camera image I neu (It has depth channels, for example, RGBD images), therefore, (u i ,v i ) * From (the position (u i ,v i ) * (By projecting the depth value in the respective work area coordinate system) the i-th reference point p i The 3D position can be determined.

[0086] If the spatial positions of multiple reference points are known, the pickup pose can be determined from them, as shown in Figure 5.

[0087] For example, the positions of two reference points p1 and p2 on object 500 in space are determined, and these two are combined linearly, for example, their average value is taken to determine the anchor point 504. To define the gripping orientation, the first axis 501 is determined by the directions of p1 and p2, and the second axis 502 is arbitrarily determined by the anchor point 504, for example, in the z-axis direction of the camera 114 or the axis direction of the work area coordinate system. The third axis 503 passing through the anchor point 504 can be calculated by the vector product of the direction vector of the first axis 501 and the direction vector of the second axis 502. These three axes 501 to 503 and the anchor point 504 define the pickup pose for object 500. The robot can then be controlled to encompass the axis of object 500 extending in the first axis direction. Reference points p1 and p2 are determined, for example, to extend along the axis, that is, along an elongated subsection suitable for gripping an object, as shown in the figure.

[0088] Similarly, three or more reference points can be placed on the gripping surface of the object, and from these positions, the complete 6D pickup pose of the object or the orientation of the gripping surface capable of grasping the object can also be determined.

[0089] It should be noted that grippers do not necessarily have to be scissor-shaped; for example, they can have a suction device to pick up an object by attracting it to a suitable surface. To move the suction device into the correct position, it may be desirable to determine a pickup pose, for example, which indicates the orientation and position of the object's surface suitable for suction. This can be done, for example, by determining the anchor point and the plane normal vector within the anchor point.

[0090] To determine the pickup pose, it is also possible to use three or more reference points, for example, to reduce errors through averaging.

[0091] In summary, various embodiments provide a method as shown in Figure 6.

[0092] Figure 6 shows a method for controlling a robot that picks up objects in various orientations.

[0093] In 601, multiple reference points are determined on the object, and the orientation of these reference points in three-dimensional space determines the object's pickup pose.

[0094] In 602, a first camera image of the object in a known orientation is received, thereby making the position of the reference point in the first camera image known.

[0095] In 603, the first camera image is mapped to a first descriptor image using a machine learning model trained to assign visible object points within the camera image to descriptors independently of their positions within the camera image.

[0096] In 604, the descriptor of the reference point is identified from the first descriptor image by reading the first descriptor image at a known location of the reference point.

[0097] In 605, a second camera image of the object in an unknown orientation is received.

[0098] In 606, the second camera image is mapped to the second descriptor image using a machine learning model.

[0099] In 607, the descriptor of the identified reference point is searched for within the second descriptor image.

[0100] In 608, the position of the reference point in the three-dimensional space of an unknown pose is determined from the position of the reference point descriptor detected in the descriptor image.

[0101] In 609, the pickup pose of an object in an unknown posture state is determined from the position of a reference point found in three-dimensional space.

[0102] In the 610, a robot for picking up objects to However, it is controlled according to the requested pickup pose.

[0103] In various embodiments, in other words, the positions of multiple reference points are identified based on their descriptors in a camera image showing an object in an unknown pose, and their positions in three-dimensional space are determined from their positions in the camera image. Then, from the positions of the multiple reference points in three-dimensional space, a pickup pose (e.g., the orientation of one or more axes for grasping an object) is determined.

[0104] "Picking up" should be understood as meaning, for example, grasping with a gripper. However, other types of holding mechanisms can also be used, such as a suction device for picking up an object. Furthermore, it should be understood that "picking up" does not necessarily mean that the object moves on its own. For example, it is possible to pick up a component from a larger structure and bend it without separating it from the larger structure.

[0105] Machine learning models include, for example, neural networks. However, it is also possible to use other machine learning models that have been adequately trained.

[0106] According to various embodiments, a machine learning model assigns descriptors to the pixels of an object (within the image plane of each camera image). This can be considered an indirect encoding of the object's surface topology. This connection between descriptors and surface topology can be explicitly made by rendering to map the descriptors onto the image plane. Note that descriptor values ​​for the plane of an object model (i.e., points other than vertices) can be determined using interpolation. For example, if a face is given by three vertices of an object model with descriptor values ​​y1, y2, and y3, then at any point on this face, the descriptor value y can be calculated as the weighted sum of these values ​​w1·y1+w2·y2+w3·y3. In other words, descriptor values ​​are interpolated at the vertices.

[0107] To generate image pairs for training data for a machine learning model, for example, an image of an object (e.g., an RGB image) is mapped to a (dense) descriptor image that includes the object (or multiple objects) with a known 3D (e.g., CAD) model and a known pose coordinate system (global (i.e., world) coordinate system), which is optimal in the sense that it is generated by searching for descriptors that minimize the deviation in geometric properties (particularly the proximity of object points) between the object model and its representation (embedding) in descriptor space. In practical use, the theoretical optimal solution for minimization is generally not found because the search is limited to a specific search space. Nevertheless, the estimation of the minimum value is determined within the constraints of the actual application (e.g., available computational precision, maximum number of iterations).

[0108] Therefore, descriptors for vertices are searched by performing a minimization process on the sum of the distances between descriptors of vertex pairs across the combined vertex pairs, weighted by the edge weights between the vertex pairs, where each descriptor for each vertex of a vertex is searched.

[0109] Each training data image pair includes a training input image and a target image of an object, the target image being generated by projecting the descriptors of the visible vertices in the training input image onto the training input image plane according to the pose the object has in the training input image.

[0110] These images, along with their associated target images, are used for supervised training of machine learning models.

[0111] Therefore, machine learning models are trained to recognize the unique features of an object (or multiple objects). This information can be used in various applications of robot control, such as predicting object gripping postures for assembly, using real-time evaluation of the machine learning model. Note that in supervised training approaches, explicit encoding of symmetry information is possible.

[0112] The method in Figure 6 may be carried out by one or more computers, each containing one or more data processing units. The term “data processing unit” can be understood to mean any type of item that enables the processing of data or signals. For example, these data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. A data processing unit may include, or may consist of, analog circuits, digital circuits, composite signal circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof. Any other methods for implementing each of the functions described in more detail below can also be understood as data processing units or logic circuit units. Of course, one or more method steps described in detail herein can be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0113] Various embodiments can be used to receive and utilize sensor signals from various sensors, such as (RGB) cameras, video, radar, LiDAR, ultrasound, and thermal imaging products, in order to acquire sensor data indicating an object. These embodiments can be used to generate training data and train machine learning systems, for example, to autonomously control a robot, such as a robotic manipulator, or to accomplish various operational tasks under various scenarios. In particular, these embodiments are applicable, for example, to the control and monitoring of the execution of operational tasks on an assembly line.

[0114] While specific embodiments have been illustrated and described herein, it will be apparent to those skilled in the art that these specific embodiments illustrated and described can be replaced with various alternative and / or equivalent embodiments without departing from the scope of protection of the present invention. This application encompasses all adaptations or variations of the specific embodiments discussed herein. Therefore, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for controlling a robot that picks up objects in various postures, The steps include determining a plurality of reference points on the object, thereby determining the pickup pose of the object based on the state of the reference points in three-dimensional space, The steps include receiving a first camera image of the object in a known orientation, thereby making the position of a reference point in the first camera image known, The steps include mapping the first camera image to a first descriptor image using a machine learning model trained to assign visible object points within the camera image to descriptors independently of their positions within the camera image, The steps include identifying the descriptor of the reference point from the first descriptor image by reading the first descriptor image at the known position of the reference point, The steps include receiving a second camera image of the object in an unknown orientation state in which the object is picked up, The steps include mapping the second camera image to a second descriptor image using the machine learning model, The steps include searching for the descriptor of the identified reference point within the second descriptor image, The steps include determining the position of the reference point in the three-dimensional space of the unknown posture state from the position of the descriptor of the reference point detected in the descriptor image, The steps include determining the pickup pose of the object for the aforementioned unknown posture situation from the position of the reference point determined in the three-dimensional space, The steps include controlling the robot that picks up the object according to the determined pickup pose, A method that includes this.

2. The method according to claim 1, wherein the position of the reference point in three-dimensional space in the unknown posture state is determined from the detected position of the descriptor of the reference point in the second descriptor image by projecting depth information for the position of the reference point in the second camera image, which corresponds to the position of the reference point in the second descriptor image, into three-dimensional space.

3. The method according to claim 1 or 2, wherein the reference points have at least two reference points determined to be positioned along an elongated portion of the object, and the pickup pose of the object for the unknown posture situation is determined by determining the axis extending through the reference points.

4. The method according to any one of claims 1 to 3, wherein the reference points have at least three reference points determined to be positioned on the plane of the object, and the pickup pose of the object for the unknown posture situation is determined by determining a plane extending through the reference points.

5. The method further comprises the step of training the machine learning model, The step of training the aforementioned machine learning model is: A step of obtaining a 3D model of the object, including a grid of vertices, The steps include determining a descriptor for each vertex of the grid, The step of generating training data image pairs, wherein each training data image pair includes a training input image representing an object and a target image, wherein generating the target image is: A substep of determining the vertex positions of the vertices of the object model of the object having vertices in the training input image, A step including: for each specific vertex position in the training input image, assigning a descriptor determined for the vertex at that vertex position to a position in the target image; The machine learning model is trained by supervised learning using the training data image pairs as training data, The method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

6. The method according to claim 5, wherein the step of generating the training data image pairs includes the steps of acquiring a plurality of images of an object in different poses, and generating a training data image pair from each acquired image by generating a target image for each acquired image.

7. The method according to claim 5 or 6, further comprising the step of determining the vertex positions of the vertices of the object model of the object having vertices in the training input image from the respective poses of the object in the training input image.

8. The method according to any one of claims 5 to 7, wherein the vertices of a 3D model are joined by edges, each edge having a weight that identifies the proximity of two vertices joined by edges in the object, and the step of determining a descriptor for each vertex of the grid is performed by searching for a descriptor for a vertex, which minimizes the sum of the distances between descriptors of a vertex pair across a joined vertex pair, weighted by the edge weights between the vertex pairs.

9. A robot control device configured to carry out the method described in any one of claims 1 to 8.

10. A computer program that includes instructions causing a processor to perform the method according to any one of claims 1 to 8 when the instructions are executed by the processor.

11. A computer-readable storage medium storing instructions for causing a processor to perform the method according to any one of claims 1 to 8 when the instructions are executed by the processor.

Images