Devices and methods for training machine learning models to identify object topology from images of objects.

Abstract

According to various embodiments, a method for training a machine learning model to identify object topology from an image of an object is described, comprising obtaining a 3D model of the object, wherein the 3D model comprises a mesh of vertices connected by edges, wherein each edge has a weight specifying the proximity of two vertices connected by the edge in the object; determining a descriptor for each vertex of the mesh by searching for descriptors of the vertices, wherein the descriptors of the vertices minimize the sum of the distances between the descriptors of the vertex pairs weighted by the weights of the edges between the vertex pairs over the connected vertex pairs; generating training data image pairs, wherein each training data image pair comprises a training input image showing the object and a target image; and wherein generating the target image comprises determining the vertex positions of the vertices of the object model of the object in the training input image, and for each determined vertex position in the training input image, assigning a descriptor determined for the vertex at the vertex position to the position in the target image; and training the machine learning model by supervised learning using the training data image pairs as training data.

Description

[0001] This disclosure relates to apparatus and methods for training machine learning models to identify object topology from images of objects.

[0002] To allow robots to flexibly produce or manipulate objects, it is desirable for the robot to be able to handle objects regardless of their orientation within the robot's workspace. Therefore, the robot should be able to identify which parts of an object are located in which positions, enabling it to grasp the object in the correct position, for example, to mount it onto another object, or to weld the object in the current location. This means the robot should be able to identify the (surface) topology of the object, for example, from one or more images acquired from a camera mounted on the robot. This is achieved by determining descriptors—points (vectors) in a predefined descriptor space to portions of the object (i.e., pixels representing the object in the image plane)—where the robot is trained to assign the same descriptors to the same portions of the object independently of its current pose.

[0003] In the publication “Dense Object Nets: Learning Dense VisualObject Descriptors By and For Robotic Manipulation” by Peter Florence et al. (hereinafter referred to as Reference 1), dense object nets are described as a model for self-supervised dense descriptor learning.

[0004] However, the effectiveness of the method in Reference 1 is heavily dependent on the collected training data and the sensors involved, and the descriptors determined from self-supervised learning often lack interpretability. Therefore, in this respect, methods to enhance the training for object topology recognition are desirable.

[0005] According to various embodiments, a method is provided for training a machine learning model to identify the object topology from an image of an object, comprising obtaining a 3D model of the object, wherein the 3D model comprises a mesh of vertices connected by edges, wherein each edge has a weight specifying the proximity of two vertices connected by the edge in the object, determining a descriptor for each vertex of the mesh by searching for descriptors of the vertices, the descriptors of the vertices being minimized by the weights of the edges between the vertex pairs, the sum of the distances between the descriptors of the vertex pairs over the connected vertex pairs, generating training data image pairs, wherein each training data image pair comprises a training input image showing the object and a target image, and wherein generating the target image comprises determining the vertex positions of the vertices of the object model of the object in the training input image, and for each determined vertex position in the training input image, assigning a descriptor determined for the vertex at the vertex position to the position in the target image, and training the machine learning model by supervised learning using the training data image pairs as training data.

[0006] Compared to self-supervised training, the training methods described above allow machine learning models (such as neural networks) to perform more accurate predictions (i.e., descriptor determination) (i.e., allowing for a wider variety of applications for the network). Furthermore, it provides greater flexibility in adapting machine learning models to different problems, reduces training data requirements (e.g., the amount of training data needed), and produces interpretable machine learning tools.

[0007] Various examples are given below.

[0008] Example 1 is a method described above for training a machine learning model to identify the object topology from an image of an object.

[0009] Example 2 is the method of Example 1, wherein generating training data image pairs includes obtaining multiple images of objects in different poses, and generating training data image pairs from each obtained image by generating target images for the obtained images.

[0010] This allows machine learning models (e.g., robots with robot controllers that implement machine learning models) to, for example, identify the topology of objects in the robot's workspace, regardless of the objects' pose.

[0011] Example 3 is a method of Example 1 or 2, which involves determining the vertex positions of the vertices of the object model in the training input image from the corresponding pose (e.g., in camera coordinates) that the object has in the training input image.

[0012] This allows for precise determination of vertex positions, which in turn allows for the use of accurate target images for supervised training.

[0013] Example 4 is a method of any of Examples 1 to 3, wherein searching for a descriptor involves determining the eigenvectors of the Laplacian operator matrix of the graph formed by the vertices and edges of the 3D model, and taking the components of the eigenvectors as components of the descriptor.

[0014] This allows for the efficient determination of (near) optimal descriptors. For example, each descriptor is a vector of dimension d and therefore has d components (e.g., (3, 4) is a vector of dimension 2 and has components 3 and 4).

[0015] Example 5 is the method of Example 4, which includes associating each vertex with the component position in the feature vector, and for each vertex, taking the component at the component position in the feature vector associated with that vertex as the component of the vertex's descriptor.

[0016] In particular, the descriptor space dimension can be flexibly selected by choosing multiple feature vector components of the descriptor that correspond to the desired descriptor space dimension.

[0017] Example 6 is the method of Example 5, wherein determining the descriptor further includes compressing the descriptor by combining components of feature vectors whose feature values ​​differ by less than a predetermined threshold into a single component.

[0018] When the object is symmetrical, this allows for a reduction in the dimensions of the descriptor space. In other words, unnecessary distinctions between the symmetrical parts of an object can be avoided.

[0019] Example 7 is a method of any of Examples 1 to 6, wherein obtaining a 3D model of an object includes obtaining a 3D mesh of the vertices and edges of the object being modeled, and assigning edge weights as edge weights by applying the Laplace-Beltrami operator to the mesh.

[0020] Using this method allows the model's geometry to be taken into account when determining the proximity of two vertices by using geodesic distances on the object rather than Euclidean measurements in the surrounding space (which can be inaccurate when the object is curved).

[0021] Example 8 is a method for controlling a robot, including training a machine learning model according to any one of Examples 1 to 7, obtaining an image showing an object, feeding the image to the machine learning model, determining the pose of the object from the output of the machine learning model, and controlling the robot based on the determined object pose.

[0022] Example 9 is a method of Example 8, wherein determining the object pose includes determining the position of a portion of the object, and wherein controlling the robot, depending on the determined object pose, includes controlling the robot's end effector to move to the position of that portion of the object and interact with that portion of the object.

[0023] Example 10 is a software or hardware agent, particularly a robot, including a camera configured to provide image data of an object, a controller configured to implement a machine learning model, and a training device configured to train the machine learning model using any of the methods in Examples 1 through 7.

[0024] Example 11 is a software or hardware agent according to Example 10, including at least one actuator, wherein the controller is configured to control at least one actuator using the output from a machine learning model.

[0025] Example 12 is a computer program that includes instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 1 through 9.

[0026] Example 13 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 1 through 9.

[0027] In the accompanying drawings, similar reference numerals generally refer to the same parts throughout the different views. The drawings are not necessarily to scale; instead, the focus is generally on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, wherein:

[0028] Figure 1 The robot is shown.

[0029] Figure 2 The illustration shows the training of a neural network according to an embodiment.

[0030] Figure 3 An example embedding of a 4-node graph used for descriptor determination is shown.

[0031] Figure 4 The illustration shows the definition of the angle at the vertex of a 3D model based on the Laplace-Beltrami operator for determining edge weights.

[0032] Figure 5 A method for training a machine learning model to identify the object topology from an image of an object, according to an embodiment, is shown.

[0033] The following detailed description is taken with reference to the accompanying drawings, which illustrate by way of illustration specific details and aspects of the present disclosure in which the invention can be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the invention. The various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0034] The various examples will be described in more detail below.

[0035] Figure 1 Robot 100 is shown.

[0036] Robot 100 includes a robotic arm 101, such as an industrial robotic arm for handling or assembling a workpiece (or one or more other objects). Robotic arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 supporting the manipulators 102, 103, 104. The term "manipulator" refers to a movable component of robotic arm 101 whose actuation enables physical interaction with the environment, such as performing a task. For control, robot 100 includes a (robot) controller 106 configured to interact with the environment according to a control program. The last component 104 of the manipulators 102, 103, 104 (farthest from the support 105) is also referred to as an end effector 104 and may include one or more tools, such as a welding torch, a gripping instrument, a painting apparatus, etc.

[0037] Other manipulators 102, 103 (closer to support 105) can form a positioning device, providing a robotic arm 101 with an end effector 104 at its end, together with the end effector 104. The robotic arm 101 is a robotic arm capable of providing functions similar to a human arm (possibly with a tool at its end).

[0038] The robotic arm 101 may include joint elements 107, 108, and 109 that interconnect manipulators 102, 103, and 104 with each other and with support 105. Joint elements 107, 108, and 109 may have one or more joints, each capable of providing rotational movement (i.e., rotational motion) and / or translational movement (i.e., displacement) relative to the associated manipulator. Movement of manipulators 102, 103, and 104 may be initiated by means of actuators controlled by controller 106.

[0039] The term "actuator" can be understood as a component adapted to influence a mechanism or process in response to a drive. An actuator can realize mechanical movement from a command (so-called activation) issued by the controller 106. An actuator (e.g., an electromechanical converter) can be configured to convert electrical energy into mechanical energy in response to a drive.

[0040] The term "controller" can be understood as any type of logical implementation entity, which may include, for example, circuitry and / or a processor, firmware, or a combination thereof, capable of executing software stored in a storage medium, and capable of issuing instructions, such as instructions to an actuator in this example. A controller can be configured, for example, via program code (e.g., software) to control the operation of a system (in this example, a robot).

[0041] In this example, controller 106 includes one or more processors 110 and a memory 111 for storing code and data, which the processors 110 use to control the robotic arm 101. According to various embodiments, controller 106 controls the robotic arm 101 based on a machine learning model 112 stored in memory 111.

[0042] According to various embodiments, the machine learning model 112 is configured and trained to allow the robot 100 to recognize, for example, an object 113 placed in the workspace of the robotic arm 101. For example, the robot may determine how to handle the object 113 based on what the object 113 is (e.g., object type), or it may also identify and determine which part of the object 113 the end effector 109 should grasp. The robot 100 may, for example, be provided with one or more cameras 114, thereby allowing it to acquire images of its workspace. The cameras 114 are mounted, for example, on the robotic arm 101, so that the robot can acquire photographs of the object 113 from various perspectives by moving around the robotic arm 101.

[0043] An example of a machine learning model 112 for object recognition is a dense object network. As described in Reference 1, a dense object network maps an image (e.g., an RGB image provided by camera 114) to a descriptor space image of arbitrary dimension (dimension D).

[0044] Dense Object Networks (DONNs) are neural networks trained to use self-supervised learning to provide a descriptor space image of an input image. However, the effectiveness of this method is heavily dependent on the collected training data and the sensor involved (e.g., camera 114). Furthermore, interpreting the network's predictions can be difficult.

[0045] According to various embodiments, a method for identifying objects and their poses is provided, assuming that a 3D model of the object (e.g., a CAD (Computer-Aided Design) model) is known, typically for industrial assembly or processing tasks. According to various embodiments, nonlinear dimensionality reduction techniques are used to compute an optimal target image for use as input to a neural network. Therefore, supervised training of the neural network is used instead of self-supervised training.

[0046] According to one embodiment, in order to generate training data for training machine learning model 112, data collection is first performed. Specifically, for example, registered RGB (red, green, blue) images are collected. Here, a registered image refers to an RGB image with known internal and external features of the camera. In a real-world scenario, a camera 114 mounted on a robot (e.g., a camera mounted on the robot's wrist) is used, for example, to scan objects as the robot (e.g., robot arm 101) moves around. Other external estimation techniques, such as ChArUco marking, can be used, which involves placing the object in different positions and poses relative to a ChArUco board to obtain an arrangement image (of the ChArUco board and the object). In a simulated scenario, a realistically generated RGB image using known object poses is used.

[0047] After collecting RGB images, the target image of the RGB images is rendered for supervised training of the neural network.

[0048] Assume that the pose of each object in world coordinates is known in each collected RGB image. This is straightforward for simulated scenes, but requires manual tuning for real-world scenes, such as placing objects in predefined locations.

[0049] Using this information and vertex descriptor computation techniques, for example as described below, for each RGB image (i.e., the training input image), a descriptor image (i.e., the training output image, also known as the target image or the baseline ground truth image) is rendered.

[0050] Once a target image has been generated for each RGB image, i.e., an RGB image and target image pair has been formed, these training input images and associated target image pairs can be used as training data for purposes such as... Figure 2 The neural network is trained as shown in the diagram.

[0051] Figure 2 The illustration shows the training of a neural network 200 according to an embodiment.

[0052] Neural Network 200 is Tensor (input image) mapped to Fully convolutional networks for tensors (output images).

[0053] It includes multiple 204 convolutional layers followed by pooling layers, upsampling layers 205, and skip connections 206 to combine the outputs of different layers.

[0054] For training, neural network 200 receives training input image 201 and outputs output image 202 with pixel values ​​in descriptor space (e.g., color components according to descriptor vector components). A training loss is calculated between output image 202 and target image 203 associated with the training input image. This may occur over a batch of training input images, and the training loss can be averaged over the training input images, and the weights of neural network 200 are trained using stochastic gradient descent with the training loss. The training loss calculated between output image 202 and target image 203 is, for example, an L2 loss function (to minimize the pixel-wise least squared error between target image 203 and output image 202).

[0055] The training input image 201 illustrates the object, and the target image and output image comprise vectors in the descriptor space. The vectors in the descriptor space can be mapped to colors, making the output image 202 (and the target image 203) resemble heatmaps of the object.

[0056] The vectors in the descriptor space (also called (dense) descriptors) are assigned to each pixel in the corresponding image (e.g., assigned to each pixel in input image 201, assuming input image 201 and output image 202 have the same dimensions). d dimensional vectors (e.g., d (It is 1, 2, or 3). Dense descriptors implicitly encode the surface topology of the objects shown in the input image 201, invariant to their pose or camera position.

[0057] Given a 3D model of an object, it is possible to analytically determine the optimal (in the Riemannian sense) and unique descriptor vector for each vertex of the object's 3D model. According to various embodiments, using these optimal descriptors (or by optimizing estimates of these determined descriptors), a target image is generated for the registered RGB image, resulting in fully supervised training of a neural network 200. Additionally, the descriptor space becomes interpretable and optimal, regardless of the chosen descriptor dimension. d how.

[0058] In the following text, 3D models are considered to be embedded in... Riemannian manifolds This necessitates calculating geodesics (the shortest paths between vertices) while considering the geometry. This is achieved by embedding the 3D model into... d In a Euclidean descriptor space, the optimal surface topology can be explicitly encoded such that the geodesic distances between adjacent vertices are preserved as well as possible. This Euclidean space is considered the descriptor space, and optimal mappings are performed. The search is performed by computing the Laplacian operator of the mesh and its eigenvalue decomposition to determine (or at least estimate) the optimal embedding of vertices in the descriptor space. Thus, instead of separating geodesic computation and mapping optimization, descriptors are extracted in a single frame by computing the Laplacian operator of the 3D model.

[0059] Using the method described below, the embedding of a 3D object model in Euclidean space into the descriptor space is determined to preserve the distance between vertices (e.g., geodesic distance).

[0060] For dimension reduction via the Laplace operator, let a set of points This corresponds to a node in an undirected graph. Let... Mark two nodes and The strength or tightness of the connection between them, for example .

[0061] The goal is to find d Dimensional Embedding (generally d < D ), so that if and If they are close, then their embeddings should also be close:

[0062] (1)

[0063] in .

[0064] Optimization problem (1) is equivalent to

[0065] (2)

[0066] in It is a positive semidefinite Laplace operator matrix. It is an adjacency matrix, which has elements and It should be noted that the optimal solution It can have arbitrary scales and biases. To remove this randomness, the weighted second moment can be used... Regularization, which enforces unit variance across different dimensions. The resulting optimization problem then becomes

[0067] (3)

[0068] Use with Lagrange parameters Constraint optimization

[0069] (4)

[0070] This is a generalized eigenvalue problem, which can be solved using standard linear algebra libraries. Because and It is a (semi)positive definite matrix, therefore its eigenvalues ​​can be written as .

[0071] Furthermore, the first eigenvector ( The first column of the eigenvectors is equal to 1 (a vector of only one), which is a trivial solution that maps each vertex to a point. Additionally, any two eigenvectors are mutually orthogonal. The solution to the eigenvalue problem is given by... N eigenvalues ​​and having dimension N The corresponding eigenvectors. However, in practice, only the first few eigenvalues ​​corresponding to the lowest eigenvalues ​​are used. d There are eigenvectors (excluding trivial solutions).

[0072] therefore, The i Columns are nodes i In R d The embeddings in the data are represented by a single line, where each line represents the embedding of each point in different orthogonal dimensions.

[0073] Figure 3 An example embedding of a 4-node graph is shown.

[0074] Eigenvalues ​​are crucial for the optimality of embeddings. In the optimal embedding... Under the condition that the constraints are satisfied And therefore

[0075] (5)

[0076] That is, the eigenvalues ​​correspond to the embedding errors in different dimensions. For simplicity, let... d = 1, in this case, each All mapped to points y = 1. In this case, (5) simplifies to

[0077] (6).

[0078] because This means that if all vertices of an object are mapped to a single point, the embedding error is 0 because the distance between all points is zero. y The value is 0. This is useless for practical applications, so the first eigenvalue and eigenvector are omitted. Use d = 2 corresponds to each point Mapped to a line, and This corresponds to the embedding error, and so on. Since the feature vectors are orthogonal to each other, this increases... dThis will add a new dimension to the embedding, the purpose of which is to minimize the error in the new orthogonal dimension. The same effect can be seen in (3): due to The original objective can be transformed to minimize the embedding error in each dimension. Therefore, regardless of the chosen... d In any case, the resulting descriptor vectors are optimal.

[0079] In some cases, the subsequent eigenvalues ​​are equivalent, i.e. (See Figure 3 For example, where the eigenvalues ​​of d=2 and d=3 are the same. This carries some information about symmetry, where there are multiple orthogonal dimensions with equivalent embedding errors. In fact, in Figure 3 In the example of a 4-node graph, if the graph is fully connected, the embeddings are symmetric in each dimension, and all eigenvalues ​​are identical except for trivial solutions.

[0080] The graph embedding methods described above can be directly applied to meshes, point clouds, etc. For example, the K-Nearest Neighbor (KNN) algorithm can be used to form local connections between vertices and construct an adjacency matrix. This method is sufficient to construct the Laplacian operator of the graph and compute the embedding for each vertex. However, this method is essentially built on Euclidean distance metrics and heuristics, and it does not necessarily adhere to the underlying Riemannian geometry of the 3D object model. For example, some edges may traverse objects or connect non-adjacent vertices of the mesh. Even a few incorrect entries in the adjacency matrix can lead to poor embedding performance. Therefore, according to one embodiment, when working with the model, it is ensured that the geodesic distance between any two vertices is correct or has minimal approximation error.

[0081] Generally speaking, object models such as meshes or point clouds can be represented as embeddings. Riemannian manifolds in [the region / area]. A metric with smooth changes. g A Riemannian manifold M can be considered a “local Euclidean” that captures the local smoothness properties of real-world objects. The generalization of the Laplace operator to Riemannian manifolds is the Laplace-Beltramm (LB) operator. Similar to the Laplace operator in Euclidean space, the LB operator applied to a function is the divergence of the function's gradient. While the Laplace operator in graphs and Euclidean space is easy to compute (based on adjacency information or finite differences), the LB operator in differential geometry is built upon external calculus and is generally not easily applicable to manifolds.

[0082] However, for known discrete manifolds, such as meshes, the LB operator can be approximated. This provides a highly efficient and simple computational framework when working with meshes, point clouds, etc. Since the Riemannian equivalence of the Laplace operator is Laplace-Beltrami, the above embedding method can be directly applied. . eigenvectors The optimal representation of grid vertices d Vioclimatic embedding.

[0083] The following are efficient calculations for grid computing. Assuming to utilize N vertices V ,noodle F and edge E Provide a grid. In this case... The size is . The i The line describes the first i The adjacency information of each vertex and its connected vertices. Let... It is any function on the grid. Then, the discrete LB operator is applied to this function to map to... The first of the functions i Each element can be described by the following formula:

[0084] (7).

[0085] Figure 4 The diagram illustrates the angle. and Definition.

[0086] The sum of cotan terms Act as connection weight wij According to one embodiment, these weights appearing in (7) – i.e., the weights of the LB operator when applied to the grid – are used to determine equation (2). and The weight.

[0087] It should be noted that, due to Therefore, negative connection weights may occur. wij This is especially true if one angle is significantly larger than others (not a good face). To overcome this problem, edge flipping can be used to approximate the connection weights.

[0088] The descriptor generation method described above treats each vertex uniquely. That is, each vertex is assigned a unique descriptor. However, objects may be symmetric, and therefore, assigning unique descriptors to seemingly identical vertices will result in asymmetric embeddings.

[0089] To address this issue, according to various embodiments, the intrinsic symmetry of a shape is detected and symmetry embeddings are compressed such that symmetric vertices map to the same descriptor. It can be shown that a shape possesses intrinsic symmetry if the eigenfunctions of the Laplace-Beltramian operator exhibit symmetry in Euclidean space. In other words, symmetry features of meshes, point clouds, etc., are detected when their geodesics, preserving the Euclidean embeddings (descriptor space), exhibit Euclidean symmetry. This is further demonstrated if a homeomorphism exists that preserves the geodesic distance between each vertex of the manifold. T Then the compact manifold has an inherent symmetry.

[0090] To compress symmetric descriptors, the so-called Global Intrinsic Symmetric Invariant Function (GISIF) can be used. Given a globally intrinsic symmetric homeomorphism... in g Represents geodesic distance and manifold f The function on the manifold, then, if for every point on the manifold

[0091] (8)

[0092] but f It is GISIF. For example, on a torus, let the homeomorphism be an arbitrary rotation about the z-axis. This means that if f If it is GISIF, then it must be invariant to that rotation.

[0093] Furthermore, it can be shown that, in the case of equivalent eigenvalues In this case, such a GISIF is the sum of squares of the eigenvector components of that point, i.e.

[0094] .

[0095] This resonates with the above analysis of equivalent eigenvalues, which is a necessary condition for symmetric embedding. Since equivalent eigenvalues ​​are practically rare due to numerical constraints, a heuristic can be used where eigenvalues ​​lie in the same... Sphere (with small) Within a predetermined threshold, when the difference is less than 0.1% or 0.01%, the feature values ​​are considered equivalent. Since only one symmetry dimension needs to be found for a given object, this can be done manually.

[0096] For example, let the first 7 eigenvalues ​​of the eigenvalue decomposition of the torus be...

[0097] .

[0098] Ignoring trivial solutions, embedding The GISIF in the formula is then given by the following formula.

[0099] .

[0100] In the case of multiple objects, these can be represented as multiple disconnected connected graphs. In this case, the adjacency matrix is ​​block diagonal. The symmetric positive definite Laplacian operator will again have orthogonal eigenvectors. Compared to the single-graph embedding case, the eigenvalue decomposition results in two differences: First, the non-decreasing eigenvalues ​​will be the unordered embedding errors of all objects. Second, since the corresponding eigenvectors will remain orthogonal, the eigenvectors will have zero entries. This means that each dimension of the descriptor space will correspond to only one object embedding. Furthermore, the dimensions are ordered relative to the embedding errors of the corresponding objects. Therefore, in the case where a 3D embedding of two objects should be generated, d=8 is used because there exists a corresponding eigenvalue. Two trivial solutions.

[0101] This direct approach processes multiple objects independently, and while suboptimal methods may exist, it still offers a lower probability of success by leveraging the correlations between objects. d It is embedded quite well.

[0102] Given the pose of an object, a target image can be generated by projecting descriptors onto the image plane. As a non-object (background) descriptor space image, random noise or a single descriptor mapped to the farthest point in the descriptor space can be used.

[0103] To improve the robustness of the trained network 200, image enhancement methods such as Gaussian blur, cropping, or discarding domain randomization or perturbation can be applied.

[0104] In summary, the method illustrated in Figure 5 is provided according to various embodiments.

[0105] Figure 5 A method for training a machine learning model to identify the object topology from an image of an object, according to an embodiment, is shown.

[0106] In 501, a 3D model of the object is obtained, wherein the 3D model comprises a mesh of vertices connected by edges, wherein each edge has a weight specifying the proximity of two vertices in the object connected by the edge. The weights can be obtained (i.e. determined) from, for example, a CAD model (e.g., provided by the user).

[0107] In 502, a descriptor for each vertex of the mesh is determined by searching for the descriptors of the vertices, which minimizes the sum of the distances between the descriptors of the vertex pairs, weighted by the weights of the edges between the vertex pairs, over the connected vertex pairs.

[0108] In 503, training data image pairs are generated, wherein each training data image pair includes a training input image showing an object and a target image, and wherein generating the target image includes determining the vertex positions of the vertices of the object model of the object in the training input image, and for each determined vertex position in the training input image, assigning a descriptor determined for the vertex at the vertex position to the position in the target image.

[0109] In 504, training data image pairs are used as training data to train a machine learning model through supervised learning.

[0110] According to various embodiments, in other words, objects are embedded into a descriptor space by locally preserving geometric properties. As described in equation (1), this is accomplished by searching for embeddings that minimize the sum of the distances between the embeddings of vertex pairs, weighted by the edges between them, over the connected vertex pairs. Although in the above example the machine learning network is described as a neural network, other kinds of regressors that map 3D tensors to another 3D tensor can be used.

[0111] According to various embodiments, the machine learning model assigns descriptors to the pixels of an object (in the image plane). This can be viewed as indirectly encoding the surface topology of the object. This connection between the descriptors and the surface topology can be made explicit through rendering to map the descriptors to the image plane. It should be noted that the descriptor values ​​at faces (i.e., points that are not vertices) of the object model can be determined by interpolation. For example, when a face is given by the three vertices of the object model with their descriptor values ​​y1, y2, and y3 respectively, then at any point on the face, the descriptor value y can be calculated as a weighted sum of these values. In other words, the descriptor values ​​at the vertices are interpolated.

[0112] To generate image pairs for training data, the images of objects (e.g., RGB images) consist of objects (or multiple objects) with known 3D (e.g., CAD) models and poses (in a global (i.e., world) coordinate system) to a (dense) descriptor image. This (dense) descriptor image is optimal in the sense that it is generated by searching descriptors to minimize the deviation of geometric properties (specifically, the proximity of points of the object) between the object model and its representation (embedding) in the descriptor space. In practice, because the search is restricted to a certain search space, a theoretically optimal solution for minimization will generally not be found. However, the estimate of the minimum is determined within the constraints of the practical application (available computational accuracy, maximum number of iterations, etc.).

[0113] Therefore, the descriptors of vertices are searched by performing a process of minimizing the sum of the distances between the descriptors of the vertex pairs, which are weighted by the edges between the vertex pairs, over the connected vertex pairs, where each descriptor is searched for the corresponding vertex of the vertex.

[0114] Each training data image pair includes a training input image and a target image of the object, wherein the target image is generated by projecting descriptors of visible vertices in the training input image onto the training input image plane according to the pose of the object in the training input image.

[0115] Images, along with their associated target images, are used to train machine learning models in a supervised manner.

[0116] The machine learning model is thus trained to identify the unique features of an object (or multiple objects). This information, through real-time evaluation of the machine learning model, can be used for various applications in robot control, such as predicting object grasping postures for assembly. It should be noted that supervised training methods allow for the explicit encoding of symmetric information.

[0117] Figure 5 The methods described herein can be performed by one or more computers comprising one or more data processing units. The term "data processing unit" can be understood as any type of entity that allows the processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. A data processing unit may include analog circuits, digital circuits, composite signal circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof. Any other means of implementing the corresponding functions (which will be described in more detail below) can also be understood as a data processing unit or logic circuit. It will be understood that one or more method steps described in detail herein can be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0118] The term "robot" can be understood to refer to any physical system (with its mechanical parts whose movement is controlled), such as computer-controlled machines, vehicles, household appliances, power tools, manufacturing machines, personal assistants, or access control systems.

[0119] Various embodiments can receive and utilize sensor signals from various sensors to, for example, obtain sensor data showing an object, such as (e.g., RGB) cameras, video sensors, radar, LiDAR, ultrasound, thermal imaging, etc. Embodiments can be used to generate training data and train machine learning systems, for example, for controlling robots, such as robot manipulators, to autonomously perform various manipulation tasks in different scenarios. In particular, embodiments are suitable for controlling and monitoring the execution of manipulation tasks, for example, in an assembly line.

[0120] Although specific embodiments have been illustrated and described herein, those skilled in the art will appreciate that various alternatives and / or equivalent implementations may be used instead of the illustrated and described embodiments without departing from the scope of the invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, the invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for training a machine learning model to identify the object topology from an image of an object, the method comprising: Obtain a 3D model of the object, wherein the 3D model comprises a mesh of vertices connected by edges, wherein each edge has a weight specifying the proximity of two vertices in the object connected by the edge; The descriptor of each vertex of the mesh is determined by searching for the descriptors of the vertices, wherein the descriptor of the vertex minimizes the sum of the distances between the descriptors of the vertex pairs, which are weighted by the weights of the edges between the vertex pairs, on the connected vertex pairs, wherein searching for the descriptors includes determining the eigenvectors of the Laplacian operator matrix of the graph formed by the vertices and edges of the 3D model, and taking the components of the eigenvectors as the components of the descriptors. Generate training data image pairs, wherein each training data image pair includes training input images showing an object and a target image, and wherein generating the target image includes Determine the vertex positions of the vertices of the object model in the training input image; and For each determined vertex position in the training input image, the descriptor determined for the vertex at that vertex position is assigned to the position in the target image; as well as Using training data image pairs as training data, a machine learning model is trained through supervised learning.

2. The method according to claim 1, wherein, Generating training data image pairs involves obtaining multiple images of an object in different poses and generating training data image pairs from each obtained image by generating target images for the obtained images.

3. The method according to claim 1 or 2, comprising determining the vertex positions of the vertices of the object model in the training input image from the corresponding pose of the object in the training input image.

4. The method of claim 1 or 2, further comprising associating each vertex with a component position in a feature vector, and for each vertex, taking the component at the component position in the feature vector associated with the vertex as a component of the vertex's descriptor.

5. The method according to claim 4, wherein, Determining a descriptor further includes compressing the descriptor by combining components of feature vectors whose feature values ​​differ by less than a predetermined threshold into a single component.

6. The method according to claim 1 or 2, wherein, Obtaining a 3D model of an object involves obtaining a 3D mesh of the vertices and edges of the object being modeled, and assigning the edge weights of the Laplace-Beltrami operator applied to the mesh as the edge weights.

7. A method for controlling a robot, comprising: Train a machine learning model according to any one of claims 1 to 6; Obtain an image showing the object; Feed the image into the machine learning model; Determine the pose of an object from the output of a machine learning model; and The robot is controlled based on the determined pose of the object.

8. The method according to claim 7, wherein, Determining an object pose includes determining the position of a portion of the object, and controlling the robot, depending on the determined object pose, includes controlling the robot's end effector to move to the position of said portion of the object and interact with said portion of the object.

9. A robot, comprising The camera is configured to provide image data of objects; The controller is configured to implement a machine learning model; and The training device is configured to train the machine learning model using the method described in any one of claims 1 to 6.

10. The robot of claim 9, comprising at least one actuator, wherein the controller is configured to control the at least one actuator using output from the machine learning model.

11. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

12. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images