Method for training machine learning model to generate descriptor image for image displaying one or more objects

Abstract

According to various embodiments, a method for training a machine learning model to generate a descriptor image for an image displaying one or more objects is described, with: recording a plurality of camera images, where each camera image displays one or more objects; forming training image pairs from a plurality of camera images, wherein each training image pair has a first training image and a second training image; determining a loss for the training image pair and the at least one keypoint from the distance between the position of the keypoint and the estimated position of the keypoint in the first training image or a transformed version of the first training image, the position is determined by transitioning from the first training image to the second training image and back to the first training image or a transformed version thereof; and adapting the machine learning model to reduce a total loss including the determined loss at least for a portion of the training image pair and the keypoint.

Description

Technical Field

[0001] The present disclosure relates to methods for training a machine learning model to generate descriptor images for images showing one or more objects. Background Art

[0002] In order to enable the flexible manufacture or processing of objects by robots, it is desirable that the robot can manipulate the object regardless of the posture of the object in the robot's workspace. Therefore, the robot should be able to recognize which parts of the object are located at which positions, so that the robot can, for example, grab the object at the correct position, so as to, for example, fasten or weld the object to another object. This means that the robot should be able to recognize the pose (position and orientation) of the object, or determine the position of the part for recording or processing, for example, from one or more images recorded by a camera fastened to the robot. A scheme for implementing this is to determine a descriptor, i.e., a point (vector) in a predefined descriptor space, for a part of the object (i.e., a pixel of the object represented in the image plane), wherein the robot is trained to assign the same descriptor to the same part of the object regardless of the current pose of the object, and thereby recognize the topology of the object in the image, so that, for example, it is known which corner of the object is located in the image. In the case of knowing the pose of the camera, the pose of the object can then be inferred. The recognition of the topology can be achieved using a machine learning model trained accordingly. Summary of the invention

[0003] Methods for efficiently training such machine learning models are desirable.

[0004] According to various embodiments, there is provided a method for training a machine learning model to generate a descriptor image for an image showing one or more objects, having:

[0005] recording of multiple camera images, where each camera image shows one or more objects,

[0006] A plurality of camera images are used to form training image pairs, wherein each training image pair comprises a first training image and a second training image.

[0007] For each training image pair and for each training image pair for each of the one or more key points (the one or more key points are at least contained in the corresponding first training image)

[0008] o generating a first descriptor image for the first training image by feeding the first training image to the machine learning model (wherein the first descriptor image comprises, for each point of the first training image, a corresponding descriptor assigned to the point by the machine learning model based on the first training image));

[0009] o generating a second descriptor image for the second training image by feeding the second training image to the machine learning model (wherein the second descriptor image contains, for each point of the second training image, a corresponding descriptor assigned to the point by the machine learning model based on the second training image));

[0010] o determining a descriptor, which the first descriptor image contains for a key point (e.g. an object feature point or a point in the background);

[0011] o for each (e.g., all) of the plurality of points of the second training image, estimating a probability that the point corresponds to a keypoint (i.e., a point showing the same object feature point or background), wherein the greater the estimated probability, the closer the descriptor included in the first descriptor image for the keypoint is to the descriptor included in the second descriptor image for the corresponding point of the second training image among the plurality of points of the second training image;

[0012] o estimating the descriptor that the second descriptor image will contain for the keypoint by a weighted sum of the descriptors that the second descriptor image contains for a plurality of points of the second training image (which need not be the case if the second descriptor image actually contains the estimated descriptor for one of the points), wherein each descriptor is weighted by a probability determined for the point that the descriptor is contained in the second descriptor image;

[0013] o estimating the positions of points in the first training image, or a transformed version of the first training image, to which the machine learning model would assign the estimated descriptor (which need not be the case if the machine learning model actually assigns the estimated descriptor to one of the points); and

[0014] o determining a loss for the training image pair and the keypoint from the distance between the position of the keypoint in the first training image or a transformed version of the first training image and the estimated position; and

[0015] Adapting the machine learning model to reduce an overall loss (e.g., batch loss) that includes the determined losses for at least a portion of the training image pairs and keypoints.

[0016] Compared to methods that require, for example, RGB-D image sequences, 3D reconstructions or other ground truth annotation schemes, the above method enables a simplified collection of training data and a correspondingly less complex training of a machine learning model to generate descriptor images. The use of losses for self-supervised training according to the above method enables training with unordered RGB images or pairs of different images (i.e. different recordings, and not just enhancements of the same image). In this case, the correspondence of points between the different images is automatically determined with the aid of a "loop" (transition from the first image to the second image and back again).

[0017] Various embodiments are described below.

[0018] Embodiment 1 is a method for training a machine learning model to generate a descriptor image for an image as described above.

[0019] Embodiment 2 is a method according to embodiment 1, wherein the position of a point in the first training image or a transformed version of the first training image is estimated by estimating, for each of a plurality of points in the first training image or a transformed version of the first training image (e.g., for all points), the probability that the point corresponds to a point to which the machine learning model will assign the estimated descriptor (which need not be the case if the machine learning model actually assigns the estimated descriptor to one of the points), wherein the greater the estimated probability, the closer the estimated descriptor is to the descriptor assigned by the machine learning model to the corresponding point in a plurality of points of the first training image or a transformed version of the first training image, and each coordinate of the position of the point is estimated as a weighted sum of the values ​​of the coordinates of a plurality of points of the first training image or a transformed version of the first training image, wherein each value of the coordinate is weighted by the probability determined for the point whose position the value describes (see equations (3) and (4) below).

[0020] In other words, the "backward" transition of the cycle (i.e., the transition from the second image to the first image) is also performed probabilistically similar to the "forward" transition of the cycle (i.e., the transition from the first image to the first image), so that no hard selection is performed when determining the correspondence, thereby ensuring differentiability, which enables the use of gradient descent methods for training.

[0021] Embodiment 3 is a method according to embodiment 1 or 2, wherein the total loss for each training image pair for which the loss is determined for each of a plurality of key points is included as a weighted sum of the losses determined for the key points, wherein the lower each determined loss is weighted, the greater the measure of the variance of the estimate of the position of a point in the first training image or a transformed version of the first training image to which the machine learning model will assign the estimated descriptor and / or the estimate of the position of a point in the second training image to which the machine learning model will assign the estimated descriptor when determining the loss for the corresponding key point.

[0022] This avoids keypoints that are not present (ie appear) at all in the second training image from causing losses which would degrade the training.

[0023] Embodiment 4 is a method according to embodiment 3, wherein if the measure of the variance is above a threshold, the determined loss in the total loss is weighted by zero.

[0024] The training can thereby be protected from particularly unreliable correspondence determinations. For example, the threshold is determined such that a determined loss which belongs to a certain quantile (eg the 15% quantile) due to its value of the measure for the variance is not taken into account in the overall loss.

[0025] Embodiment 5 is a method according to any one of Embodiments 1 to 4, wherein the transformed version of the first training image is generated by enhancing the first training image, or wherein at least some of the training image pairs constituting the training image pairs have: the first training image is generated by enhancing the corresponding recorded camera image, and wherein the transformed version of the first training image is generated by another enhancement of the corresponding recorded camera image.

[0026] Thereby (when using the transformed version for the "reverse") it is avoided that the machine learning model will learn "shortcuts" (that have no correspondence with the second training image).

[0027] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the machine learning model is a neural network.

[0028] According to various implementations, for example, dense object nets are trained. These can be used to obtain good results for generating descriptor images.

[0029] Embodiment 7 is a method for controlling a robot for picking up or handling an object, comprising:

[0030] Training a machine learning model according to any one of embodiments 1 to 6; recording a camera image, wherein the camera image displays an object in a current control scene; feeding the camera image to the machine learning model to generate a descriptor image; determining the position of a part used to pick up or handle an object in the current control scene from the descriptor image; and controlling the robot according to the determined position.

[0031] Embodiment 8 is a method according to embodiment 7, comprising: identifying a reference part in a reference image; determining a descriptor of the identified reference part by feeding the reference image to a machine learning model; determining the position of the reference part in the current control scene by searching for the determined descriptor in a descriptor image generated by a camera image; and determining the position of a part for picking up or processing an object in the current control scene from the determined position of the reference part.

[0032] Embodiment 9 is a control device configured to implement the method according to any one of embodiments 1 to 8.

[0033] Embodiment 10 is a computer program having instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 8.

[0034] Embodiment 11 is a computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 8. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In the accompanying drawings, like reference numerals generally refer to the same parts in completely different views. The accompanying drawings are not necessarily drawn to scale, with emphasis instead generally being placed on depicting the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0036] Figure 1 A robot is shown.

[0037] Figure 2 Elucidate the loop for correspondence prediction of keypoints.

[0038] Figure 3 A flow chart representing a method for training a machine learning model to generate a descriptor image for an image showing one or more objects is shown according to one embodiment. DETAILED DESCRIPTION

[0039] The following detailed description relates to the attached drawings, which illustrate the particular details and aspects of the present disclosure for the purpose of illustration, in which the present invention may be performed. Other aspects may be used, and structural, logical and electrical changes may be implemented without departing from the scope of protection of the present invention. Because some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects, the various aspects of the present disclosure are not necessarily mutually exclusive.

[0040] Various examples are described in more detail below.

[0041] Figure 1 A robot 100 is shown.

[0042] The robot 100 comprises a robot arm 101, such as an industrial robot arm, for manipulating or installing a work piece (or one or more other objects). The robot arm 101 comprises manipulators 102, 103, 104 and a base (or pillar) 105, by which the manipulators 102, 103, 104 are supported. The expression "manipulator" relates to a movable member of the robot arm 101, the operation of which enables physical interaction with the environment, for example, in order to perform a task. For control, the robot 100 comprises a (robot) control device 106, which is designed to implement interaction with the environment according to a control program. The last member 104 (furthest from the pillar 105) of the manipulators 102, 103, 104 is also called the end effector 104, and may include one or more tools, such as welding torches, gripping instruments, painting tools, etc.

[0043] The other manipulators 102, 103 (closer to the pillar 105) may constitute positioning means such that the robotic arm 101 is provided at its end with an end effector 104. The robotic arm 101 is a robotic arm that may provide similar functionality to a human arm (possibly with a tool at its end).

[0044] The robot arm 101 may include joint elements 107, 108, 109, which connect the manipulators 102, 103, 104 to each other and to the support 105. The joint elements 107, 108, 109 may have one or more joints, which may provide the associated manipulators with a rotatable movement (i.e., rotational movement) and / or a translational movement (i.e., displacement) relative to each other. The movement of the manipulators 102, 103, 104 may be initiated by means of actuators, which are controlled by the control device 106.

[0045] The expression "actuator" may be understood as a component that is designed to cause a mechanism or process in response to the actuation of the component. The actuator may implement a command (so-called activation) created by the control device 106 into a mechanical movement. An actuator, for example an electromechanical converter, may be designed to convert electrical energy into mechanical energy in response to its actuation.

[0046] The expression "control device" may be understood as any type of logic implementation entity, which may include, for example, a circuit and / or a processor capable of executing software, firmware or a combination thereof stored in a storage medium and dispatching instructions, for example to actuators in the present example. For example, the control device may be configured by program code (e.g., software) in order to control the operation of the system (in the present example, the robot).

[0047] In this example, the control device 106 includes one or more processors 110 and a memory 111 storing code and data based on which the processor 110 controls the robotic arm 101. According to various embodiments, the control device 106 controls the robotic arm 101 based on a machine learning model 112 stored in the memory 111.

[0048] The control device 106 uses the machine learning model 112 to determine, for example, the pose of an object 113 placed in the workspace of the robot arm 101. Based on the determined pose, the control device 106 can, for example, decide which part of the object 113 should be grasped (or otherwise processed) by the end effector 109.

[0049] The control device 106 uses one or more camera images of the object 113 to determine the pose using the machine learning model 112. For example, the robot 100 can be equipped with one or more cameras 114, which enable the robot to record images of its workspace. For example, the camera 114 is fastened to the robot arm 101, so that the robot can record images of the object 113 from different perspectives, in such a way that the robot moves the robot arm 101 around. However, one or more fixed cameras can also be provided.

[0050] According to various embodiments, the machine learning model 112 is a (deep) neural network which generates for the camera image a feature map, e.g. in the form of an image in a feature space, which feature map enables the assignment of points of a (3D) object to points in the (2D) camera image.

[0051] For example, the machine learning model 112 can be trained to assign specific (unambiguous) feature values ​​(also called descriptor values) in the feature space to specific corners of an object. If the camera image is then fed to the machine learning model 112 and the machine learning model 112 assigns the feature value to a point in the camera image, it can be concluded that the corner is located at that location (i.e., the location in space whose projection onto the camera plane corresponds to the point in the camera image). If the location of multiple points of the object in the camera image is thus known, the pose of the object in space can be determined.

[0052] The machine learning model 112 must be trained appropriately for the task.

[0053] An example of a machine learning model 112 for object recognition is a dense object net (DON). A dense object net transforms an image (e.g., an RGB image provided by a camera 114) into Mapping to an arbitrary dimensional (with user-defined dimension D, e.g. D=16) descriptor space image (also called descriptor image) That is, a descriptor value is assigned to each pixel.

[0054] Such dense visual descriptors provide flexible, simple to use and easy to learn object representations for robotic manipulation. They show potential for generalizing objects at the category level, can describe non-rigid objects, and can be seamlessly used for state representation for control.

[0055] Training of dense descriptor nets can be implemented with the help of self-supervision based on multiple views of the same object and dense pixel correspondences computed from the 3D geometry of the object. Alternatively, RGB image augmentation can be used to generate alternative views of the same image, where pixel correspondences are tracked. The difference between these two schemes is only how the pixel correspondences between multiple images are generated and which hypothesis the images have for the training dataset, otherwise the same loss function can be applied, such as contrastive loss.

[0056] Alternative views of the same object are encoded with the help of training on pixel correspondences computed through 3D geometry, which leads to view-invariant learned descriptors. However, this requires a registered RGB-D dataset or a trained NeRF (neural Radiance Field), which is usually very expensive due to camera calibration, hardware setup and data recording. In the case of RGB image enhancement, only an unordered collection of RGB images recorded by a smartphone is needed to depict one / multiple objects. However, the descriptors learned from them cannot handle excessive changes in camera viewpoint and are therefore not always view-invariant, which limits their practicality.

[0057] According to various embodiments, a method for training a machine learning model to map a camera image to a descriptor image (i.e., a descriptor image of, for example, DON) is provided, which enables a simple scheme for training data acquisition, i.e., a series of unordered RGB images showing one or more objects respectively are sufficient for training, but still enables the machine learning model to learn descriptors in a view-invariant manner, i.e., the descriptors determined by the machine learning model are more robust to changes in camera viewing angle than in the case of training methods based on RGB image enhancement.

[0058] According to various embodiments, a cycle-correspondence loss (CCL for English: Cycle-Correspondence Loss) is used for this purpose. This is a loss for self-supervised training of a machine learning model for mapping camera images to descriptor images, which enables training based on selected pairs of training images, in particular RGB images. The idea behind this loss can be that for an image pair (I A , I B ), when in the first (training) image I A When there is an explicit descriptor in the image I A Each correct position prediction of a key point (e.g. a specific object feature point, but also a specific point in the background) in the image I B can be used to predict image I A The (original) positions of the key points in , thereby creating a loop for correspondence prediction.

[0059] Figure 2 Explain the key point k A The corresponding relationship of the predicted loop.

[0060] Key point k A The key points are provided in the image I by their descriptors (which the machine learning model assigns to the key points according to its current training state) BThe estimation 201 of the positions of the key points in the image I, said estimation (more precisely the estimated descriptor values, see below) in turn providing A (or a transformed version thereof in this embodiment). That is, by A Transition to I B And switching back again (i.e. one loop), an estimate of the original position of the keypoint is obtained. From this, the loss for the training image pair can be determined, the better the estimate 202 corresponds to the original position, the smaller the loss. The machine learning model can then be adapted to reduce the (total) loss, which includes these losses for multiple training image pairs (i.e. the machine learning model is adapted in the direction of the descending gradient of the total loss, i.e. its parameters (typically weights) are changed so that re-determining the descriptor image for the training image of the training image pair will provide a smaller value for the total loss).

[0061] In this way, the machine model can learn independently to identify valid correspondences, without relying on correspondence annotations. It is assumed in this case that the training image set largely depicts the same content - but this can also include random object arrangements and different backgrounds.

[0062] For the following detailed description of the proposed method, it is therefore assumed that there are two (training) images I A and I B (In this example, I A and That is, for example, an RGB image). And assuming that image I A Some pixels in image I B This assumption can be satisfied, for example, by epipolar restriction.

[0063] With the help of machine learning model f θ (.) (with its current training state, i.e. the value of its parameter θ corresponding to its current training state), maps each of the two images to a corresponding descriptor image D with descriptor values ​​from the latent descriptor space A , Image I A The key points or their pixel positions in A =(x A ,y A ) indicates that, and as long as it exists, the image I B The corresponding pixels in (i.e., pixels corresponding to the same key point) are called k B =(x B ,y B ). Here, x refers to the row coordinate and y refers to the column coordinate. For each keypoint k A By DA Allocate the corresponding descriptor And also give I B The corresponding key point assignment descriptor in

[0064] CCL requires the ability to predict the location of a keypoint in one image given its descriptor in another image. When k B The simplest way is to In D B The closest descriptor in the latent space (i.e. the descriptor space) is determined from . This is typically sufficient at inference time (e.g. to determine the pose of an object in an image), but there is a problem at training time: here the (x, y) coordinates are obtained in a non-differentiable way (so that gradient descent methods cannot be used).

[0065] Therefore, according to various embodiments, in order to determine the corresponding pixel, first determine With D B The pairwise distance between each descriptor of B Distance heat map on Right now

[0066]

[0067] where Δ is a distance function, such as the I2 norm, or a similarity measure, such as cosine similarity. In the following, cosine similarity and normalized descriptors are assumed.

[0068] Now we can determine the probability distribution The approach is to apply a temperature-scaled softmax function to the distance heatmap, i.e.

[0069]

[0070] where t is the temperature. If the expected values ​​of the marginal distributions are interpreted as coordinates, then k B =(x*,y*) can be estimated by

[0071]

[0072] The corresponding variance Correspondingly, as about (x-μx) 2 or (y-μy) 2 If there is a ground truth annotation (i is the keypoint index here), for example when the pixel-by-pixel correspondence comes from 3D geometry, it is simple to directly optimize the prediction error via spatial expectation, for example using the loss function

[0073]

[0074] According to various embodiments, the above concepts are used for a fully self-supervised training method. This means that no ground truth is given And the error to be optimized cannot be defined directly as in equation (5).

[0075] For the following explanation, temporarily assume the following condition: If this is also unknown, then for each (e.g. sampled) in I B There is a corresponding pixel in Given the assumption that it is known that if the prediction For Transformation I A →I B is correct, then the descriptor By transforming I B →I A Providing forecasts Make The cycle of comparing the corresponding relationships will be completed.

[0076] because is known, so the prediction error can be measured, which allows the error term loss to be defined as follows for keypoint i:

[0077]

[0078] See also Figure 2 Used to clarify.

[0079] Although the loss according to (6) can be conceptually simple to state, some practical considerations must be taken into account. The following discusses various steps that are necessary or useful for training a model using the outlined scheme.

[0080] although has its own descriptors, but simply indexing the descriptor image according to the coordinates is not differentiable. In order to maintain differentiability, according to various embodiments, the descriptors are obtained by indexing the descriptors in a similar way to the spatial expectation values ​​according to (3) and (4). Calculate the expected value:

[0081]

[0082] The points according to equations (3) and (4) can be viewed as estimates of the points to which the machine learning model would assign descriptors according to (7).

[0083] If the descriptor is normalized, then additional Normalized, which is assumed here to be implicitly given. In this case the descriptor belongs to the hypersphere manifold Compared to directly determining the expectation on the manifold, the averaging and normalization described above leads to practically the same results for the trained network. Definition for Transformation I B →I A Distribution And using spatial expectation (according to equations (3) and (4))

[0084] According to various embodiments, various enhancements are applied to the training images. For example, affine transformations (rotation, scaling), perspective distortion, and color shift (the latter mainly for brightness and enhancement) can be used. In order to ensure that the machine learning model does not ignore the image I B And learn an identity assignment via a shortcut, producing (at least for a subset of the training image pairs) an input image I A A copy of or for both or both ( and I A ) is enhanced and used as the first image of the training image pair or as the reverse target image. If the enhancement is known, then Right now exist The position in is also known.

[0085] This situation is Figure 2 It is explained in: A and The (eg respectively random) enhancements are obtained from the same recorded camera images; they are thus obtained by transforming T(I A ) are related to each other. B is another recorded camera image (which may also be enhanced in order to increase the diversity of the training images), the content of which is at least partially similar to I A (and therefore also Overlap. For example, keypoint k A isI A The randomly sampled parts in , and determined according to the transformation The corresponding position in I B The corresponding point in can, but need not, exist, but the potential correspondence can be measured by the variance (see below in conjunction with equation (8)) are determined or evaluated (these variances can be overlaid with the training images, for example as a heat map, in order to make the correspondence visible). By using the probability distribution according to equation (3) and I B The descriptor image can be calculated according to (7) It is possible to reverse the process and estimate The machine learning model is adapted so as to reduce the error between the true position and an estimate (e.g. summing or averaging a batch of training image pairs).

[0086] When training with unordered RGB images, not only can the background be expected to change from image to image, but also which objects are present in the image and in what orientation. Therefore, according to various embodiments, the above assumption is relaxed, that is, each in I B There are analogs in this. Two difficulties arise from this: first, in the absence of analogs, Each l i (See Eq. (6)) violates the basic assumption of cycle consistency, and the calculated gradient may be completely invalid. Second, due to Basically the most similar The weighted sum of the descriptors, so the machine learning model is B In theory, it is still possible to find arrive This can lead to shortcut learning and is typically undesirable.

[0087] In order to detect and prevent such situations, the previously determined probability distribution (according to equation (2)) is utilized according to various embodiments. To this end, the variance of the prediction for the i-th keypoint is calculated. Intuitively, it is assumed that if there is a clear analogue and the model is reliable, the variance will be small. If there is no agreement or the model is unreliable, the variance should increase.

[0088] For example, variance can be used in two ways. First, all N keypoints are computed for image I B and The summed variance of The q quantiles are formed, for example q = 15%. This summed variance can be regarded (for each training image pair and each key point) as a measure of the variance of the two position estimates in the two training images (according to (3) and (4) respectively). The q% most reliable identified points are thus obtained, and only these points are considered for the loss calculation. Next, equation (6) is modified in such a way that for each single loss l i The contribution of is scaled, thereby obtaining the loss for the corresponding training image pair

[0089]

[0090] One is added to the denominator to prevent When ∝ is less than 1, this term becomes unacceptably large. It is important to free the gradient of the variance from the computation graph, because otherwise the model simply learns to make predictions with low confidence rather than solving the prediction task.

[0091] In summary, according to various embodiments, there is provided Figure 3 The method shown in .

[0092] Figure 3 A flowchart 300 is shown representing a method for training a machine learning model to generate a descriptor image for an image showing one or more objects, according to one embodiment.

[0093] In 301 , a plurality of camera images are recorded, wherein each camera image shows one or more objects.

[0094] At 302 , training image pairs are formed from a plurality of camera images, wherein each training image pair has a first training image and a second training image.

[0095] In 303, for each training image pair and for each training image pair, for each of the one or more key points (the one or more key points are at least included in the corresponding first training image):

[0096] At 304, a first descriptor image is generated for the first training image by feeding the first training image to a machine learning model (wherein the first descriptor image contains, for each point of the first training image, a corresponding descriptor assigned to the point by the machine learning model based on the first training image));

[0097] At 305, a second descriptor image is generated for the second training image by feeding the second training image to the machine learning model (wherein the second descriptor image contains a corresponding descriptor for each point of the second training image (the machine learning model assigns the descriptor to the point based on the second training image));

[0098] At 306 , a descriptor is determined, which the first descriptor image contains for a key point (eg, an object feature point or a point in the background);

[0099] In 307, for each (e.g., all) of the plurality of points of the second training image, a probability that the point corresponds to a key point (i.e., a point showing the same object feature point or background) is estimated, wherein the greater the estimated probability is, the closer the descriptor included in the first descriptor image for the key point is to the descriptor included in the second descriptor image for the corresponding point of the second training image among the plurality of points of the second training image;

[0100] at 308, estimating the descriptor that the second descriptor image would contain for the keypoint by a weighted sum of the descriptors that the second descriptor image contains for a plurality of points of the second training image (which need not be the case if the second descriptor image actually contains the estimated descriptor for one of the points), wherein each descriptor is weighted by a probability determined for the point for which the descriptor is contained in the second descriptor image;

[0101] estimating in 309 the positions of points in the first training image or a transformed version of the first training image to which the machine learning model would assign the estimated descriptor (this need not be the case if the machine learning model actually assigns the estimated descriptor to one of the points); and

[0102] • In 310, a loss is determined for the training image pair and the keypoint from the distance between the position of the keypoint in the first training image or a transformed version of the first training image and the estimated position.

[0103] At 311, a machine learning model is adapted to reduce a total loss (eg, batch loss) that includes the determined losses for at least a portion of the training image pairs and keypoints.

[0104] Figure 3 The method can be implemented by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions implemented by a data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, an integrated circuit of a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any combination thereof or constructed thereof. Any other means for implementing the corresponding functions described in more detail here may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail here can be performed (e.g., implemented) by a data processing unit through one or more specific functions, and the specific functions are implemented by the data processing unit.

[0105] According to various embodiments, the method is therefore in particular computer-implemented.

[0106] With the aid of the trained machine learning model, control signals can ultimately be generated for a robotic device (e.g., by using the trained machine learning model for determining the pose of an object or determining a part to be processed). For this purpose, the relevant parts of any type of object for which the machine learning model is trained can be determined. The term "robotic device" can be understood as any physical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. Control criteria for the physical system are learned, and the physical system is then controlled accordingly.

[0107] For example, images are recorded by means of an RGB-D (color image plus depth) camera, processed by a trained machine learning model (e.g. a neural network), and relevant locations in the working area of ​​the robotic device are determined, wherein the robotic device is controlled as a function of the determined locations. For example, objects (i.e. their position and / or pose) can be tracked in the input sensor data. The descriptors can also be further processed in order to detect objects (and, for example, perform semantic segmentation), such as objects to be manipulated or traffic signs, road surface, pedestrians and vehicles.

[0108] The camera image is for example an RGB image or an RGB-D (color image plus depth) image, but may also be other types of camera images, such as (only) depth images or thermal images, videos, radar images, lidar images, ultrasound images or motion images. A depth image is not mandatory. The output of the trained machine learning model can be used to determine the object pose, for example for controlling a robot, for example for assembling larger objects from sub-objects, moving objects, etc. Figure 3 The method can be used for any pose determination method. Based on the relevant parts determined from the descriptor images, it is possible to determine, for example, the positions of parts of the object where the robot can grasp (or suck) the object and the positions of such parts where the object should not be grasped. Therefore, they provide valuable information for manipulation tasks, for example.

Claims

1. A method for training a machine learning model (112) to generate a descriptor image for an image showing one or more objects (113), comprising: recording a plurality of camera images, wherein each camera image shows one or more objects (113), A plurality of camera images are used to form training image pairs, wherein each training image pair comprises a first training image and a second training image, For each training image pair and for each training image pair for each of the one or more key points, generating a first descriptor image for a first training image by feeding the first training image to a machine learning model (112); generating a second descriptor image for the second training image by feeding the second training image to a machine learning model (112); determining a descriptor, the first descriptor image comprising the descriptor for the key point; for each of the plurality of points of the second training image, estimating a probability that the point corresponds to a key point, wherein the greater the estimated probability is, the closer the descriptor included in the first descriptor image for the key point is to the descriptor included in the second descriptor image for the corresponding point of the second training image in the plurality of points of the second training image; estimating a descriptor that the second descriptor image will contain for the keypoint by taking a weighted sum of descriptors contained in the second descriptor image for a plurality of points of the second training image, wherein each descriptor is weighted by a probability determined for the point for which the descriptor is contained in the second descriptor image; estimating locations (202) of points in the first training image or a transformed version of the first training image to which the machine learning model (112) will assign estimated descriptors; and determining a loss for the training image pair and the keypoint from a distance between the position of the keypoint in the first training image or a transformed version of the first training image and the estimated position (202); as well as Adapting a machine learning model (112) to reduce a total loss that includes the determined losses for at least a portion of the training image pairs and keypoints.

2. A method according to claim 1, wherein the position of a point in the first training image or a transformed version of the first training image is estimated by estimating, for each of a plurality of points in the first training image or a transformed version of the first training image, the probability that the point corresponds to a point to which the machine learning model (112) will assign the estimated descriptor, wherein the larger the estimated probability is, the closer the estimated descriptor is to the descriptor assigned by the machine learning model (112) to the corresponding point in the plurality of points of the first training image or the transformed version of the first training image, and each coordinate of the position of the point is estimated as a weighted sum of the values ​​of the coordinates of the plurality of points of the first training image or the transformed version of the first training image, wherein each value of the coordinate is weighted by the probability determined for the point whose position the value describes.

3. A method according to claim 1 or 2, wherein the total loss for each training image pair for which the loss is determined for each of a plurality of key points is included as a weighted sum of the losses determined for the key points, wherein the lower each determined loss is weighted, the greater the measure of the variance of the estimate of the position of a point in a first training image or a transformed version of the first training image to which the machine learning model (112) will assign the estimated descriptor and / or the estimate of the position of a point in a second training image to which the machine learning model (112) will assign the estimated descriptor when determining the loss for the corresponding key point.

4. The method of claim 3, wherein if the measure of the variance is above a threshold, the determined loss in the total loss is weighted by zero.

5. A method according to any one of claims 1 to 4, wherein the transformed version of the first training image is generated by enhancing the first training image, or wherein at least some of the training image pairs constituting the training image pairs have: the first training image is generated by enhancing the corresponding recorded camera image, and wherein the transformed version of the first training image is generated by another enhancement of the corresponding recorded camera image.

6. The method according to any one of claims 1 to 5, wherein the machine learning model (112) is a neural network.

7. A method for controlling a robot (101) for picking up or handling an object (113), comprising: Training a machine learning model (112) according to any one of claims 1 to 6; Recording a camera image, wherein the camera image displays an object in a current control scene (113); Feeding the camera image to a machine learning model (112) to produce a descriptor image; Determining from the descriptor image the location of a part for picking up or handling an object (113) in the current control scene; and The robot (101) is controlled according to the determined position.

8. The method according to claim 7, comprising: identifying a reference site in a reference image; determining a descriptor of the identified reference region by feeding the reference image to a machine learning model (112); determining a location of the reference part in the current control scene by searching the determined descriptor in a descriptor image generated from the camera image; and The position of a part for picking up or handling an object (113) in a current control scenario is determined from the determined position of the reference part. 9 . A control device configured to carry out the method according to claim 1 .

10. A computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images