Method for training a machine learning model to generate descriptor images for images showing one or more objects

Abstract

According to various embodiments, a method for training a machine learning model to generate descriptor images of images showing one or more objects is described, comprising: capturing multiple camera images, each camera image showing one or more objects; forming training image pairs from the multiple camera images, each training image pair comprising a first training image and a second training image; determining a loss for the training image pair and at least one key point from a distance between the position of the key point in the first training image or a transformed version of the first training image and an estimated position of the key point obtained by transitioning from the first training image to the second training image and back to the first training image or a transformed version of the first training image.its transformed version is determined and the machine learning model is adapted to reduce overall loss, which includes the determined losses for at least some of the training image pairs and key points.

Description

[0001] The present disclosure relates to methods for training a machine learning model to generate descriptor images for images showing one or more objects.

[0002] In order to enable flexible production or processing of objects by a robot, it is desirable for the robot to be able to handle an object regardless of the position in which the object is placed in the robot's workspace. Therefore, the robot should be able to recognize which parts of the object are in which positions so that it can, for example, grasp the object at the correct location in order to attach or weld it to another object. This means that the robot should be able to recognize the pose (position and orientation) of the object, for example, from one or more images taken by a camera attached to the robot, or to determine the position of points for picking up or processing. One approach to achieving this is to use descriptors, i.e. points (vectors) in a predefined descriptor space, for parts of the object (i.e.The robot is trained to assign the same descriptors to the same parts of an object, regardless of the object's current pose, and thus to recognize the topology of the object in the image. For example, it is known where each corner of the object is located in the image. Knowing the camera's pose allows inferences to be drawn about the object's pose. Topology recognition can be achieved using a machine learning model that is trained accordingly.

[0003] Approaches for efficient training of such a machine learning model are desirable.

[0004] According to various embodiments, a method for training a machine learning model for generating descriptor images for images showing one or more objects is provided, comprising: • Recording multiple camera images, each camera image showing one or more objects, • Forming training image pairs from the plurality of camera images, each training image pair comprising a first training image and a second training image, • For each training image pair and for each of one or more key points per training image pair (which is / are contained at least in the respective first training image) ◯ Generating a first descriptor image for the first training image by feeding the first training image to the machine learning model (wherein the first descriptor image contains a respective descriptor for each point of the first training image (which the machine learning model assigns to it based on the first training image)); ◯ Generating a second descriptor image for the second training image by feeding the second training image to the machine learning model (wherein the second descriptor image contains a respective descriptor for each point of the second training image (which the machine learning model assigns to it based on the second training image)); ◯ Determine the descriptor contained in the first descriptor image for the key point (e.g. object feature point or background point); ◯ estimating, for each of a plurality of points of the second training image (e.g., all), a probability that the point corresponds to the key point (i.e., shows the same object feature point or point in the background), wherein the probability is estimated to be greater the closer the descriptor contained in the first descriptor image for the key point is to the descriptor contained in the second descriptor image for the respective point of the second training image of the plurality of points of the second training image; ◯ estimating a descriptor that the second descriptor image would contain for the keypoint (if it actually contains the estimated descriptor for one of the points, which is not necessarily the case), by weighted sum of the descriptors that the second descriptor image contains for the plurality of points of the second training image, each descriptor being weighted by the probability determined for the point for which it is contained in the second descriptor image; ◯ Estimating the position of a point in the first training image or a transformed version of the first training image to which the machine learning model would assign the estimated descriptor (if it actually assigns the estimated descriptor for one of the points, which is not necessarily the case); and ◯ Determining a loss for the training image pair and the keypoint from a distance between the position of the keypoint in the first training image or the transformed version of the first training image and the estimated position; and • Adapting the machine learning model to reduce an overall loss (e.g., batch loss) that includes the determined losses for at least a subset of the training image pairs and keypoints.

[0005] The method described above enables simplified training data collection and, accordingly, less complex training of a machine learning model for generating descriptor images compared to, for example, methods that require RGB-D image sequences, 3D reconstructions, or other ground-truth annotation approaches. The use of a loss for self-supervised training according to the method described above enables training using unordered RGB images or even pairs of different images (i.e., different images and not just augmentations of the same image). Correspondences of points between different images are automatically determined using a "cycle" (transition from the first image to the second image and back again).

[0006] Various examples of implementation are given below.

[0007] Embodiment 1 is a method for training a machine learning model to generate descriptor images to images as described above.

[0008] Embodiment 2 is the method according to embodiment 1, wherein the position of the point in the first training image or in the transformed version of the first training image is estimated by estimating, for each of a plurality of points in the first training image or in the transformed version of the first training image (e.g. for all points), a probability that the point corresponds to a point to which the machine learning model would assign the estimated descriptor (if it actually assigns the estimated descriptor to one of the points, which is not necessarily the case), wherein the probability is estimated to be greater the closer the estimated descriptor is to the descriptor that the machine learning model assigns to the respective point of the plurality of points in the first training image orthe transformed version of the first training image, and each coordinate of the position of the point is estimated as a weighted sum of the values ​​of the coordinates of the plurality of points of the first training image or of the transformed version of the first training image, where each value of the coordinate is weighted by the probability determined for the point whose position it describes (see equations (3) and (4) below).

[0009] In other words, the "backward direction" of the cycle (ie the transition from the second image to the first image) is carried out analogously to the transition of the "forward direction" of the cycle (ie the transition from the first image to the second image) via probabilities, so that no hard selection takes place in the correspondence determination and thus differentiability is ensured, which enables the use of gradient descent methods for training.

[0010] Embodiment 3 is the method according to embodiment 1 or 2, wherein the total loss for each training image pair for which the loss is determined for each plurality of keypoints contains the losses determined for the keypoints as a weighted sum, wherein each determined loss is weighted the less, the greater a measure of the variance of the estimate of the position of the point in the first training image or in the transformed version of the first training image to which the machine learning model would assign the estimated descriptor and / or the estimate of the position of the point in the second training image to which the machine learning model would assign the estimated descriptor is when determining the loss for the respective keypoint.

[0011] This avoids a loss that is caused by a key point that is not present (ie shown) in the second training image, which would degrade the training.

[0012] Embodiment 4 is the method according to embodiment 3, wherein the determined loss in the total loss is weighted with zero if the measure of variance is above a threshold.

[0013] This protects the training process from particularly unreliable correspondence determinations. For example, the threshold is determined in such a way that losses that belong to a specific quantile (e.g., the 15% quantile) with respect to their value for the measure of variance are not included in the overall loss.

[0014] Embodiment 5 is the method according to any one of embodiments 1 to 4, wherein the transformed version of the first training image is generated by an augmentation of the first training image or wherein the forming of at least some of the training image pairs comprises that the first training image is generated by augmenting a respective recorded camera image and wherein the transformed version of the first training image is generated by another augmentation of the respective recorded camera image.

[0015] This can prevent the machine learning model from learning “shortcuts” (without correspondence to the second training image) when using the transformed version for the “backward direction”.

[0016] Embodiment 6 is the method according to any one of embodiments 1 to 5, wherein the machine learning model is a neural network.

[0017] According to various embodiments, for example, a dense object network is trained. These can achieve good results for generating descriptor images.

[0018] Embodiment 7 is a method for controlling a robot to pick up or process an object, comprising training a machine learning model according to any one of embodiments 1 to 6, picking up a camera image showing the object in a current control scenario, feeding the camera image to the machine learning model to generate a descriptor image, determining the position of a location for picking up or processing the object in the current control scenario from the descriptor image; and controlling the robot according to the determined position.

[0019] Embodiment 8 is the method according to embodiment 7, comprising identifying a reference point in a reference image, determining a descriptor of the identified reference point by feeding the reference image to the machine learning model, determining the position of the reference point in the current control scenario by searching for the determined descriptor in the descriptor image generated from the camera image, and determining the position of the point for recording or processing the object in the current control scenario from the determined position of the reference point.

[0020] Embodiment 9 is a control device configured to carry out a method according to one of embodiments 1 to 8.

[0021] Embodiment 10 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 8.

[0022] Embodiment 11 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 8.

[0023] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a robot. Fig. Figure 2 illustrates a cycle of correspondence predictions for a key point. Fig. 3 shows a flowchart illustrating a method for training a machine learning model to generate descriptor images for images showing one or more objects, according to one embodiment.

[0024] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0025] Various examples are described in more detail below.

[0026] Fig. 1 shows a robot 100.

[0027] The robot 100 includes a robot arm 101, for example, an industrial robot arm for handling or assembling a workpiece (or one or more other objects). The robot arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 by means of which the manipulators 102, 103, 104 are supported. The term "manipulator" refers to the movable components of the robot arm 101, the actuation of which enables physical interaction with the environment, e.g., to perform a task. For control, the robot 100 includes a (robot) controller 106 designed to implement the interaction with the environment according to a control program. The last component 104 (furthest from the support 105) of the manipulators 102, 103, 104 is also referred to as the end effector 104 and may include one or more tools, such as a welding torch, a gripping instrument, a painting device or the like.

[0028] The other manipulators 102, 103 (located closer to the support 105) can form a positioning device, so that, together with the end effector 104, the robot arm 101 is provided with the end effector 104 at its end. The robot arm 101 is a mechanical arm that can provide functions similar to a human arm (possibly with a tool at its end).

[0029] The robot arm 101 may include joint elements 107, 108, 109 that connect the manipulators 102, 103, 104 to each other and to the support 105. A joint element 107, 108, 109 may have one or more joints, each of which can provide rotational movement (i.e., rotary movement) and / or translational movement (i.e., displacement) for associated manipulators relative to each other. The movement of the manipulators 102, 103, 104 may be initiated by actuators controlled by the controller 106.

[0030] The term "actuator" can be understood as a component configured to effect a mechanism or process in response to its drive. The actuator can implement instructions generated by the controller 106 (the so-called activation) into mechanical movements. The actuator, e.g., an electromechanical converter, can be configured to convert electrical energy into mechanical energy in response to its drive.

[0031] The term "controller" can be understood as any type of logic-implementing entity, which may, for example, include a circuit and / or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and which can issue instructions, e.g., to an actuator in the present example. The controller can, for example, be configured by program code (e.g., software) to control the operation of a system, a robot in the present example.

[0032] In the present example, the controller 106 includes one or more processors 110 and a memory 111 that stores code and data based on which the processor 110 controls the robot arm 101. According to various embodiments, the controller 106 controls the robot arm 101 based on a machine learning model 112 stored in the memory 111.

[0033] The controller 106 uses the machine learning model 112 to determine the pose of an object 113, which is placed, for example, in a workspace of the robot arm 101. Depending on the determined pose, the controller 106 can, for example, decide which point of the object 113 should be grasped (or otherwise processed) by the end effector 109.

[0034] The controller 106 determines the pose using the machine learning model 112 based on one or more camera images of the object 113. The robot 100 can, for example, be equipped with one or more cameras 114 that allow it to capture images of its workspace. For example, a camera 114 is attached to the robot arm 101, allowing the robot to capture images of the object 113 from different perspectives by moving the robot arm 101 around. However, one or more fixed cameras can also be provided.

[0035] According to various embodiments, the machine learning model 112 is a (deep) neural network that generates a feature map for a camera image, e.g., in the form of an image in a feature space, which makes it possible to assign points in the (2D) camera image to points of the (3D) object.

[0036] For example, the machine learning model 112 can be trained to assign a specific (unique) feature value (also referred to as a descriptor value) in the feature space to a specific corner of the object. If a camera image is then fed to the machine learning model 112 and the machine learning model 112 assigns this feature value to a point in the camera image, it can be concluded that the corner is located at this point (i.e., at a point in space whose projection onto the camera plane corresponds to the point in the camera image). If the position of several points of the object in the camera image is known, the pose of the object in space can be determined.

[0037] The machine learning model 112 must be trained appropriately for this task.

[0038] An example of a machine learning model 112 for object detection is a dense object net (DON). A dense object net forms an image (e.g., an RGB image I ∈ ℝ provided by the camera 114). H×W×3 ) to an arbitrary-dimensional (with user-defined dimension D, e.g. D = 16) descriptor space image (also called descriptor image) I D ∈ ℝ H×W×D , ie each pixel is assigned a descriptor value.

[0039] Such dense visual descriptors have proven to provide a flexible, easy-to-use, and easy-to-learn object representation for robot manipulation. They demonstrate potential for class-level object generalization, they can describe non-rigid objects, and they can be seamlessly used for state representation for control.

[0040] Training a dense descriptor network can be performed using self-supervision, which relies on multiple views of the same object and dense pixel correspondence computed from the object's 3D geometry. Alternatively, RGB image augmentations can be used to generate alternative views of the same image, keeping track of pixel correspondence. These two approaches differ only in how pixel correspondences between multiple images are generated and their assumptions about the training dataset; otherwise, the same loss function can be applied, e.g., contrastive losses.

[0041] Training using pixel correspondence computed from 3D geometry encodes alternative views of the same object, resulting in view-invariant learned descriptors. However, this requires a registered RGB-D dataset or a trained NeRF (neural radiance field), which is often very costly due to camera calibration, hardware setup, and data logging. RGB image augmentation, on the other hand, only requires a disordered set of RGB images depicting the object(s), which can even be captured with a smartphone. However, the descriptors learned from these cannot handle excessive changes in camera perspective and are therefore not always view-invariant, which limits their applicability.

[0042] According to various embodiments, an approach is provided for training a machine learning model for mapping camera images to descriptor images (e.g., a DON), which enables a simple approach to training data acquisition, ie a series of unordered RGB images, each showing one or more objects, is sufficient for training, but still allows the machine learning model to learn descriptors in a view-invariant manner, ie the descriptor determination by the machine learning model is more robust to changes in the camera perspective than in training methods based on RGB image augmentation.

[0043] According to various embodiments, a cycle correspondence loss (CCL) is used for this purpose. This is a loss for the self-supervised training of a machine learning model for mapping camera images to descriptor images, which enables training based on selected (training image) pairs of training images (in particular RGB images). The idea underlying this loss can be seen in the fact that for an image pair (I A , I B ) if there are unique descriptors in the first (training) image I A any correct position prediction of a key point from image I A (e.g. a specific object feature point but also possibly a specific point in the background) in image I B again to predict the (original) position of the key point in Image I Acan be used, creating a cycle of correspondence predictions.

[0044] Fig. 2 illustrates a cycle of correspondence predictions for a key point k A .

[0045] The key point k A provides an estimate 201 of the position of the key point in image I via its descriptor (which the machine learning model assigns to it according to its current training status). B which (strictly speaking an estimated descriptor value, see below) in turn is an estimate 202 of the original position in image I A (or in this embodiment, a transformed version of it). That is, by transitioning from image I A to I Band back again (i.e., one cycle), an estimate of the original position of the keypoint is obtained. From this, a loss can be determined for this training image pair, which is smaller the better the estimate 202 corresponds to the original position. The machine learning model can then be adapted to reduce an (overall) loss that contains these losses for several training image pairs (i.e., it is adapted towards a decreasing gradient of the overall loss, i.e., its parameters (typically weights) are changed such that re-determining descriptor images for the training images of the training image pairs would yield a lower value for the overall loss).

[0046] This allows the machine model to learn to recognize valid correspondences independently, without relying on correspondence annotations. This assumes that the set of training images largely depicts the same content—but this can also include random object arrangements and different backgrounds.

[0047] For the following detailed description of the provided approach, it is assumed that two (training) images I A and I B (in this example I A and I B ∈ ℝ 3×H×W , e.g. RGB images) and it is assumed that some pixels in image I A a corresponding pixel in image I B This assumption can be met, for example, by epipolar constraints.

[0048] Using the machine learning model f θ(.) (with its current training level, ie values ​​of its parameters θ corresponding to its current training level) each of the two images is mapped to a respective descriptor image D A , D B ∈ ℝ D×H×W with descriptor values ​​from a latent descriptor space. A keypoint or its pixel position in image I A is marked with k A = (x A , y A ) and, if present, a corresponding pixel in image I B (i.e. a pixel corresponding to the same keypoint) as k B = (x B , y B ). Where x refers to the row coordinate and y to the column coordinate. Each key point k A is replaced by D A a corresponding descriptor d kA = D A [x A , y A ] ∈ ℝ D assigned, and also to the corresponding key point in I B a descriptor d kB .

[0049] The CCL requires the ability to make a prediction about the position of a keypoint from one image in another image, given its descriptor. The simplest method to B to find if d kA is to find the next descriptor in the latent space (ie descriptor space) in D B in relation to the kA While this is typically sufficient for inference (e.g., to determine the pose of an object in an image), the problem with training is that (x, y) coordinates are obtained in a non-differentiable way (so gradient descent cannot be used).

[0050] Therefore, according to various embodiments, to determine the corresponding pixel, a distance heatmap H kA→B about D B by determining the pairwise distance between d kA and each descriptor of DB, ie HxykA→B=Δ(dkA,DBxy) where Δ is a distance function, e.g., the I2 norm, or a similarity measure such as cosine similarity. The following assumes cosine similarity and normalized descriptors.

[0051] A probability distribution P(x, y | d kA , D B ) can be determined by applying a temperature-scaled softmax function to the distance heatmap, ie P(x,y|dkA,DB)=exp(HxykA→B / τ)∑i=1H∑j=1Wexp(HijkA→B / τ) where t is the temperature. If the expectation value of the marginal distributions is interpreted as coordinates, k B = (x*, y*) can be estimated by x*=μx=∑i=1Hi⋅∑j=1WP(i,j|dkA,DB) y*=μy=∑j=1Wj⋅∑i=1HP(i,j|dkA,DB)

[0052] The respective variances σx2,σy2 are accordingly expected values ​​over (x-µ x ) 2 or (y-µ y ) 2. If ground truth annotations kB(i) is the keypoint index here), e.g., in pixel-wise correspondence from the 3D geometry, it is easy to optimize the prediction error directly via the spatial expectation, e.g., with the loss function Ldist,A→B=∑iN‖kB*(i)−kB(i)‖2

[0053] According to various embodiments, the above concept is used for a fully self-supervised training procedure. This means that no ground truth kB(i) is given and an error to be optimized cannot be directly defined as in equation (5).

[0054] For the following explanation, it is temporarily assumed that for each (e.g. sampled) kA(i) a corresponding pixel kB(i) in I Bis present, even if this is unknown. Given this assumption, it is known that if the prediction kB*(i) for transition I A → I B is correct, the corresponding descriptor dkB*(i) through the transition I B → I A a prediction kA*(i) delivers so that kA(i)=kA*(i). This would complete one cycle of correspondence reconciliation.

[0055] There kA(i) is known, the prediction error can be measured, which allows the error term loss to be defined for the key point i as follows: li=‖kA*(i)−kA(i)‖ see also Fig. 2 for illustration.

[0056] Although the loss according to (6) is conceptually simple to formulate, some practical considerations must be taken into account. Below, we discuss various steps that are necessary or helpful for training a model using the outlined approach.

[0057] Although kB*(i) has an associated descriptor, the simple indexing of the descriptor image based on the coordinates is not differentiable. To maintain differentiability, according to various embodiments, the expected value is calculated over the descriptors d kB in a similar way as calculated for the spatial expectation value according to (3) and (4): d¯kB=∑i=1H∑j=1WDBij⋅P(i,j|dkA,DB)

[0058] The point according to equations (3) and (4) can be considered as an estimate of the point to which the machine learning model would assign the descriptor according to (7).

[0059] If the descriptors are normalized, the kB which is assumed to be implicitly given here. While in this case the descriptors of the hypersphere manifold S D averaging and normalization as described above leads to virtually identical results for a trained network compared to directly determining the expectation on the manifold. The distribution P(x, y | d kB , D a ) for transition I B → I A is done by means of d kB defined and kA*(i) is determined using the spatial expectation (according to equations (3) and (4)).

[0060] According to various embodiments, various augmentations are applied to the training images. For example, affine transformations (rotations, scaling), perspective distortions, and color shifts—the latter primarily for brightness and color augmentation—may be used. To ensure that the machine learning model correctly interprets the image. B is not ignored and learns a kind of identity assignment via a shortcut, a copy I Â of the input image I A generated, both or at least one of the two (I Â and I A ) are augmented and used as the first image of the training image pair or as the target image for the backward direction. If the augmentations are known, kA^(i) ie the position of kA(i) in I A known.

[0061] This case is in Fig. 2 illustrates: I A and I  are created by (e.g. random) augmentation from the same recorded camera image, they are therefore related by a transformation T(I A ) relate to each other. I B is another recorded camera image (which may also be augmented to increase the variety of training images) whose content is at least partially identical to I A (and thus lA^ ) overlaps. The key point k A is, for example, a randomly sampled location in I A and the corresponding position k  in I  is determined according to the transformation. A corresponding point in I Bcan but does not have to exist, a potential correspondence can be determined or evaluated using a variance measure (χ, see below in connection with equation (8)) (these variances can be superimposed on the training images as a heat map, for example, to make correspondences visible). Using the probability distribution according to equation (3) and the descriptor image of I B the expected descriptor d kB can be calculated according to (7). Using this expected descriptor, the process can be reversed and the key point in I  The machine learning model is adjusted to reduce the error between the true position and the estimate (e.g., summed or averaged over a batch of training image pairs).

[0062] When training with disordered RGB images, not only can the backgrounds be expected to change from image to image, but also which objects are present in the images and in which orientation. According to various embodiments, the above assumption is therefore relaxed that each kA(i) has an equivalent in I B This gives rise to two difficulties: Firstly, every l i (see equation (6)) for kA(i) without counterparts against the underlying assumption of cycle consistency, and calculated gradients could be completely counterproductive. Second, the machine learning model, since the kB is essentially a weighted sum of the descriptors that d kA are most similar, theoretically still a way of kA(i) to kA^*(i) find, even without a counterpart in I BThis could lead to shortcut learning and is typically not desirable.

[0063] In order to detect and prevent such cases, the previously determined probability distributions (according to equation (2)) are used according to various embodiments. For this purpose, the variance Xi=σx,i2+σy,i2 a prediction for the i-th keypoint. Intuitively, the variance is assumed to be small when there is a clear correspondence and the model is reliable. If there is no correspondence or the model is unreliable, the variance should increase.

[0064] For example, the variance is used in two ways. First, the q-quantile, e.g., q = 15%, is calculated over the summed variances for image I B and I Â χ i = χ Â,i + χ B,iof all N keypoints. This summed variance can be seen (for each training image pair and each keypoint) as a measure of the variance of the two position estimates in the two training images (according to (3) and (4) respectively). This yields the q% most reliably detected points, and only these are considered for the loss calculation. Second, equation (6) is modified by calculating the contribution of each individual loss l i is scaled with respect to the associated uncertainty, whereby the loss Lcyc=∑iN11+Xi‖kA^*(i)−kA^(i)‖ for each training image pair, with one added to the denominator to prevent the term from becoming prohibitively large when some Xi are less than 1. It is important that the gradients of the variances must be detached from the computational graph, otherwise the model will simply learn to make predictions with low confidence instead of solving the prediction task.

[0065] In summary, according to various embodiments, a method is provided as described in Fig. 3 shown.

[0066] Fig. 3 shows a flowchart 300 illustrating a method for training a machine learning model to generate descriptor images for images showing one or more objects, according to one embodiment.

[0067] In 301, several camera images are taken, each camera image showing one or more objects.

[0068] In 302, training image pairs are formed from the plurality of camera images, wherein each training image pair comprises a first training image and a second training image.

[0069] In 303, for each training image pair and for each of one or more key points per training image pair (which is / are contained at least in the respective first training image) • in 304, a first descriptor image is generated for the first training image by feeding the first training image to the machine learning model (wherein the first descriptor image contains a respective descriptor for each point of the first training image (which the machine learning model assigns to it based on the first training image)); • in 305, a second descriptor image is generated for the second training image by feeding the second training image to the machine learning model (wherein the second descriptor image contains a respective descriptor for each point of the second training image (which the machine learning model assigns to it based on the second training image)); • in 306 the descriptor is determined which the first descriptor image contains for the key point (e.g. object feature point or point in the background); • in 307, for each of a plurality of points of the second training image (e.g., all), a probability is estimated that the point corresponds to the key point (e.g., shows the same object feature point or point in the background), the probability being estimated to be greater the closer the descriptor contained in the first descriptor image for the key point is to the descriptor contained in the second descriptor image for the respective point of the second training image of the plurality of points of the second training image; • in 308, a descriptor that the second descriptor image would contain for the keypoint (if it actually contains the estimated descriptor for one of the points, which is not necessarily the case) is estimated by weighted sum of the descriptors that the second descriptor image contains for the plurality of points of the second training image, each descriptor being weighted by the probability determined for the point for which it is contained in the second descriptor image; • in 309, the position of a point in the first training image or a transformed version of the first training image to which the machine learning model would assign the estimated descriptor is estimated (if it actually assigns the estimated descriptor for one of the points, which is not necessarily the case); and • in 310, a loss for the training image pair and the keypoint is determined from a distance between the position of the keypoint in the first training image or the transformed version of the first training image and the estimated position.

[0070] In 311, the machine learning model is adapted to reduce an overall loss (e.g., batch loss) that includes the determined losses for at least a portion of the training image pairs and keypoints.

[0071] The procedure of Fig. 3 can be performed by one or more computers having one or more data processing units. The term “data processing unit” can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be treated according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit can comprise or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0072] According to various embodiments, the method is therefore particularly computer-implemented.

[0073] Using the trained machine learning model, a control signal for a robotic device can ultimately be generated (e.g., by using the trained machine learning model to determine an object pose or locations to be processed). Relevant locations of any type of object for which the machine learning model was trained can be determined. The term "robotic device" can be understood as referring to any physical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule for the physical system is learned, and the physical system is then controlled accordingly.

[0074] For example, images are captured using an RGB-D (color plus depth) camera, processed by the trained machine learning model (e.g., a neural network), and relevant locations within the robot's workspace are determined. The robot is controlled based on the determined locations. For example, an object (i.e., its position and / or pose) can be tracked in input sensor data. The descriptors can also be further processed to detect objects (and, for example, perform semantic segmentation), e.g., objects to be manipulated or traffic signs, road surfaces, pedestrians, and vehicles.

[0075] The camera images are, for example, RGB images or RGB-D (color plus depth) images, but can also be other types of camera images such as (only) depth images or thermal, video, radar, LiDAR, ultrasound, or motion images. Depth images are not mandatory. The output of the trained machine learning model can be used to determine object poses, for example, to control a robot, e.g., for assembling a larger object from sub-objects, moving objects, etc. The approach of Fig. 3 can be used for any pose determination method. Relevant points determined from the descriptor images can be used, for example, to determine the location of an object where the object can be grasped (or even sucked up) by a robot, and those where it should not be grasped. They thus provide valuable information for a manipulation task, for example.

Claims

[1] A method for training a machine learning model (112) for generating descriptor images for images showing one or more objects (113), comprising: Recording several camera images, each camera image showing one or more objects (113), Forming training image pairs from the plurality of camera images, each training image pair comprising a first training image and a second training image, For each training image pair and for each of one or more keypoints per training image pair Generating a first descriptor image for the first training image by feeding the first training image to the machine learning model (112); Generating a second descriptor image for the second training image by feeding the second training image to the machine learning model (112); Determining the descriptor contained in the first descriptor image for the keypoint; Estimating, for each plurality of points of the second training image, a probability that the point corresponds to the key point, wherein the probability is estimated to be greater the closer the descriptor contained in the first descriptor image for the key point is to the descriptor contained in the second descriptor image for the respective point of the second training image of the plurality of points of the second training image; Estimating a descriptor that the second descriptor image would contain for the key point by a weighted sum of the descriptors contained in the second descriptor image for the plurality of points of the second training image, wherein each descriptor is weighted by the probability determined for the point for which it is contained in the second descriptor image;Estimating the position (202) of a point in the first training image or a transformed version of the first training image to which the machine learning model (112) would assign the estimated descriptor; and; Determining a loss for the training image pair and the keypoint from a distance between the position of the keypoint in the first training image or the transformed version of the first training image and the estimated position (202); and Adapting the machine learning model (112) to reduce an overall loss that includes the determined losses for at least a portion of the training image pairs and keypoints. [2] Method according to claim 1, wherein the position of the point in the first training image or in the transformed version of the first training image is estimated by estimating, for each plurality of points of the first training image or in the transformed version of the first training image, a probability that the point corresponds to a point to which the machine learning model (112) would assign the estimated descriptor, the probability being estimated to be greater the closer the estimated descriptor is to the descriptor that the machine learning model (112) assigns to the respective point of the plurality of points of the first training image or the transformed version of the first training image, and each coordinate of the position of the point is calculated as a weighted sum of the values ​​of the coordinates of the plurality of points of the first training image orthe transformed version of the first training image, where each value of the coordinate is weighted by the probability determined for the point whose position it describes. [3] Method according to claim 1 or 2, wherein the total loss for each training image pair for which the loss is determined for each plurality of keypoints contains the losses determined for the keypoints as a weighted sum, wherein each determined loss is weighted the less, the greater a measure of the variance of the estimate of the position of the point in the first training image or in the transformed version of the first training image to which the machine learning model (112) would assign the estimated descriptor and / or the estimate of the position of the point in the second training image to which the machine learning model (112) would assign the estimated descriptor is when determining the loss for the respective keypoint. [4] The method of claim 3, wherein the determined loss in the total loss is weighted with zero if the measure of variance is above a threshold. [5] The method of any one of claims 1 to 4, wherein the transformed version of the first training image is generated by an augmentation of the first training image, or wherein forming at least some of the training image pairs comprises generating the first training image by augmenting a respective captured camera image, and wherein the transformed version of the first training image is generated by another augmentation of the respective captured camera image. [6] The method of any one of claims 1 to 5, wherein the machine learning model (112) is a neural network. [7] Method for controlling a robot (101) for picking up or processing an object (113), comprising: Training a machine learning model (112) according to one of claims 1 to 6; capturing a camera image showing the object (113) in a current control scenario; feeding the camera image to the machine learning model (112) to generate a descriptor image; Determining the position of a location for picking up or processing the object (113) in the current control scenario from the descriptor image; and Controlling the robot (101) according to the determined position. [8] Method according to claim 7, comprising: Identifying a reference point in a reference image; Determining a descriptor of the identified reference location by feeding the reference image to the machine learning model (112); Determining the position of the reference point in the current control scenario by searching the determined descriptor in the descriptor image generated from the camera image; and Determining the position of the location for picking up or processing the object (113) in the current control scenario from the determined position of the reference location. [9] Control device which is arranged to carry out a method according to one of claims 1 to 8. [10] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 8. [11] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 8.

Images