Computer-implemented method for training instance segmentation model of object detector

Abstract

A computer-implemented method for training an instance segmentation model of an object detector. The method for training the instance segmentation model comprises the following steps: providing an unlabeled image and a labeled image representing a labeled object; generating a first image by calculating one or more of the annotation objects into the unannotated image, generating a second image by calculating one or more additional annotation objects into the first image and / or removing at least one of the one or more annotation objects from the first image, generating a third image by spatially enhancing the first image; training the model in the following mode: respectively inputting the first image, the second image and the third image into the model to generate a first prediction result, a second prediction result and a third prediction result; the first prediction result represents a pseudo tag, determining an embedding loss for the first prediction result and the second prediction result, determining a regularization loss for the first prediction result and the third prediction result, the first prediction result representing a pseudo tag, and training the model using the embedding loss and the regularization loss.

Description

Prior Art

[0001] The ability of robots to manipulate objects relies heavily on their visual perception capabilities. Machine learning models can be trained to accomplish this visual perception task. Many methods for training such machine learning models rely on a large amount of annotated datasets. However, annotating (i.e., labeling) sensor data, such as images, is costly in terms of effort (and thus, cost). Therefore, it can be desirable to train machine learning models using a partially annotated dataset (i.e., a dataset with labeled images as well as unlabeled images).

[0002] One method for training machine learning models using a partially annotated dataset is semi-supervised learning (SSL). In SSL, a machine learning model is trained using the labeled images in the partially annotated dataset, and the model then uses its own prediction results as pseudo-labels to extract learning signals from the remaining unlabeled images. However, a model that learns from its own (noisy) labels at the early stage of training can stagnate instead of generalizing. SUMMARY

[0003] The present disclosure relates to a computer-implemented method for training an instance segmentation model of an object detector. The method allows for a significant reduction of the amount of required labeled data while generalizing to unfamiliar objects. This is achieved by employing semi-supervised learning (SSL) and by learning through interactions (LTI).

[0004] LTI employs temporal awareness by considering temporal changes in a scene (which typically requires a large amount of effort to annotate temporal image frames). While SSL allows for the use of a partially annotated dataset and LTI improves the model’s generalizability, merely combining the two methods exploits their shortcomings (which reinforce noisy labels across the entire sequence).

[0005] The computer-implemented method disclosed herein combines SSL and LTI in a synergistic manner, which eliminates the need for a specialized dataset required by LTI by using a pseudo-sequence (generated using SSL).

[0006] According to various embodiments, a method for training an instance segmentation model includes providing a partially labeled dataset including a plurality of labeled images and a plurality of unlabeled images, wherein each of the plurality of labeled images respectively represents one or more than one of a plurality of labeled objects, wherein each of the plurality of labeled objects is associated with a respective label; generating a plurality of image triplets, wherein generating a respective image triplet of the plurality of image triplets includes generating a first image of the respective image triplet by accounting for (e.g., adding) one or more of the plurality of labeled objects into an unlabeled image, generating a second image of the respective image triplet by accounting for (e.g., adding) one or more additional labeled objects of the plurality of labeled objects into the first image and / or by removing at least one of the one or more labeled objects from the first image, and generating a third image of the respective image triplet by spatially augmenting the first image; training the instance segmentation model using each of the plurality of image triplets, wherein training the instance segmentation model using a respective image triplet of the plurality of image triplets includes generating a first instance segmentation prediction by inputting the first image of the respective image triplet into the instance segmentation model, generating a second instance segmentation prediction by inputting the second image of the respective image triplet into the instance segmentation model, generating a third instance segmentation prediction by inputting the third image of the respective image triplet into the instance segmentation model, determining a first loss value representing an embedding (contrastive) loss of the first instance segmentation prediction and the second instance segmentation prediction, determining a second loss value representing a regularization loss of the first instance segmentation prediction and the third instance segmentation prediction, wherein the first instance segmentation prediction represents a pseudo-label, and training the instance segmentation model using the first loss value and the second loss value.

[0007] Illustratively, the method allows the model to learn by observing scene changes and to exploit visual consistency even with time gaps, and does not require curated interactive sequence data.

[0008] In the following, various examples are described.

[0009] Example 1 is a method for training an instance segmentation model as described above.

[0010] In Example 2, generating the first image of the respective image triple includes augmenting at least one (e.g., each) of the one or more annotated objects prior to incorporating the one or more annotated objects into the unannotated image; and / or wherein generating the second image of the respective image triple by incorporating the one or more additional annotated objects includes augmenting at least one (e.g., each) of the one or more additional annotated objects prior to incorporating the one or more additional annotated objects into the first image. As detailed herein, optionally, generating the second image can also include augmenting the first image itself (e.g., after incorporating and / or removing the at least one object). Illustratively, not only can the incorporated objects be augmented, but the image itself can be augmented.

[0011] Augmenting the (annotated) objects to be incorporated into the first image improves the ability of the trained model to generalize to unfamiliar objects.

[0012] In Example 3, the subject matter of Example 2 can optionally include that augmenting at least one of the one or more annotated objects and / or at least one of the one or more additional annotated objects includes one or more of: changing a scale and / or position and / or color of the at least one annotated object, rotating and / or cropping the at least one annotated object, and / or flipping the at least one annotated object.

[0013] In Example 4, the subject matter of any one of Examples 1-3 can optionally include that the plurality of annotated images includes a first number of images, and the plurality of unannotated images includes a second number of images, wherein the second number is at least nine times the first number.

[0014] As detailed above, the method allows for a significant reduction in the amount of required annotated data. In some aspects, the number of annotated images within the partially annotated dataset can be equal to or less than 10% (e.g., equal to or less than 1%). The methods disclosed herein allow for training a model on a partially annotated dataset where the number of annotated images is equal to or less than 1% (i.e., 99% of the images are unannotated), for example, the number of annotated images is equal to or less than 0.5% (i.e., 99.5% of the images are unannotated).

[0015] In Example 5, the subject matter of any one of Examples 1-4 can optionally include that incorporating (e.g., adding) respective objects of the one or more annotated objects into the unannotated image and / or incorporating (e.g., adding) respective objects of the one or more additional annotated objects into the first image includes determining a position at which the respective object is to be incorporated according to a predefined probability distribution, and incorporating (e.g., adding) the respective object at the position.

[0016] In Example 6, the subject matter of any one of Examples 1-5 can optionally include that the including (e.g., adding) respective objects of the one or more annotated objects into the unannotated image and / or the including (e.g., adding) respective objects of the one or more additional annotated objects into the first image includes determining a position at which the respective object is to be included such that a respective overlap between the respective object and each object represented by the unannotated image is equal to or less than a predefined threshold, and including (e.g., adding) the respective object at the position.

[0017] Including objects (e.g., inserting) into a scene can result in significant occlusions and can even obscure objects to be learned. Thus, including objects in a manner such that the overlap is less than a threshold (see Example 6), and / or including objects based on considering a distribution of positions of objects within the scene (see Example 5), reduces occlusions of objects in the scene and thus improves the visual perception capabilities of the trained model.

[0018] In Example 7, the subject matter of any one of Examples 1-6 can optionally include that the spatially augmenting the first image includes one or more of: color jittering, Planck jittering, Gaussian blurring, and / or changing color proportions (e.g., converting the first image to a grayscale image, re-mapping a color scheme, inverting a color scheme, etc.).

[0019] In Example 8, the subject matter of any one of Examples 1-7 can optionally include that the first instance segmentation prediction includes a plurality of class labels, where a respective class label of the plurality of class labels is for each object instance of the plurality of object instances, where the respective class label is associated with a corresponding prediction score; where the training the instance segmentation model using the respective image triplets further includes determining a first subset of class labels from the plurality of class labels that have a corresponding prediction score equal to or greater than a predefined prediction score threshold, and determining a second subset of class labels from the first subset of class labels according to a predefined quantile of highest prediction scores, where the second subset of class labels represents class labels of pseudo-labels; and where the predefined prediction score threshold and the predefined quantile increase during training of the instance segmentation model.

[0020] In Example 9, the subject matter of any one of Examples 1-7 can optionally include that the first instance segmentation prediction includes a plurality of class labels, where a respective class label of the plurality of class labels is for each object instance of the plurality of object instances, where the respective class label is associated with a corresponding prediction score; where training the instance segmentation model using the respective image triplets further includes determining a first subset of class labels from the plurality of class labels according to a predefined quantile of highest prediction scores, and determining a second subset of class labels from the first subset of class labels having a corresponding prediction score equal to or greater than a predefined prediction score threshold, where the second subset of class labels represents the class labels of the pseudo labels; and where the predefined prediction score threshold and the predefined quantile increase during training of the instance segmentation model.

[0021] Examples 8 and 9 use a combination of a predefined prediction score threshold and a predefined quantile that is dynamically adapted during training. Using this dynamic threshold prediction condition to filter the pseudo labels allows to weed out low quality prediction results, thereby improving the average precision of the trained instance segmentation model.

[0022] In Example 10, the subject matter of any one of Examples 1-9 can optionally include that the first instance segmentation prediction includes a respective mask and a respective bounding box for each object instance of the plurality of object instances; where training the instance segmentation model using the respective image triplets further includes, for each object instance, determining a respective pseudo bounding box that bounds the respective mask, where the pseudo label for the respective object instance includes the respective pseudo bounding box.

[0023] It has been found that the instance segmentation model learns to predict high quality masks well before it becomes effective in predicting bounding boxes. Therefore, determining the pseudo bounding box from the mask predicted by the instance segmentation model and using this pseudo bounding box (instead of the bounding box predicted by the instance segmentation model) for training stabilizes the bounding box prediction results during early self-supervised learning, thereby overcoming a major obstacle of SSL.

[0024] In Example 11, the subject matter of any one of Examples 1-10 can optionally include that the first instance segmentation prediction includes a respective prediction for each object instance of a first plurality of object instances; where the second instance segmentation prediction includes a respective prediction for each object instance of a second plurality of object instances; where the third instance segmentation prediction includes a respective prediction for each object instance of a third plurality of object instances; where non-maximum suppression is not applied to the first plurality of object instances and / or the second plurality of object instances and / or the third plurality of object instances during training of the instance segmentation model.

[0025] It is a convention in image detection and segmentation to apply non-maximum suppression (NMS) to eliminate redundant prediction results. It has been found that not using NMS and thus extracting additional learning signals from duplicate prediction results of the same object (instead of keeping only one prediction result per object) increases the average precision of the trained model.

[0026] Example 12 is a method for controlling a robotic device, comprising: training an instance segmentation model according to the method of any of examples 1-11; acquiring an (e.g., camera) image showing one or more objects; feeding the image into the instance segmentation model to detect the one or more objects; and controlling the robotic device in view of the detected one or more objects (e.g., controlling the robotic device to pick up an object of the one or more objects).

[0027] Example 13 is a data processing device configured to perform the method of any of examples 1-11.

[0028] Example 14 is a computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to any of examples 1-11.

[0029] Example 15 is a computer-readable (e.g., non-volatile and / or non-transitory memory) medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any of examples 1-11. BRIEF DESCRIPTION OF DRAWINGS

[0030] In the drawings, like reference numerals are generally used to refer to like components throughout the various figures. The drawings are not necessarily to scale, with an emphasis instead placed on illustrating the principles of the application. In the following description, various aspects are described with reference to the following drawings in which:

[0031] Figure 1 A robotic device arrangement according to various aspects is shown;

[0032] Figure 2 A flowchart of a method for training a machine learning model according to various aspects is shown;

[0033] Figure 3 An exemplary determination of an instance pool according to various aspects is illustrated;

[0034] Figure 4 A flowchart of training a machine learning model according to various aspects is illustrated; and

[0035] Figure 5 An exemplary distribution for binning objects into an image according to various aspects is shown. DETAILED DESCRIPTION

[0036] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and aspects of the present disclosure. Other aspects can be utilized and structural, logical, and electrical changes can be made without departing from the scope of the present disclosure. The various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure can be combined with one or more other aspects of the present disclosure to form new aspects.

[0037] In the following, various examples will be described in more detail.

[0038] Figure 1 A robotic device arrangement 100 according to various aspects is shown. The robotic device arrangement 100 can comprise a robotic device 101 (short: robot). In Figure 1 The robotic device 101 shown in Fig. 1 and described in the following by way of example is an exemplary robotic device for illustration and can comprise an industrial robot in the form of a robot arm 120, for moving, assembling or machining workpieces, for bin picking, etc. It should be noted that the robotic device is for illustration and can in general be any type of computer-controlled device, such as a robot (e.g. manufacturing robot, maintenance robot, household robot, medical robot, etc.), a vehicle (e.g. autonomous vehicle), a household appliance, a production machine, a personal assistant, an access control system, etc., as well as any other type of robotic device.

[0039] The robot arm 120 can comprise manipulators 102, 103, 104 and a base (or generally support) 105 supporting the manipulators 102, 103, 104. The term “manipulator” can refer to a movable component of the robotic device 101, the actuation of which realizes a physical interaction with the environment, e.g. to perform a task, e.g. to perform one or more skills of the robotic device 101.

[0040] For control of the robotic device 101, the robotic device arrangement 100 can comprise a (robotic) controller 106 configured to realize the interaction with the environment according to a control program. The last of the manipulators 104 (furthest from the support 105) of the manipulators 102, 103, 104 is also referred to as end effector 104 and can comprise one or more tools, such as a grasping (or gripping) tool. The grasping tool can also be a suction device (e.g. a suction head) or the like.

[0041] The other manipulators 102, 103 (closer to the support 105) can form a positioning device, such that together with the end effector 104, a robot arm 120 is provided having the end effector 104 at its end. The robot arm 120 can be a mechanical arm capable of providing similar functions as a human arm.

[0042] The robot arm 120 can comprise joint elements 107, 108, 109 interconnecting the manipulators 102, 103, 104 to each other and to the support 105. The joint elements 107, 108, 109 can have one or more joints, wherein each joint can provide rotational movement (i.e. swivel movement) and / or translational movement (i.e. displacement) of the associated manipulator relative to each other. Movement of the manipulators 102, 103, 104 can be initiated by means of actuators controlled by the controller 106.

[0043] The term “actuator” can be understood as a component which is adapted to influence a mechanism or a process in response to being driven. The actuator can implement an instruction (so-called activation) issued by the controller 106 as a mechanical movement. The actuator (e.g. an electromechanical transducer) can be configured to convert electrical energy into mechanical energy in response to being driven.

[0044] The term “controller” can be understood as any kind of logic implementing entity that can comprise, for example, a circuit and / or a processor capable of executing software, firmware or a combination thereof stored in a memory medium, and that can issue instructions, for example to an actuator, in the present example. The controller can be configured, for example by program code (e.g. software), to control operation of a system, in the present example a robot.

[0045] In the present example, the controller 106 can comprise one or more processors 110 and a memory 111 storing code and data based on which the processors 110 control the robot arm 120. According to various embodiments, the controller 106 controls the robot arm 120 based on a machine learning model (e.g. a machine learning model trained as detailed herein) 112 stored in the memory 111.

[0046] For example, a task of the robot is to perform bin picking, i.e. to grasp an object of a plurality of objects 114 (where grasping also includes picking up an object 114 with a suction cup), and to, for example, show the object 114 to a scanner or to move the object 114 to another bin. In order to be able to determine the object 114 to be picked up and to determine a suitable grasping position on the object 114, the controller 106 can use images of the robot workspace in which the object 114 is located. These images can be provided by one or more imaging sensors 113 (e.g. attached to the robot arm 120 or mounted in any other way such that the controller 106 can control the viewing angle of the one or more imaging sensors 113).

[0047] An imaging sensor used herein can be, for example, a camera (e.g., a standard camera, a digital camera, an infrared camera, an array of cameras, an event camera, a stereo camera, etc.), a radar sensor, a LIDAR sensor, an ultrasonic sensor, etc. Thus, an image can be an RGB image, an RGB-D image, or a depth image (also referred to as a D image). A depth image described herein can be any type of image that includes depth information. Illustratively, a depth image can have three-dimensional information about one or more objects. For example, a depth image described herein can include a point cloud provided by a LIDAR sensor and / or a radar sensor. For example, a depth image can be an image with depth information provided by a LIDAR sensor.

[0048] The controller 106 can be configured to control the robot arm 120 based on an output of the machine learning model 112 in response to inputting an image into the machine learning model 112.

[0049] The machine learning model 112 can be an object detector trained to accomplish such visual perception. According to various embodiments, the machine learning model 112 can be or can include an instance segmentation model. An instance segmentation model can be a prediction model for predicting an instance segmentation result. An instance segmentation result for an input image can include an instance prediction result for each pixel of the input image. The machine learning model 112 can be an object detector capable of detecting instances (i.e., having instance detection capability) using an instance segmentation result predicted by an instance segmentation model. Instance detection can provide (e.g., indicate) a location (e.g., given in pixel coordinates) and / or a bounding box of an (object) instance in an input image.

[0050] Various aspects relate to training such an instance segmentation model using a partially labeled dataset. The instance segmentation model can be generated (e.g., learned or trained) when the robotic device 101 is not in operation. The generated machine learning model 112 can then be used during operation of the robotic device 101 to determine a skill to be performed by the robotic device 101. Optionally, the generated machine learning model 112 can be additionally trained during operation of the robotic device 101.

[0051] Figure 2 A flowchart of a (computer-implemented) method 200 for training a machine learning model is shown in accordance with various aspects.

[0052] The method 200 can include (at 202) providing a partially labeled dataset including a plurality of labeled images and a plurality of unlabeled images. Each labeled image of the plurality of labeled images can represent one or more labeled objects of a plurality of labeled objects, respectively. Each labeled object of the plurality of labeled objects can be associated with a respective label.

[0053] The method 200 can include generating (in 204) a plurality of image triplets. Generating a respective image triplet of the plurality of image triplets can include generating a first image of the respective image triplet by including one or more annotated objects of the plurality of annotated objects into the unannotated image (in 204a), generating a second image of the respective image triplet by including one or more additional annotated objects of the plurality of annotated objects into the first image and / or by removing at least one of the one or more annotated objects from the first image (in 204b), and generating a third image of the respective image triplet by spatially augmenting the first image (in 204c).

[0054] The method 200 can include training (in 206) the instance segmentation model using each image triplet of the plurality of image triplets. Training the instance segmentation model using a respective image triplet of the plurality of image triplets can include generating a first instance segmentation prediction by inputting the first image of the respective image triplet into the instance segmentation model (in 206a), generating a second instance segmentation prediction by inputting the second image of the respective image triplet into the instance segmentation model (in 206b), generating a third instance segmentation prediction by inputting the third image of the respective image triplet into the instance segmentation model (in 206c), determining a first loss value representing an embedding (contrastive) loss of the first instance segmentation prediction and the second instance segmentation prediction (in 206d), determining a second loss value representing a regularization loss of the first instance segmentation prediction and the third instance segmentation prediction, where the first instance segmentation prediction represents a pseudo label (in 206e), and training the instance segmentation model using the first loss value and the second loss value (in 206f).

[0055] In the following, various aspects of the method 200 are described in further detail.

[0056] The method 200 can include a supervised learning phase and a self-supervised learning phase. In the supervised learning phase, the instance segmentation model can be trained using the annotated images 304 of the partially annotated dataset 302. With reference to Figure 3 The method 200 can include adding annotated objects 310 (referred to as annotated instances in some aspects) to the instance library 308. Each annotated object can be associated with an object in an annotated image and a corresponding label. Thus, when operating on the annotated images 304, object instances that are successfully segmented by the instance segmentation model can be stored in the instance library 308 (as annotated objects 310).

[0057] The instance segmentation model can be any kind of instance segmentation model capable of performing instance segmentation on images. As an example, the instance segmentation model can comprise a feature extractor for performing feature extraction on an input image, a feature decoder (e.g., comprising three decoding heads) for predicting, for each object instance i, a respective class, a respective bounding box, and a respective mask, and an embedding decoder capable of learning an embedding of an object query. As one example embodiment, the feature extractor can be a Resnet50, Resnet101, or Swin-L network, the embedding decoder can be a deformable DETR network, and the feature decoder can comprise a feedforward network for predicting the class, a feedforward network for predicting the bounding box, and a feature pyramid network for predicting the mask. It will be appreciated that this is merely an example, and the instance segmentation model can have any other configuration.

[0058] Training the instance segmentation model using the annotated objects 310 can employ a supervised loss wherein:

[0059] wherein is a class loss, is a bounding box loss, is a mask loss, and λ1 and λ2 are loss coefficients.

[0060] The self-supervised learning phase can employ the instance library 308. In the self-supervised learning phase, the instance segmentation model is trained using the unannotated images 306 in the partially annotated dataset 302. Figure 4 A flowchart 400 illustrating an iteration of the self-supervised learning phase using an unannotated image 402 in the unannotated images 306 is shown.

[0061] According to various aspects, an image triple comprising a first image 408, a second image 410, and a third image 412 can be generated.

[0062] The first image 408 can be generated by incorporating (e.g., adding) one or more annotated objects, such as the first object 404 and the second object 406, from the instance library 308 into (e.g., into) the unannotated image 402. Optionally, the first object 404 and / or the second object 406 can be augmented before they are incorporated into the unannotated image 402. The augmentation can be a “mild” augmentation, such as changing the scale and / or position and / or color of the annotated objects, rotating and / or cropping the annotated objects, and / or flipping the annotated objects, etc.

[0063] The second image 410 can be generated by adding (e.g., inserting) one or more additional annotated objects from the instance library 308 into the first image 408 and / or by removing at least one of the one or more annotated objects from the first image 408 (such as the second image 406). Alternatively, they can also be a“slight” augmentation of the second image 410 (e.g., as shown by the flip). Illustratively, the image itself can also be augmented when generating the second image 406 in addition to the annotated object(s) that are added. Figure 4

[0064] The third image 412 can be generated by spatially augmenting the first image 408. This spatial augmentation can be considered a“major” augmentation and can be color jitter, plank jitter, Gaussian blur, and / or changing color ratios (e.g., converting the first image to grayscale, remapping color schemes, inverting color schemes, etc.), among others. Thus, the degree of augmentation (e.g., spatial) applied to the third image 412 can be greater than the degree of augmentation (e.g., temporal) applied to the first image 408 and the second image 410.

[0065] Illustratively, the first pair of images (in some aspects referred to as a temporal pair of images) including the first image 408 and the second image 410 can represent a temporal change to the scene, and the second pair of images (in some aspects referred to as a spatial pair of images) including the first image 408 and the third image 412 can represent a spatial change to the scene. With reference to the illustrative example of the robotic device arrangement 100, the temporal pair of images can represent an interaction of the robotic arm 120 with the scene. For example, the first image 408 can represent the scene before the robotic arm 120 interacts with the scene, and the second image 410 can represent the scene after the robotic arm 120 interacts with the scene, where the interaction includes adding an object to the scene (e.g., to a bin) and / or removing an object from the scene (e.g., picking up from a bin). The methods disclosed herein allow for training a corresponding model using augmented (e.g., using the instance library) static images to simulate the interaction of the robotic arm 120 with the scene (thereby lowering the usage threshold). On the other hand, the spatial pair of images can represent other (spatial) changes to the scene, such as changes in lighting, shadows, occlusions, etc.

[0066] Inserting an object into a scene can cause significant occlusions and even obscure the object to be learned. According to various aspects, the object can be randomly inserted into the image (e.g., to generate the first image 408 and / or the second image 410) according to a probability distribution Beta(a, b). The probability distribution Beta(a, b) can be used to determine the top-left corner (u, v) of the location of the object insertion according to the following manner:

[0067] ​(u, v) = Beta(a, b) · [w, h] (2), where w and h are feasible horizontal and vertical regions, respectively, that ensure the object is contained within the image. Figure 5 An example probability distribution 500 is shown. For example, a Beta(0.5, 0.5) distribution that attempts to preserve most of the information in the image by implanting objects at the edges of the image (rather than occluding objects that are typically in the center of the input image). Moreover, when more than one object from the instance library is inserted, this approach can ensure that the overlap with previously inserted objects does not exceed a predefined threshold.

[0068] The image triplets can then be used to train the instance segmentation model 414. Illustratively, a learning signal is computed by forward passing the image triplets into the instance segmentation model 414. The instance segmentation model 414 can then output:

[0069] - a first instance segmentation prediction 416 in response to inputting the first image 408 into the instance segmentation model 414.

[0070] - a second instance segmentation prediction 418 in response to inputting the second image 410 into the instance segmentation model 414, and

[0071] - a third instance segmentation prediction 420 in response to inputting the third image 412 into the instance segmentation model 414.

[0072] The method 200 can then include (in 206d) determining a first loss value 422, the first loss value 422 representing an embedding (contrastive) loss between the first instance segmentation prediction 416 and the second instance segmentation prediction 418 (i.e., the predictions associated with the temporal image pair) embedding (contrastive) loss The embedding (contrastive) loss can be determined as detailed in T. Chen et al., “A Simple Framework for Contrastive Learning of Visual Representations,” Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607, 2020. The embedding (contrastive) loss The corresponding (i.e., same instance) embeddings can be pulled closer together, while the representations of different instances are pushed apart.

[0073] This (LTI) approach leverages temporal learning signals by enforcing consistency between the embeddings of temporal image pairs. Illustratively, the model aims to produce similar representations for the same object, even with changes in illumination, shadows, occlusions, deformations, articulations, etc.

[0074] The method 200 can then include (in 206e) determining a second loss value 424, the second loss value 424 representing a regularization loss of the first instance segmentation prediction 416 and the third instance segmentation prediction 420 According to various aspects, the first instance segmentation prediction 416 can represent pseudo labels. Thus, the class labels, bounding boxes, and segmentation masks of instances in the first image 408 can be considered as pseudo label targets (in the absence of ground truth information of unlabelled images). This regularization loss may be similar to the supervised loss (see equation (1)), with the difference that pseudo labels are used instead of (or as) ground truth labels.

[0075] It has been found that the instance segmentation model 414 learns to predict high quality masks well before it becomes effective in predicting bounding boxes. Therefore, according to various aspects, pseudo bounding boxes can be determined from the predicted masks and then can be used during training (instead of the predicted bounding boxes). This pseudo bounding box can be determined to bound the pixels u, v of the predicted mask by

[0076]

[0077] This method can be referred to as the mask-to-bounding box method.

[0078] Optionally, the pseudo class labels (thus, part of the first instance segmentation prediction 416) can be filtered to select only some of them as pseudo class ground truth information. This can reduce the errors the model suffers during self-supervised learning.

[0079] In general, a predefined prediction score threshold (to filter out pseudo class labels with a prediction score below the predefined prediction score threshold) or a predefined quantile of the highest prediction scores can be used for the pseudo class label filtering. However, when using a predefined prediction score threshold, the model can issue most of its predictions slightly below this threshold during early iterations of the self-supervised training, resulting in very few labels to help learning. On the other hand, a quantile completely ignores the scores and allows through any prediction (as long as its score meets the ranking requirement of the quantile). Because most models output a fixed number of predictions to accommodate crowded scenes (e.g., more than 300 predictions), the quantile can become too lenient and include low quality low-score predictions, potentially degrading the performance of the model as training progresses.

[0080] According to various aspects, a combination of a predefined prediction score threshold and a predefined quantile of the highest prediction score is used to filter out pseudo class labels. It has been found that this significantly improves the average precision. The largest improvement is found when first pre-filtering the pseudo class labels using a predefined prediction score threshold and then applying a predefined quantile of the highest prediction score on the pre-filtered pseudo class labels. The resulting pseudo class labels may be determined by

[0081]

[0082] wherein is the prediction score of the predicted class label for instance i, is a predefined prediction score threshold, and Q(p t ) is a predefined quantile.

[0083] According to various aspects, the predefined prediction score threshold and / or the predefined quantile Q(p t ) can be changed dynamically during training. This allows to weed out low quality prediction results, thereby improving the average precision of the trained instance segmentation model. The predefined prediction score threshold and the predefined quantile can be increased during training of the instance segmentation model 414. As an example, the probability p t of the predefined quantile can decay according to where t is the respective training iteration and T is the total number of training iterations. Thus, as training progresses, the predefined prediction score threshold filters out more pseudo class labels that are considered unusable and the quantile allows more prediction results to enter the model. The advantage of this strategy is that it mitigates false model beliefs and reduces confirmation bias. The dynamically changed predefined quantile Q(p t ) can also be referred to as a predefined and dynamic quantile.

[0084] As detailed herein, all unfiltered labels can be used for training (i.e. considered in the loss). Illustratively, NMS can not be applied, thereby extracting additional learning signals from duplicate prediction results of the same object, which increases the average precision of the trained model. This approach can be referred to as a multi-label matching approach.

[0085] The first loss value 422 and the second loss value 424 can then be used (in 206f) to train the instance segmentation model 414. According to various aspects, the first loss value 422 and the second loss value 424 can be used to determine a total loss value and the instance segmentation model 414 can be trained to reduce (e.g. minimize) the total loss value the total loss value may be given by

[0086]

[0087] wherein indicates: supervision loss only for labeled objects (due to unsupervised regularization loss for unlabeled objects), and λ3and λ4are loss coefficients. Thus, the total loss may include regularization losses (e.g., including classification loss, bounding box regression loss, mask loss) and embedding loss, which enhances the predictive power of the model even with (strong) visual modifications applied in the third image 412.

[0088] A method for controlling a robot (e.g., the robotic device 101) can include acquiring an image (e.g., using one or more imaging sensors as described herein) showing one or more objects (in the vicinity of the robotic device 101). The method for controlling the robot can include feeding the image into a robot control model (e.g., including a machine learning model) to detect the one or more objects, and controlling the robot (e.g., the robotic arm 120) in consideration of the detected one or more objects (e.g., controlling the robotic arm 120 to pick up (e.g., grasp) an object of the one or more objects).

[0089] Although in the above embodiments, Figure 2 the method in the above equations is applied to control the robotic arm 120, it can also be applied to compute control signals for controlling any technical system, like e.g. a computer-controlled machine, like a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system, in scenarios where the object level plays a role.

Claims

1. A computer-implemented method (200) for training an instance segmentation model (414) of an object detector, the method (200) comprising: Providing (202) a partially annotated dataset (302), the partially annotated dataset (302) comprising a plurality of annotated images (304) and a plurality of unannotated images (306), wherein each of the plurality of annotated images (304) represents one or more annotated objects in a plurality of annotated objects (310), wherein each of the plurality of annotated objects (310) is associated with a corresponding label; Generating (204) a plurality of image triplets, wherein generating a corresponding image triplet in the plurality of image triplets comprises: ○ generating (204a) a first image (408) of the corresponding image triplet by counting one or more labeled objects (404, 406) of the plurality of labeled objects (310) into the unlabeled image (402), ○ generating (204b) a second image (410) of the corresponding image triplet by incorporating one or more additional annotated objects from the plurality of annotated objects (310) into the first image (408) and / or by removing at least one (406) of the one or more annotated objects (404, 406) from the first image (408), and ○ generating (204c) a third image (412) of the corresponding image triplet by spatially enhancing the first image (408); Using each image triplet in the plurality of image triplets to train the instance segmentation model (414), wherein using the corresponding image triplet in the plurality of image triplets to train the instance segmentation model (414) comprises: ○ generating (206a) a first instance segmentation prediction (416) by inputting the first image (408) of the corresponding image triple into the instance segmentation model (414), ○ generating (206b) a second instance segmentation prediction (418) by inputting the second image (410) of the corresponding image triple into the instance segmentation model (414), ○ generating (206c) a third instance segmentation prediction (420) by inputting the third image (412) of the corresponding image triple into the instance segmentation model (414), ○ determining (206d) a first loss value (422) representing an embedding loss of the first instance segmentation prediction result (416) and the second instance segmentation prediction result (418), ○ determining (206e) a second loss value (424) representing a regularized loss for the first instance segmentation prediction (416) and the third instance segmentation prediction (418), wherein the first instance segmentation prediction represents a pseudo-label, and ○ The instance segmentation model (414) is trained (206f) using the first loss value (422) and the second loss value (424).

2. The method (200) according to claim 1, wherein generating (204a) the first image (408) of the corresponding image triplet comprises: before counting the one or more labeled objects into the unlabeled image, enhancing at least one of the one or more labeled objects; and / or The generating (204b) of the second image (410) of the corresponding image triple by including the one or more additional annotated objects comprises: enhancing at least one of the one or more additional annotated objects before including them in the first image.

3. The method (200) according to claim 2, The enhancing of at least one of the one or more annotation objects and / or at least one of the one or more additional annotation objects includes one or more of the following: changing the scale and / or position and / or color of the at least one annotation object, rotating and / or cropping the at least one annotation object, and / or flipping the at least one annotation object.

4. The method (200) according to any one of claims 1 to 3, wherein the plurality of labeled images (304) includes a first number of images and the plurality of unlabeled images (306) includes a second number of images, wherein the second number is at least nine times the first number.

5. The method (200) according to any one of claims 1 to 4, The step of counting the corresponding object of the one or more labeled objects into the unlabeled image and / or counting the corresponding object of the one or more additionally labeled objects into the first image comprises: determining the position at which the corresponding object is to be counted according to a predefined probability distribution, and • Counting the corresponding object at the position.

6. The method (200) according to any one of claims 1 to 5, The step of counting the corresponding object of the one or more labeled objects into the unlabeled image and / or counting the corresponding object of the one or more additionally labeled objects into the first image comprises: determining a position at which the corresponding object is to be counted such that a respective overlap between the corresponding object and each object represented by the unlabeled image is equal to or less than a predefined threshold, and • Counting the corresponding object at the position.

7. The method (200) according to any one of claims 1 to 6, The spatial enhancement of the first image includes one or more of the following: color dithering, Planckian dithering, Gaussian blurring, and / or changing a color scale.

8. The method (200) according to any one of claims 1 to 7, wherein the first instance segmentation prediction result (416) includes a plurality of class labels, wherein a corresponding class label in the plurality of class labels is for each object instance in the plurality of object instances, wherein the corresponding class label is associated with a corresponding prediction score; Wherein using the corresponding image triplets to train the instance segmentation model (414) further comprises: determining a first subset of class labels from the plurality of class labels, the plurality of class labels having corresponding prediction scores equal to or greater than a predefined prediction score threshold, and determining a second subset of class labels from the first subset of class labels based on a predefined quantile of highest prediction scores, wherein the second subset of class labels represents class labels of the pseudo labels; and Wherein the predefined prediction score threshold and the predefined quantile are increased during training the instance segmentation model (414).

9. The method (200) according to any one of claims 1 to 7, wherein the first instance segmentation prediction result (416) includes a plurality of class labels, wherein a corresponding class label in the plurality of class labels is for each object instance in the plurality of object instances, wherein the corresponding class label is associated with a corresponding prediction score; Wherein using the corresponding image triplets to train the instance segmentation model (414) further comprises: determining a first subset of class labels from the plurality of class labels based on a predefined quantile of highest prediction scores, and determining a second subset of class labels from the first subset of class labels, the first subset of class labels having corresponding prediction scores equal to or greater than a predefined prediction score threshold, wherein the second subset of class labels represents class labels for the pseudo labels; and Wherein the predefined prediction score threshold and the predefined quantile are increased during training the instance segmentation model (414).

10. The method (200) according to any one of claims 1 to 9, wherein the first instance segmentation prediction result (416) includes a corresponding mask and a corresponding bounding box for each object instance in the plurality of object instances; Wherein using the corresponding image triplets to train the instance segmentation model (414) further comprises: For each object instance, determining a corresponding pseudo bounding box bounding the corresponding mask, wherein the pseudo label for the corresponding object instance includes the corresponding pseudo bounding box.

11. The method (200) according to any one of claims 1 to 10, wherein the first instance segmentation prediction result (416) includes a corresponding prediction result for each object instance in the first plurality of object instances; wherein the second instance segmentation prediction result (418) includes a corresponding prediction result for each object instance in the second plurality of object instances; wherein the third instance segmentation prediction result (420) includes a corresponding prediction result for each object instance in the third plurality of object instances; in, During training of the instance segmentation model (414), non-maximum suppression is not applied to the first plurality of object instances and / or the second plurality of object instances and / or the third plurality of object instances.

12. A method for controlling a robotic device (101), the method comprising: · training an instance segmentation model (414) according to the method (200) of any one of claims 1 to 11; Acquire an image showing one or more objects; feeding the image into the instance segmentation model (414) to detect the one or more objects; and • controlling the robotic device taking into account the detected one or more objects.

13. A data processing device configured to perform the method according to any one of claims 1 to 11.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

15. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

Images