Method for adjusting an annotation in a dataset to train a machine learning model

Abstract

The invention relates to a method (100) for adapting an annotation in a data set for training a machine learning model, comprising the following steps: - Providing (101) the data set, wherein the data set comprises at least one image, wherein the at least one image represents a multi-component object (1) to be detected by the machine learning model, wherein the data set further comprises an annotation for the represented multi-component object (1) assigned to a respective image, - Analyzing (102) the at least one image to determine whether the representation of the multi-component object (1) in the image satisfies at least one defined criterion, - Adjusting (103) the annotation assigned to the respective image based on a result of the analysis (102), wherein the adjustment is carried out by changing a granularity of the annotation with respect to the multi-component object (1). Furthermore, the invention relates to a computer program, a device and a storage medium for this purpose.

Description

[0001] The invention relates to a method for adapting an annotation in a data set for training a machine learning model. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] In object detection, the size and position of objects in image data are predicted. A simple example is the detection of a single object type, such as a vehicle. In multi-class object detection, for example, not only the size and position of the object are predicted, but also the type of object.

[0003] For object detection applications in automated driving, the annotation classes can include, for example, different vehicle types, people, and other objects that may be visible during driving, such as signs, barriers, and pylons. However, the annotation of vehicles with multiple and potentially overlapping components, such as semi-trailers consisting of a tractor unit and a trailer, is sometimes inconsistent. Disclosure of the invention

[0004] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9, and a computer-readable storage medium having the features of claim 10. Further features and details of the invention emerge from the respective subclaims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention, and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure of the individual aspects of the invention, reference is or can always be made to each other.

[0005] The invention particularly relates to a method for adapting an annotation in a dataset for training a machine learning model, comprising the following steps, wherein the steps can be performed repeatedly and / or sequentially. An annotation can also be understood and referred to as a label and serves in particular as a reference, in particular as ground truth, for training the machine learning model with the dataset.

[0006] In a first step, the data set is preferably provided, wherein the data set comprises at least one image. The at least one image represents, in particular, a multi-component object to be detected by the machine learning model. The at least one image can result, for example, from detection by a camera sensor, a radar sensor, an ultrasonic sensor, a LiDAR sensor, or a thermal imaging camera. Accordingly, the image can be embodied as a camera image or also as a radar image, ultrasonic image, thermal image, or LiDAR image. The data set can also comprise at least one image that has been synthesized, i.e., in particular, simulates the real data of one of the aforementioned sensors. The data set preferably further comprises an annotation for the represented multi-component object, assigned to a respective image.

[0007] In a further step, the at least one image is preferably analyzed to determine whether the representation of the multi-component object in the image meets at least one defined criterion. The analysis can be carried out using various algorithms to recognize patterns and / or features in the at least one image. In order to check whether the representation of the multi-component object in the image meets at least one defined criterion, at least one property, such as a size or an association of individual parts, can be assigned to the multi-component object or individual parts of the multi-component object in the respective image during the analysis, and this at least one property can be compared with a respective defined limit or comparison value for a respective property.

[0008] In a further step, the annotation assigned to the respective image is preferably adapted based on the result of the analysis, wherein the adaptation is carried out by changing the granularity of the annotation with respect to the multi-component object. The granularity of the annotation expresses, in particular, the number of levels at which the annotation is performed, i.e., the level of detail or nuance. For example, the multi-component object as a whole can be assigned a single annotation, which would correspond to an annotation at a higher-level, or individual parts or components of the multi-component object can receive a respective annotation, which would correspond to an annotation at a lower-level.When adapting the annotation, a boundary, for example a bounding box, can also be adapted for the multi-component object representing the annotation or for the part of the multi-component object that is represented by the annotation.

[0009] Preferably, it can be provided that the adaptation of the annotation assigned to the respective image is carried out in such a way that a single annotation for the multi-component object is divided into at least two annotations for respective parts of the multi-component object, or that at least two annotations for respective parts of the multi-component object are combined into a single annotation for the multi-component object. The division or combination further comprises, in particular, the boundary for the multi-component object representing the annotation or for the part of the multi-component object that is represented by the annotation. During the division, it can be provided that the newly determined boundaries of the parts of the multi-component object overlap, in particular over a defined length. This can advantageously represent a sense of togetherness. It can further be provided that the individual parts, orComponents can be linked together via descriptive parameters despite the annotation being split up. During the summarization, it can be provided that the summarized multi-component object adopts at least one property of the individual parts, such as an inclination in the case of a vehicle as a multi-component object. The respective parts can be individual components of the multi-component object. In the example of a truck, for example, a respective part can be a front, or a tractor, and a trailer of the truck. This advantageously allows either individual parts of the multi-component object or the multi-component object as a whole to be more strongly represented in the data set by the adapted annotation and thus to be taken more into account when training the machine learning model.

[0010] Optionally, it is conceivable that either the splitting of the individual annotation or the combining of the at least two annotations is carried out uniformly for the data set in order to provide a consistent annotation granularity in the data set with regard to the multi-component object. The uniform implementation expresses in particular that for respective images of the data set which have an undesired annotation granularity, e.g. an annotation at a subordinate level although an annotation is only desired at a superordinate level, the splitting or the combining is carried out accordingly. Thus, advantageously, an annotation is provided uniformly for the data set either consistently at a superordinate or a subordinate level.

[0011] It can further advantageously be provided that the at least one defined criterion is a size of the multi-component object and / or a shape of the multi-component object and / or an association of individual parts of the multi-component object. For example, the annotation can be divided into multiple annotations if the multi-component object exceeds a certain size and individual parts or components are associated with one another, for example, through a connecting element or a common direction of movement. The common direction of movement could be determined, for example, by analyzing multiple images.

[0012] It is also optionally conceivable that the analysis of the at least one image is carried out on the basis of a semantic segmentation and / or a pixel-based classification in order to detect and / or classify the multi-component object and / or individual parts of the multi-component object.

[0013] A further advantage can be achieved within the scope of the invention if the machine learning model is trained on the basis of the data set for classification and / or object detection. Accordingly, the training can result in a trained machine learning model that can be used for classification and / or object detection. The use and thus the inference can be provided, for example, in a vehicle. The data points of the input data can, for example, be pixels of image data or be based on these in order to carry out the classification and / or object detection of the data points based on the pixels. The input data can comprise sensor and / or image data that result at least partially from acquisition with a sensor, preferably a camera sensor, and / or that have been at least partially synthesized, i.e. in particular simulate the real data of a sensor.Specifically, it can be provided that the surroundings of a sensor and / or a vehicle and / or a traffic scene are represented by the values ​​of image points, preferably pixels, of the image data. Classification, preferably image classification and / or object detection, can be provided on the basis of these values. This makes it possible, for example, to detect objects in the surroundings and / or the traffic scene. The classification can also be provided in the form of semantic segmentation (i.e., pixel- or area-wise classification) and / or object detection. The image data can, for example, be images from a radar sensor and / or an ultrasonic sensor and / or a LiDAR sensor and / or a thermal imaging camera.

[0014] Accordingly, the images can also be implemented as radar images and / or ultrasound images and / or thermal images and / or lidar images.

[0015] Furthermore, it is optionally possible within the scope of the invention for the multi-component object to be a vehicle, in particular a truck, wherein, as part of the analysis based on the at least one defined criterion, it is preferably at least determined whether the vehicle, based on the at least one image, exceeds a defined length. The method can be particularly advantageous for trucks, since a different configuration, for example with or without an articulated joint, can lead to a different type of movement, which can be particularly important for a prediction in the context of at least partially automated driving. Furthermore, other vehicles with multiple components are also conceivable, such as a passenger car with a trailer or similar configurations.

[0016] It is possible for the method according to the invention, in particular the machine learning model trained with the data set, to be used in a vehicle. The vehicle can be configured, for example, as a motor vehicle and / or passenger vehicle and / or an at least partially automated vehicle. The vehicle can have a vehicle device, for example, for providing an at least partially automated driving function and / or a driver assistance system. The vehicle device can be designed to control the vehicle at least partially automatically and / or to accelerate and / or decelerate and / or steer.

[0017] The invention also relates to a computer program, in particular a computer program product, comprising instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. Thus, the computer program according to the invention provides the same advantages as those described in detail with reference to a method according to the invention.

[0018] The invention also relates to a data processing device configured to carry out the method according to the invention. The device can be, for example, a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0019] The invention may also provide a computer-readable storage medium that has the computer program according to the invention and / or includes instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0020] Furthermore, the method according to the invention can also be implemented as a computer-implemented method.

[0021] Further advantages, features, and details of the invention will become apparent from the following description, which describes embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show: Fig. 1 a schematic visualization of a method, a device, a storage medium and a computer program according to embodiments of the invention, Fig. 2 a schematic representation of vehicles with boundary boxes according to embodiments of the invention, Fig. 3a a schematic representation of the summarization of annotations according to embodiments of the invention, Fig. 3b a schematic representation of the splitting of annotations according to embodiments of the invention.

[0022] In Fig. 1, a method 100, a device 10, a storage medium 15 and a computer program 20 according to embodiments of the invention are schematically shown.

[0023] Fig. 1 shows, in particular, a method 100 for adapting an annotation in a data set for training a machine learning model. In a first step 101, the data set is provided. The data set comprises at least one image, wherein the at least one image represents a multi-component object 1 to be detected by the machine learning model. Furthermore, the data set comprises an annotation assigned to a respective image for the represented multi-component object 1. In a second step 102, the at least one image is analyzed to determine whether the representation of the multi-component object 1 in the image fulfills at least one defined criterion. In a third step 103, the annotation assigned to the respective image is adapted based on a result of the analysis 102, wherein the adaptation is carried out by changing a granularity of the annotation with respect to the multi-component object 1.

[0024] One advantage of the method according to embodiments is an improvement in consistency when annotating images of a dataset for training a machine learning model. Consistency can be particularly important for classes with poor representation within the dataset. Furthermore, according to embodiments, a focus can be placed on images with objects, in particular vehicles, with multiple components and potentially overlapping components, which can also be referred to as multi-component objects 1 within the scope of the present invention.

[0025] For example, in a case where the term ‘truck’ is used exclusively for multi-component objects 1 related to heavy goods vehicles (HGVs), a division can be made between the ‘truck front’, where the driver is located, and the ‘truck trailer’, where the load is located, regardless of whether the truck has an articulation or not.

[0026] In another case, for example, a merger mixes a parent-level class definition (group of objects) with a child-level class definition (individual objects). Furthermore, some objects annotated with the parent class definition (e.g., an entire truck) may also include a child class definition (e.g., only a tractor-trailer), which can lead to semantic confusion because parts of the multi-component object 1 have two potentially valid annotations, e.g., "truck" and "truck-trailer" or "truck" and "tractor-trailer."

[0027] This problem can be solved by splitting the parent level into child levels, for example, the "truck front" and the "truck trailer." Furthermore, it is preferable that each part 2 of the truck has only one valid annotation and only one valid boundary 3 (bounding box).

[0028] The problems thus include, in particular, the use of multiple annotation levels, or granularities, for annotations of multi-component objects 1. Multi-component objects 1 with the parent annotation level can be very similar to the annotations of the child level in the given image data, and it can be difficult for object detection to distinguish when a parent or child annotation should be applied.

[0029] Fig. Figure 2 shows various examples of trucks annotated as two different objects (left column) or as one object (right column), with rectangular and non-rectangular loads (top and bottom rows, respectively). While the annotation “truck” is preferably applied to box trucks (single-unit, non-articulated trucks), the annotation can also be applied to semi-trailers consisting of a tractor unit and a trailer if the distinction is not visible, for example, due to orientation, distance, or occlusion. The type of annotation is based on the boundary(s) 3 in Fig. 2 represents.

[0030] One aspect of the invention is in particular the use of a scheme for adapting annotations in order to reduce the number of classes and the number of annotation levels of classes in order to advantageously increase consistency in problematic class definitions in data sets.

[0031] This can be done, for example, in one of the following two ways. Class annotations can only be used at a higher level, whereby for multi-component objects 1, only the composite of these components is considered an "object." To do this, the subclasses "Truck-Tractor" and "Truck-Trailer" can be removed and only the term "Truck" can be used for truck-related objects. In the case of semi-trailers consisting of a tractor and a trailer, the two bounding boxes 3 can be merged and the class "Truck" can be applied to the resulting object, as in Fig. 3a in a bird's eye view and side view. When implementing this annotation adaptation, a semi-trailer truck can be identified, for example, by a connecting element that connects the parts 2, or components, to each other, as well as by checking the distance between the parts 2. The combined bounding box 3 can be determined by a smallest box that can enclose both the tractor unit and the trailer in a bird's eye view. The z-dimensions of the combined box 3 preferably completely encompass both the tractor unit and the trailer. The combined truck object in particular adopts all properties of the tractor unit that are not related to position and dimensions (e.g., inclination, roll, properties related to occlusion, as well as a vehicle type and / or an activity). The merging of semi-trailers can advantageously increase the representation of the “Truck” class.

[0032] Examples of truck objects that are preferably not merged include stand-alone tractor units, trailers that are not connected to a tractor unit, trailers that are not directly connected to the tractor unit in a multi-trailer configuration, or tractor units that are pulled by another tractor unit on a transporter. These multi-component objects 1 can be identified during analysis 102, so that no merging is performed for corresponding images.

[0033] According to a second type, classes can only be annotated at a subordinate level. For objects that comprise several components, i.e., multi-component objects 1, preferably only the smallest describable parts 2, or components, are considered as "objects" and annotated accordingly, and the composite multi-component object 1 is not directly considered. However, it can be provided that the individual parts 2, or components, are linked to one another via descriptive parameters. To implement this, the class "Truck" can be eliminated and only the subordinate classes "Truck Front" and "Truck Trailer" are considered. This scheme is shown in Fig.3b in a bird's eye view and side view. For example, if the total length exceeds a threshold (e.g., 8.2 m), the truck can be assumed to be a semi-trailer truck, and the truck bounding box 3 can be split into two overlapping bounding boxes 3. In this case, the fronts of the truck can be set to a specific length (e.g., 5.5 m). The length of the truck trailer can be chosen so that the two components overlap at a specific distance, for example, 3 m. These values ​​represent an example estimate based on the most common sizes for truck fronts and can be determined specifically for a respective dataset. For trucks with a length below this threshold, it can be assumed to be a panel van, where the front (the cab) and the trailer do not overlap.In this case, it is preferable to set the truck's front end to a specific length (e.g., 2.2 m), and select the truck's trailer so that it has no overlap. In all cases, the truck's height can be used for both the truck's front end and the truck's trailers, although the trailers may actually be taller.

[0034] Semi-trailer trucks are described above as an example, but this concept is analogously applicable to other objects that comprise multiple components. In particular, these multi-component objects also have overlapping bounding boxes 3 that move in a coordinated manner and that can exist separately or together (e.g., objects that tow other objects, objects that carry or push other objects, objects that move in coordinated motion with other objects).

[0035] The advantages of the invention include, in particular, that according to advantageous embodiments, only one annotation level is used for groups of class annotations, thus providing consistent annotation. Furthermore, the schema can advantageously be applied ad hoc to existing data sets to adapt the annotations according to embodiments.

[0036] According to embodiments, the method can be applied to various types of image data, for example, camera, LiDAR, radar, or ultrasound. The method according to embodiments can be used for all downstream tasks that require the use of a data set, in particular those that require object detection of the relevant classes within the context of a driving application.

[0037] The above explanation of the embodiments describes the present invention exclusively by way of examples. Of course, individual features of the embodiments can be freely combined with one another, provided they are technically feasible, without departing from the scope of the present invention.

Claims

[1] A method (100) for adapting an annotation in a dataset for training a machine learning model, comprising the following steps: - providing (101) the data set, wherein the data set comprises at least one image, wherein the at least one image represents a multi-component object (1) to be detected by the machine learning model, wherein the data set further comprises an annotation associated with a respective image for the represented multi-component object (1), - analyzing (102) the at least one image to determine whether the representation of the multi-component object (1) in the image satisfies at least one defined criterion, - adapting (103) the annotation associated with the respective image on the basis of a result of the analysis (102), wherein the adaptation is carried out by changing a granularity of the annotation with respect to the multi-component object (1). [2] Method (100) according to claim 1, characterized by that the adaptation (103) of the annotation assigned to the respective image is carried out in such a way that a single annotation for the multi-component object (1) is divided into at least two annotations for respective parts (2) of the multi-component object (1) or that at least two annotations for respective parts of the multi-component object (1) are combined into a single annotation for the multi-component object (1). [3] Method (100) according to claim 2, characterized by that either the splitting of the individual annotation or the merging of the at least two annotations is carried out uniformly for the data set in order to provide a consistent granularity of the annotation in the data set with respect to the multi-component object (1). [4] Method (100) according to one of the preceding claims, characterized bythat the at least one defined criterion is a size of the multi-component object (1) and / or a shape of the multi-component object (1) and / or an association of individual parts (2) of the multi-component object (1). [5] Method (100) according to one of the preceding claims, characterized by that the analysis of the at least one image is carried out on the basis of a semantic segmentation and / or a pixel-based classification in order to detect and / or classify the multi-component object (1) and / or individual parts (2) of the multi-component object (1). [6] Method (100) according to one of the preceding claims, characterized by that the machine learning model is trained based on the dataset for classification and / or object detection. [7] Method (100) according to one of the preceding claims, characterized bythat the multi-component object (1) is a vehicle, in particular a lorry, wherein in the context of the analysis (102) on the basis of the at least one defined criterion it is preferably at least determined whether the vehicle exceeds a defined length based on the at least one image. [8] Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to one of the preceding claims. [9] Device (10) for data processing, which is arranged to carry out the method (100) according to one of claims 1 to 7. [10] A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) according to any one of claims 1 to 7.

Images