Method and system for automatically annotating sensor data

EP4463831A1Pending Publication Date: 2024-11-20DSPACE SE & CO KG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
EP2023701249
Authority / Receiving Office
EP · EP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-01-14
Filing Date
2023-01-13
Publication Date
2024-11-20

AI Technical Summary

Technical Problem

Current methods for annotating sensor data, particularly for autonomous driving, are hindered by the need for massive diverse training data and the inefficiency of human annotation processes, which are time-consuming and costly, limiting the feasibility of large-scale validation data generation.

Method used

A computer-implemented method using neural networks to automatically annotate sensor data frames by grouping data points based on state attributes, selectively retraining the network to improve quality, and reducing manual quality checks through iterative sampling and correction processes.

Benefits of technology

This approach significantly reduces manual effort and time required for annotation, enabling high-quality automated annotation with minimal human intervention, thereby facilitating larger annotation projects and improving neural network accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 1.1
    Figure 1.1
Patent Text Reader

Abstract

The invention relates to a computer-implemented method for automatically annotating sensor data frames, such as image frames or audio frames. Received sensor data frames are annotated, with each sensor data frame being assigned at least one data point and at least one state attribute being assigned to each data point. The data points are grouped on the basis of the at least one state attribute, the groups comprising defined value ranges of the state attribute. A sample of data points is selected from a first group and a quality level is determined for this sample. If the quality level of the first sample is below a predefined threshold value, the neural network is retrained on the basis of corrected annotations for the first sample. Once the quality level is above the predefined threshold value, the annotated sensor data frames are exported.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method and system for automatically annotating sensor data

[0002] The present invention relates to methods and computer systems for automatically annotating sensor data frames, in particular data frames from an image acquisition sensor.

[0003] Autonomous driving promises unprecedented levels of comfort and safety in everyday traffic. Despite enormous investments by various companies, existing approaches are only applicable under limited circumstances and / or only provide a subset of truly autonomous behavior. One reason for this is the lack of a sufficient quantity and variety of available driving scenarios. Thus, further progress is hampered by the need for enormous amounts of sufficiently diverse training data as well as validation data (i.e., independent ground truth data). The preparation of training data generally requires the recording of many different driving scenarios by a vehicle equipped with a set of sensors, in particular image acquisition sensors such as one or more cameras, a lidar sensor, and / or a radar sensor.Before using these recorded scenarios as training data, they must be annotated.

[0004] This is often done by annotation service providers who receive the recorded sensor data and divide it into work packages for a large number of human workers, also known as labelers. The exact annotations required (e.g., the object classes to be distinguished) depend on each project and are specified in the detailed labeling specification. The client delivers the raw data to the annotation service provider and expects high-quality annotations according to their specifications within a short timeframe. The number of labelers required to complete the annotation project increases with the growing amount of delivered data and increases with the decreasing timeframe for a fixed amount of data. For this reason, larger annotation projects, e.g.would provide enough ground truth data to validate an autonomous vehicle, would not be feasible with human labor alone, but would require automation of the annotation process.

[0005] Automation approaches use neural networks to label the recorded sensor data. An initial set of received data is manually labeled and then used to train neural networks. Once sufficiently trained, the neural networks can annotate the bulk of recorded image sensor data. Compared to a purely manual approach, this significantly reduces the effort. However, maintaining high annotation quality still requires time-consuming human quality checks. Since the quality assurance process must still be applied to all annotations, there is a linear relationship between the project volume and the effort required to meet project requirements.

[0006] Thus, improved methods for automatically annotating sensor data, especially image acquisition sensor data, are needed; it would be particularly desirable to ensure high annotation quality with a reduced number of manual quality checks.

[0007] It is an object of the present invention to provide methods and computer systems for automatically annotating sensor data frames, in particular video frames or lidar point clouds.

[0008] In a first aspect of the invention, a computer-implemented method for automatically annotating sensor data frames is provided; the method comprises the steps of receiving a plurality of sensor data frames, annotating the plurality of sensor data frames using at least one neural network, wherein the annotating comprises assigning at least one data point to each sensor data frame and at least one state attribute to each data point, grouping the data points based on the at least one state attribute, wherein a first group comprises data points for which the at least one state attribute lies within a defined range of values, selecting a first sample of one or more data points from the first group, and determining a quality measure for the data points in the first sample.If the computer determines that the quality measure of the first sample is below a predefined threshold, the method further comprises the steps of receiving corrected annotations for the data points in the first sample, retraining the neural network based on the data points in the first sample, selecting a second sample from one or more data points of the first group that were not in the first sample, annotating the sensor data frames of the second sample with the retrained neural network, and determining a quality measure for the data points in the second sample. Once the computer determines that the quality measure of the first or second sample is above a predefined threshold, the method further comprises annotating the remaining sensor data frames of the first group with the neural network, and exporting the annotated sensor data frames of the first group.

[0009] The computer system that carries out the method according to the invention can be implemented as a single host computer comprising a processor, e.g., a general-purpose microprocessor, a screen, and an input device. Alternatively, the computer system can also comprise one or more servers having a plurality of processing elements such as processor cores or dedicated accelerators, wherein the servers are connected via a network to a client comprising a screen and an input device. In this way, the annotation or the automation software, which comprises components for automatic annotation, can be executed partially or entirely on a remote server, e.g., in a cloud computing environment, so that only a graphical user interface needs to be executed locally.

[0010] A data point can describe an object or feature in a sensor data frame, in particular an image or a lidar point cloud, or specify a property of the sensor data frame. The verification of a data point is expediently carried out on the sensor data frame that contains the object or feature associated with the data point or has the property. A sensor data frame can comprise multiple data points, and the verification of a first data point in a sensor data frame can be performed independently of the verification of a second data point in the sensor data frame. For example, an object in a camera image can be annotated with data points in the form of a bounding box and an object class; depending on the class of the object, such as a passenger car in particular, this object can also be annotated as a data point with further attributes such as a turn signal status.The number of assigned data points can depend on the content of the sensor data frame, so an empty sensor data frame can also occur, to which no data point is assigned. This empty sensor data frame would then be ignored in further processing. Such ignoring of empty sensor data frames should be included in the assignment of at least one data point to each sensor data frame.

[0011] A state attribute can describe the environmental conditions or, more generally, the circumstances that prevailed when an object or feature to which the data point is assigned was recorded. The state attribute can be a static state attribute that specifically describes environmental conditions that existed at the time of recording. An environmental condition existing during the recording of the frame can have a different impact on the accuracy of the annotations depending on the type of data point. In general, for annotations that include multiple data points, the impact of a state attribute can vary depending on the data point or type of data point. For example, if the sensor data includes camera images taken at night, the position and / or class of an object can be more difficult to determine.However, an attribute of a car, such as the state of a turn signal, may be easier to perceive at night than in broad daylight. The state attribute can also be a dynamic state attribute determined during annotation using a neural network – and can also be a standalone data point. One or more of the state attributes of one type of data point can be state attributes of another type of data point. Even under the same environmental conditions, a distant object, for example, can be more difficult to detect, which both complicates classification and limits the accuracy of a bounding box. The size of the first object has no influence on the annotation quality of a second object (although any occlusion of the second object certainly does).

[0012] By grouping data points based on at least one state attribute, possible correlations between state attributes and annotation accuracy can be considered. The grouping of data points can be performed independently for different types of data points. The state attributes can have different effects from one data point type to another, so that individual data points of a specific type are preferably grouped together, while different data point types are preferably grouped according to different criteria. The invention makes it possible to identify static and dynamic state attributes that negatively impact annotation quality and to improve the neural network under these conditions through selective retraining. Furthermore, it becomes possible to reduce the effort required for manual correction and quality control.The term neural network can refer to a single neural network, a combination of different neural networks according to a predefined architecture, or any type of machine learning-based technology that learns from training data in a supervised, semi-supervised, or unsupervised manner. Different neural networks can be used for different data points; object position and / or classification can be determined with a first neural network, while attributes of the object can be determined with at least one additional neural network.

[0013] The present invention is based on the consideration that in many cases the quality of the various components of an annotation varies systematically. For example, an annotation could consist of a two-dimensional bounding box, an object class, and other attributes such as a turn signal status. While the quality is acceptable, for example for the object class, the positioning of the bounding box may need to be corrected. A single data point therefore represents the smallest unit of an annotation whose quality can be determined independently of other components of the annotation. It is useful to distinguish between different types of data point: A bounding box describes fundamentally different properties of an object than, for example, an attribute such as the turn signal status.This means that a state attribute can affect the quality of different types of data points in different ways, and some state attributes may have no influence on one type of data point, while they are crucial for others. By decomposing a complex annotation into individual data points, it becomes possible to determine the influence of state attributes on the quality of annotations at a fine-grained level and to take this into account during correction.

[0014] The steps of selecting a second sample of one or more data points from the first group that were not included in the first sample and annotating the sensor data frames of the second sample with the retrained neural network can be interchanged. For example, an entire batch of sensor data frames can be annotated with the retrained network before a second sample is selected.

[0015] Especially when the neural network needs to be retrained multiple times, the computational load is reduced, since only the data points of the second or subsequent sample need to be annotated with the retrained network. For the majority of sensor data frames, annotation can be postponed until the sample review determines that the retrained network delivers annotations of sufficiently good quality.

[0016] Exporting the annotated frames can, for example, involve saving the frames to an external storage device and / or converting or merging them into a predefined data format. Due to the fine granularity of the data points, it would also be possible, in principle, to deliver partially annotated sensor data frames. For the sake of clarity, it may be advantageous to only deliver sensor data frames to customers when a sufficiently retrained neural network is available for all occurring data point types—thus allowing the sensor data frames to be fully annotated.

[0017] Because manual work is at least largely used only to create training, test, and / or validation data for systematically improving the neural network or another machine learning-based automation component for annotating the sensor data frames, the effort required for large annotation projects can be considerably reduced. Typically, after several iterations of retraining the neural network, the quality level is sufficient to deliver automation results, i.e., annotations by the neural network, without the need for further manual checks. However, these can still be performed with a sample size, preferably small, independent of the data volume. The method according to the invention further reduces the necessary manual effort and time by focusing the retraining on those conditions where annotation quality is still lacking.

[0018] As a quality measure, for example, an area coverage between an automatically created bounding box and a bounding box manually created or adjusted as part of quality control can be used. A maximum number and / or maximum proportion of incorrectly assigned object classes and / or false positives and / or false negatives can also be required. The quality measure would then be below the predefined threshold, for example, if the bounding boxes have insufficient coverage. As a quality measure, it can also be specified that a maximum of a predefined number of false positives or incorrectly detected objects and / or false negatives or incorrectly not detected objects may occur in a sample from a predefined number of frames. The quality measure would then be below the predefined threshold, for example, if the maximum permitted number of undetected objects in the sample was exceeded.When determining the quality measure, combined conditions can also be used, for example by weighted summarization of individual values.

[0019] If the computer determines that the quality measure of the second sample is below a predefined threshold, the method preferably comprises the further steps of receiving corrected annotations for the data points in the currently checked sample, retraining the neural network based on the data points in the currently checked sample, selecting another sample from one or more data points of the first group that were not part of a previous sample, annotating the sensor data frames of the further sample with the neural network, and determining a quality measure for the data points of the further sample. The further steps are expediently repeated until the computer determines that the quality measure for the frames in the further sample is above the predefined threshold or no sensor data frames with uncorrected data points remain for the sample.Once the quality measure for the sensor data frames of a sample exceeds the predefined threshold, the method further comprises the steps of annotating the remaining sensor data frames of the first group with the—then sufficiently retrained—neural network and exporting the annotated sensor data frames of the first group. This approach enables rapid improvement of the neural network through a limited number of iterations.

[0020] Preferably, the step of receiving a plurality of sensor data frames comprises a step of preprocessing the sensor data frames, wherein at least one state attribute is determined by a dedicated neural network based on the frame, and / or at least one of the state attributes is determined based on additional sensor data recorded simultaneously with the sensor data frame. A dedicated neural network refers, in particular, to a neural network specifically trained for the respective question. The additional sensor data can be combined and / or used for queries to various services, e.g., indicating weather conditions or a type of lighting condition based on time and geographical location.

[0021] In one embodiment, the sensor data frames are image data frames, i.e., they include data from an imaging sensor, such as one or more cameras, a lidar sensor, and / or a radar sensor. The received sensor data may also include additional sensor data recorded simultaneously with the image data frames, such as a GPS position, vehicle acceleration, or data from a rain sensor.For image data frames, the state attribute is preferably a geographical location, a time of day, a weather condition, a visibility condition, a road type, a distance to an object and / or a traffic density, a size of a bounding box, an extent of occlusion and / or clipping, an ego-vehicle speed, a camera parameter, a color range and / or a contrast measure of an area encompassed by a bounding box, a heading of the ego-vehicle, or includes astronomical information such as the sun's position relative to the heading of the ego-vehicle.The distance to an object can be a distance to the nearest object, a distance to a farthest object, or an average distance to a plurality of objects detected in the frame; by considering the distance to an object as an environmental condition during recording, the influence on the object detection and / or classification performance of a neural network can be quantified. For image data frames, the at least one data point preferably includes a position of an object, a class of an object, coordinates of a bounding box, coordinates of a line, a truncation of an object, an occlusion of an object, a correlation of an object in the image data frame with an object in a previous or subsequent image data frame (as a result of tracking the object), and / or an activation of a light indicator, such as a turn signal or a brake light.The number of data points can depend on the content of the image data frame, for example, many cars and pedestrians in a large city scene with a corresponding number of object positions, object classifications, and possible attributes for the corresponding object class. For a pedestrian, for example, clothing, pose, and / or gaze direction could represent additional attributes or data points.

[0022] In one embodiment, the received sensor data frames are audio frames, i.e., comprise data from an audio sensor such as a microphone. For audio frames, the state attribute is preferably a geographical location, a gender and / or an age of a speaker, a room size, and / or a measure of background noise. For audio frames, the at least one data point comprises a phoneme and / or one or more words of text recognized from the audio frame. Words may be recognized from a plurality of subsequent audio frames, so that a data point can be derived from a plurality of audio frames. The difficulty of recognizing speech may depend, for example, on the frequency range produced by a speaker, the presence of reverberation or echo from the room, and / or the level of background noise present.

[0023] Preferably, the grouping of the plurality of data points comprises determining clusters in a multidimensional space, in particular using a nearest-neighbor algorithm and / or an unsupervised learning approach and / or a machine learning classification model. Preferably, data points of one type are assigned by the machine learning classification model to exactly one of at least two clusters, which have different expected quality levels. Assignment to a cluster can be achieved by classification or grouping in a multidimensional space spanned by a number of state attributes. Based on the combined static and dynamic state attributes, the individual data points can thus be assigned to different clusters.However, it may also be intended to use all or a given set of state attributes as the context of the data points in order to determine which state attributes have a noticeable influence on the quality of the data points of this type.

[0024] In one embodiment, annotating the sensor data frames comprises assigning at least one data point of a first type and at least one data point of a second type to the individual sensor data frames. Particularly preferably, data points of the first type are grouped based on the determination of clusters in a first multidimensional space, and data points of the second type are grouped based on the determination of clusters in a second multidimensional space, wherein the multidimensional space for a data point is spanned by a number of state attributes.This can be a state attribute assigned to the type of data points, but it can also be intended to use all or a given set of state attributes as the context of the data points and to determine which state attributes have a noticeable influence on the quality of the data points of this type by determining clusters.

[0025] Preferably, the first group is defined based on a first cluster for which the at least one state attribute lies within a first defined value range, and a second group is defined based on a second defined value range, wherein the first value range and the second value range are disjoint for at least one state attribute and / or all state attributes assigned to each data point. In principle, a division into a larger number of clusters is also possible.

[0026] Preferably, the error probability, and thus the quality level of a cluster, is determined based on the data points of a cluster using samples. For example, a first data point could be assigned to a first cluster and a second data point to a second cluster. In this example, a sample would determine the quality level of the first cluster as 100 percent and the second cluster as zero percent (or the corresponding inverse error probabilities). Statistical methods can be used to dynamically adjust the sample size during the measurement and to incorporate quality forecasts from previous measurements of the same cluster. The aim here is to raise the quality levels of the various clusters above a desired threshold with minimal manual testing and correction effort by iteratively improving the automated labeling. For a cluster with a higher error probability orFor data points of poorer quality, more samples are preferably taken and more data points are corrected for retraining. Particularly preferred is an error probability for each data point based on whether the data point is in the first or second group. This results in a prediction of the quality level: Based on the combined static and dynamic state attributes, the individual data points can be classified into different quality levels. For data points in the group with the higher error probability or poorer quality, more samples are preferably taken.

[0027] If the annotation of sensor data frames with a first type of data points is carried out based on a first neural network, and the annotation of sensor data frames with a second type of data points is carried out based on a second neural network, the further process steps for the data points of the first type and the further process steps for the data points of the second type are expediently carried out independently of one another. Determining quality levels or a statistical quality analysis can result in different error distributions for different data types. Through independent processing, error correction and retraining are carried out specifically for the respective type of data point and can be limited to the specifically required scope.

[0028] Preferably, the selection of frames for the first sample depends on the data points for which the quality measure is to be determined, in particular a random selection of individual frames for object detection and / or a random selection of batches of consecutive frames for object tracking. By applying an intelligent sampling strategy, the improvement achievable through retraining is maximized. An object detector, such as one used for traffic sign recognition, benefits from high-variance training data, so a random selection of individual frames is a useful first sample. On the other hand, a tracking component benefits from continuous data, as only then can tracking of the same object take place between consecutive frames. In this case, a sample would conveniently be randomly selected from a series of consecutive frames—for example, always 10—for a variety of objects.As an example, intelligent sampling would select frames 10 to 20, as well as frames 100 to 110, and 235 to 245 for the first sample when determining a quality measure for a tracking component. To maintain high variance in the sample, the software component performing the sampling may specify a minimum time interval between samples to ensure that different frames were acquired under different environmental conditions. Additionally or alternatively, one or more attributes may be considered during sampling. For example, if a sample is selected to quantify the capabilities of the object detector at night, different environments, such as a city, countryside, or highway, may be specified. Random selection would then be performed among all samples that meet the specified criterion.

[0029] Preferably, annotating sensor data and recording sensor data are performed alternately or simultaneously, and if it is determined that the quality measure for at least one frame in the first sample is below a predefined threshold, the computer requests the recording of additional sensor data for which the at least one condition attribute lies within the selected value range of the first packet. A value range of the condition attribute can be selected by equipping a test vehicle with an automated recording device that executes a selection program that triggers a recording as soon as a predefined recording condition is met, or by requesting a test driver to drive under specific conditions, e.g., at night. Thus, new data is recorded at least primarily for those environmental conditions for which the neural network requires further training.By carefully selecting training data, the improvement per training effort is maximized. This reduces the computation time required for training and also reduces energy consumption.

[0030] In one embodiment, receiving corrected annotations for a data point comprises receiving a plurality of preliminary annotations and determining a corrected annotation based on the plurality of preliminary annotations, in particular a selection based on a mean value or a majority decision. For data points of the bounding box type, the mean values ​​of several values ​​for the coordinates and / or the size of the bounding box can be calculated. For other types, a majority decision may be more suitable. To achieve higher annotation quality, preliminary annotations or partial annotations can be created by several labelers, on the basis of which the ground truth is determined. This is particularly advantageous for annotations of the first batches of sensor data frames, as it also enables a review of the labeling specification.

[0031] One aspect of the invention also relates to a non-transitory computer-readable medium containing instructions which, when executed by a microprocessor of a computer system, cause the computer system to carry out the method according to the invention as described above or in the appended claims.

[0032] In a further aspect of the invention, a computer system is provided comprising a host computer having a processor, a memory, a display, a device for human input, and a non-volatile memory, in particular a hard disk or a solid-state drive. The non-volatile memory contains instructions which, when executed by the processor, cause the computer system to carry out the method according to the invention. The processor may be a general-purpose microprocessor commonly used as the central processing unit of a personal computer, or it may comprise one or a plurality of processing elements designed to perform special calculations, such as a graphics processor.In alternative embodiments of the invention, the processor may be replaced or supplemented by a programmable logic device, such as an FPGA configured to provide a fixed set of functions and / or may include an IP core microprocessor.

[0033] The invention is explained in more detail below with reference to the drawings. Similar parts are labeled with identical designations. The illustrated embodiments are highly schematic, meaning that the distances and the lateral and vertical dimensions are not to scale and, unless otherwise stated, do not have any deducible geometric relationships to one another.

[0034] It shows:

[0035] Figure 1 shows an exemplary embodiment of a computer system, Figure 2 shows an example of a video frame with a schematic

[0036] Diagram of possible data points in the top left inset, Figure 3 is a schematic diagram of an automation system implementing a method according to the invention,

[0037] Figure 4 shows an example of data points grouped into clusters with different quality levels,

[0038] Figure 5a is a schematic diagram of a first step of batch processing of sensor data frames,

[0039] Figure 5b is a schematic diagram of a second step of batch processing of sensor data frames,

[0040] Figure 5c is a schematic diagram of a third step of batch processing of sensor data frames,

[0041] Figure 5d is a schematic diagram of a fourth step of batch processing of sensor data frames, and Figure 5e is a schematic diagram of a fifth step of batch processing of sensor data frames.

[0042] Fig. 1 illustrates an exemplary embodiment of a computer system.

[0043] The illustrated embodiment includes a host computer PC with a display DIS and user interface devices, such as a keyboard KEY and a mouse MOU; further, an external server may be connected via a network, as indicated by a cloud symbol.

[0044] The host computer PC comprises at least one processor CPU with one or more cores, a RAM memory, and a number of devices connected to a local bus (such as PCI Express) that exchanges data with the CPU via a bus controller BC. The devices include, for example, a graphics processor GPU for controlling the display, a USB controller for connecting peripherals, a non-volatile memory HDD, such as a hard disk or a solid state disk, and a network interface NC. Furthermore, the host computer can comprise a dedicated accelerator AI for neural networks. The accelerator AI can be implemented as a programmable logic device, such as an FPGA, as a

[0045] A graphics processor suitable for general-purpose computing, or as an application-specific integrated circuit. Preferably, the non-volatile memory contains instructions that, when executed by one or more cores of the CPU, cause the computer system to execute a method according to the invention.

[0046] In alternative embodiments, indicated in the figure as a cloud, the computer system may comprise one or more servers comprising one or more processing elements, wherein the servers are connected via a network to a client, such as the host computer PC. Thus, the annotation environment can be executed partially or entirely on a remote server, such as in a cloud computing facility. Mobile devices can also be used as a client as an alternative to a host computer; thus, a graphical user interface of the annotation environment can be executed, in particular, on a smartphone or a tablet with a touchscreen user interface.

[0047] Fig. 2 shows a camera image as an example sensor data frame with a schematic diagram of possible data points in the top left inset.

[0048] The photo of a city scene shown in the figure can be a single image or part of a video recording. In general, a recording provided by a customer can consist of video or audio data representing a sequential context, such as a 5-minute drive recorded via imaging sensors such as a camera and a LiDAR sensor, or a 10-minute voice recording. Video recordings, for example, could consist of a series of consecutive frames, each of which contains a series of objects. Using at least one neural network, the recording is processed to create annotations. Annotations can include multiple data points, where each data point describes a specific aspect.

[0049] A data point is a parameter that describes a specific property of a recording and can be applied to all levels of detail. Levels of detail can be the entire recording, a series of consecutive or random frames, a single frame, or an object on a frame. A specific example would be an annotation for a car, consisting of a bounding box describing the car's position within a certain accuracy, a vertical line marking an edge of the car, a classification to describe the car type, attributes for clipping or occlusion, turn signals, brake lights, color, and so on.Data points can be classes, boxes, segments, polygons, line strings, attributes such as turn signals, brake lights, colors, subclasses, tracking information, occlusion level, clipping level, complex classes describing the relevance of an object / frame / clip, sound, text, or any other automatically detectable information.

[0050] The inset in the top left of the figure shows various data points for a car. Cars can be of different types, e.g., a van, an SUV, or a sports car. The position, or rather the dimensions, of a car are generally specified by a bounding box—a rectangular frame or cuboid enclosing the car. Vertical lines indicate the car's boundaries. Another possible data point for a car is the activation of a light indicator, such as the turn signal shown in the inset.

[0051] A multitude of cars are present in the frame, each enclosed by a bounding box. Cars can be fully visible, such as the one driving directly in front of the camera, or they can be occluded. The traffic density of the urban scene can impair annotation quality, for example, by making it difficult to accurately determine the boundaries of the bounding box due to occlusion.

[0052] Fig. 3 is a schematic diagram of an automation system performing a method according to the invention. The automation system implements various steps of the method in dedicated components and is well suited for execution in a cloud computing environment.

[0053] In a first step, "data ingestion," unsorted recordings are received from a customer. The recordings can be normalized, e.g., divided into sensor data frames or images, to enable consistent processing. This step can also include an enrichment phase in which the sensor data frames of the recordings are automatically enriched with metadata relevant for measuring automation quality. For example, each image can be assigned the geographical location where it was captured, in particular based on the GPS coordinates received simultaneously with the images. In the context of autonomous driving, metadata or state attributes relevant to annotation quality could include a weather condition, a road type, a lighting condition, and / or a time of day.These state attributes indicate conditions during the acquisition of a sensor data frame and can also be described as static. Other state attributes, such as the size of a bounding box, which can also influence the labeling quality, e.g., in object detection (large objects are easier to detect), only emerge from annotation and can therefore be described as dynamic.

[0054] For automation efficiency, it is useful to process batches of frames or individual images together in the following steps. For projects with interleaved image capture and processing, it may be advantageous to accumulate frames captured under the same environmental conditions until a specified batch size is reached before proceeding to the next processing steps.

[0055] In a second step, the "scheduler," various batches of sensor data frames or individual images are scheduled for annotation by an automation engine. The scheduler can select one or more automation components to annotate the frames with one or more data points for execution by the automation engine. Furthermore, the scheduler can select the batch of frames to be processed based on the availability of new versions of automation components. An automation component can generate a single data point, such as a vertical line, or multiple related data points, such as the coordinates of a bounding box and an object class. The automation components can be neural networks or other machine learning-based technologies that learn from data samples in a supervised, semi-supervised, or unsupervised manner.

[0056] In a third step, the "automation engine," a batch of sensor data frames is processed by at least one automation component, which assigns data points to the frames. The automation system generates any type of data point via automation components; automation components are thus a central part of the automation system's workflow. Preferably, the data points are tagged with metadata that accurately describes the version of the automation component used. The automation engine includes techniques for precisely storing relevant metadata about automation components, such as a dedicated database. Some of the state attributes associated with a data point can be determined by a dedicated automation component. The "context," i.e., the state attributes for a data point, can include attributes that are themselves a data point.For example, the accuracy of placing a vertical line may depend on the size of the bounding box in which the line is to be drawn.

[0057] In a fourth step, "clustering," the individual data points of a specific type are grouped based on state attributes. It may be intended to assign specific state attributes to a data point type. The state attributes for a bounding box can include, for example, the size of the bounding box, the time of day and / or the weather conditions when the image was captured, and / or partial occlusion of the object. The values ​​of the state attributes of the individual bounding boxes can form multiple clusters in the multidimensional space spanned by the state attributes. Different clusters can be associated with different annotation quality.

[0058] Listing 1

[0059] Listing 1 shows an example context with multiple state attributes for two example data points, a bounding box BOI and a bounding box BO2, each of which specifies the position of an object. The individual state attributes describe conditions that could potentially influence the quality of the annotation.

[0060] Based on a multitude of individual data points of the same type, the automation system can thus determine clusters in a multidimensional space, in particular using a nearest neighbor algorithm and / or an unsupervised learning approach and / or a machine learning classification model. The identified clusters can be analyzed to establish a criterion for grouping data points and / or predicting annotation quality by defining value ranges for at least one of the data point's state attributes. Preferably, the grouping is performed based on defined value ranges for several state attributes. The grouping can, for example, be based on the dimensions of the bounding box; objects that are close to the camera or are large can enable precise placement of the bounding box.In contrast, the relative error in the placement of a bounding box around a small, distant object can be significant. Therefore, larger dimensions can correlate with higher bounding box quality. Weather is another condition attribute that can be used to group data points, e.g., due to lower contrast and / or image distortions caused by water droplets on the camera lens. Other condition attributes may not significantly influence data point quality variations and can be ignored; for example, the field of view or angle of view of a camera used to capture images may be constant for all images captured with that camera. Grouping based on ranges of values ​​can also be performed using a neural network or a machine learning classification model.

[0061] In a fifth step, "sample review," quality control is performed on a sample of data points. In a first phase, "sampling," several data points are selected for quality control based on sampling requirements. The frequency and / or size of the samples taken for a group of data points can be chosen depending on the predicted quality of the data points in the group; data points associated with state attributes that indicate poor quality can be sampled more frequently. In a second phase, "check and correct," a human annotator can be shown the frame with appropriate annotations, such as a bounding box, and asked whether the bounding box is correct.Alternatively, the annotator can be presented with a user interface for adjusting the bounding box and / or adding a bounding box in case of false negatives to annotate an object missed by the neural network. From the type and number of corrections made by the human annotator, the automation system determines a quality measure. Conveniently, the quality measure is chosen so that a missed object is weighted more heavily than a bounding box whose placement required refinement.

[0062] In a sixth step, "Sample verification passed?", the system determines whether the sample quality measure exceeded a predefined threshold (indicating sufficient annotation quality). If the automation system determines this is the case (Yes), the group of individual images comprising the selected sample can be exported and delivered to the client. If this was not the case (No), execution continues in a seventh step.

[0063] The seventh step, "Required for dataset?", determines whether the manually corrected sample should be used to retrain the automation component for the data point. Whether this is the case may depend on how many images were captured under the same conditions that were already used to train the model. If this is not the case (No), the sampled set of data points is returned to the scheduler (Re-automate with retrained model). Once a newly trained automation component is available for the data points, the scheduler sends the set of data points to the automation engine for reprocessing.If the corrected samples are to be used for retraining (Yes), the manually annotated data points are fed into the training / validation or test datasets for the respective neural network / automation component. These datasets are represented by a cylinder. In an eighth step, the "flywheel," the neural network or automation component that generated the data points rejected during sampling is retrained. By relearning the neural network, the quality of the automation is improved. Preferably, the automation components are improved to the point where manual review is no longer required for as many clusters as possible. The iteration times for retraining should be as short as possible to enable rapid efficiency improvements.

[0064] Flywheel includes techniques for efficiently storing and versioning training data sets for each automation component or data point type, monitoring changes to the training data sets, and automatically triggering retraining when predefined or automatically determined thresholds for changes to the training data sets are exceeded (e.g., a specified number of new examples). Furthermore, Flywheel includes techniques for automatically deploying retrained neural networks in automation components and informing the scheduler about version changes.

[0065] If new data frames are acquired simultaneously or alternately with the annotation of the data frames, an additional step of targeted data collection can be performed. The automation components are improved through many training iterations on a continuously refined dataset that increasingly better reflects real-world variance over time. At least for static state attributes, a systematic approach can be taken based on confidence levels or error probabilities per cluster to collect data frames for situations where automation results suffer the most. For example, automatic annotation of sensor data frames acquired at night may result in unacceptable annotation quality.Once this is identified during sample verification, targeted nighttime data acquisition can be requested to improve the automation component's training dataset under these environmental conditions. Specifically, the amount of additional training datasets under the problematic environmental conditions can be determined based on the confidence or error probability determined for the corresponding cluster. All data recorded under the same conditions can be used for retraining. Once the incorrectly annotated sensor data frames have been corrected, they are fed directly into the training dataset of the corresponding automation component. However, typically, not all data for a specific cluster and data point needs to be manually corrected.Instead, only samples up to the next threshold are collected and corrected for retraining. The rest of the data is automatically scheduled for a rerun with a higher version of the automation component. Targeted data collection includes techniques for selecting interesting samples based on clusters up to predefined sets for manual correction. Furthermore, it preferably includes techniques for flagging poor-quality samples that are not needed for retraining for automation runs on higher versions of the respective automation component.

[0066] If the automatic annotations of the sample reviewed in the sixth step are of sufficient quality, the annotations can be delivered to the customer. In a ninth step, "Customer Sample Review," the customer can review a sample of the exported sensor data frames to ensure that the annotations meet their specifications and maintain the required annotation quality. If the customer rejects the group of frames, a sample or the entire group of frames is manually processed in a tenth step, "Correction." The ninth and tenth steps are optional and can therefore be omitted. In the tenth step, "Correction," a manual annotation is performed on a sample or the entire group of sensor data frames rejected by the customer. Optionally, the manually annotated frames can be exported for re-review by the customer.The manually annotated frames are preferably used for retraining the neural network by feeding the corrected data into the training, validation or test datasets.

[0067] Figure 4 shows an example of data points grouped into clusters with different quality levels.

[0068] Shown are sections of sensor data frames recorded with a camera, each showing a vehicle. A bounding box was drawn around each vehicle to enclose its outline. Additionally, the vehicles were annotated with a vertical line, each indicating an edge of the vehicle and thus allowing conclusions to be drawn about the relative angle between the vehicle and the camera. Thus, data points of two different types are shown, with the bounding box representing a primary data type that can be present independently in an image or sensor data frame. In contrast, a vertical line is only drawn for detected vehicles and therefore represents a secondary data point.

[0069] The relative accuracy of the bounding boxes depends, for example, on the size of the object they encompass, because large objects are easier to recognize than small or distant objects. The size of the bounding box also has a significant impact on the accuracy of the vertical line. Other factors influencing the quality of annotation with vertical lines can include lighting conditions and the degree of occlusion, which can thus represent relevant state attributes. The image sections shown or detected vehicles are clustered into three groups for which different error probabilities were predicted or determined. The left column shows examples of cluster 1, which includes high-quality data points (or vertical lines) with an error probability (error WS) of 2%.The middle column shows examples of Cluster 2, which includes medium-quality data points with an error probability (error WS) of 8%. The right column shows examples of Cluster 3, which includes low-quality data points with an error probability (error WS) of 18%.

[0070] Clustering allows for the determination of value ranges for the relevant state attributes, for example, an occlusion level of more than 30% correlates with poor annotation quality. The shape of a cluster can be complex, especially when there are many relevant state attributes; such a cluster can be usefully described by trained neural networks or a machine learning classification model.

[0071] Figure 5a is a schematic diagram of a first step in batch processing of sensor data frames. The processing can be performed in an automation system such as the one shown in Figure 3; the steps not shown can be performed as part of the batch processing.

[0072] Dividing complex annotations into individual data points enables a fine-grained analysis of the state attributes relevant to the quality measure. Furthermore, the required computing time is reduced because only those data points for which, for example, a retrained neural network is available need to be considered, while other data points from the sensor data frames can be retained. The general handling of data points is explained here using a simplified example that contains only one type of data point (e.g., bounding boxes around objects) and two clusters (Cluster A = poor quality, Cluster B = good quality) for this type of data point.

[0073] The sensor data frames received as input data, e.g., camera images, are divided into batches of a fixed size to enable consistent processing by the automation engine. The figure shows two batches of 500 frames of sensor data each. The automation engine executes a neural network for object detection, which assigns bounding boxes to objects within a frame.

[0074] After the batches have been annotated, with the individual data points also being assigned a context of various condition attributes, the data points are grouped into Cluster A (dotted border) with poor quality and Cluster B (dash-dotted border) with good quality. From a single camera image or

[0075] For example, a frame can result in three data points in cluster A and two data points in cluster B. In the example shown, cluster A contains 2000 data points and cluster B 1100 data points (DP).

[0076] Figure 5b is a schematic diagram of a second step of batch processing of sensor data frames.

[0077] Once the clusters have reached a certain size and / or a specified time interval has elapsed, a sample review is performed. Based on predefined sampling requirements, a few data points are taken as a sample. This is followed by a manual review and correction step (the other steps can be performed fully automatically by a computer), and the automation system receives corrected data points for the sample. For simplicity, the entire cluster is taken as a sample.

[0078] In the example shown, cluster A has reached the size threshold for sample verification (indicated by a magnifying glass), whereas cluster B is not initially verified (indicated by an hourglass). As a result, it turned out that 30% of the data points in cluster A had to be corrected (e.g., correct 30%) to achieve the desired quality level. For simplicity, it is assumed here that all data points from a corrected sample are used to retrain the neural network. Thus, 600 corrected data points are available for inclusion in the training data sets.

[0079] Figure 5c is a schematic diagram of a third step of batch processing of sensor data frames.

[0080] The example shows that additional batches of sensor data frames or camera images were received as input data, with batches 21 and 22 currently being processed. Although the size of cluster B has not changed in the example shown, a trigger condition for the sample verification of cluster B is met: A specified number of batches (20) has been processed, whereupon a sample is to be taken from all clusters not yet verified and verified or corrected (correct / terminate all open clusters).

[0081] Figure 5d is a schematic diagram of a fourth step of batch processing of sensor data frames.

[0082] During the sample review of cluster B (indicated by a magnifying glass), it was found that 10% of the cluster's data points need to be corrected (e.g., correct 10% of the data points) to achieve the desired quality level. Since all data points of a corrected sample are used to retrain the neural network, 110 corrected data points are available for inclusion in the training data sets. In batch processing, there is also the option of training on all annotated sensor data frames of the corrected batch, or exclusively on the corrected data points. Figure 5e is a schematic diagram of a fifth and final step in batch processing of sensor data frames. As in Figure 3, a data acquisition module and a scheduler are also shown here.

[0083] Batch 1 and Batch 2 are shown; three dots indicate numerous additional batches. After dividing the data points of the batches into Cluster A and Cluster B, and checking and correcting samples, the neural network was retrained. Once the desired quality level has been achieved in further samples, the batches can be delivered with commitments regarding statistical quality (delivery to customers). This allows significant portions of the annotated sensor data frames to be delivered without the need for manual rework.

[0084] The described method can also be used for sensor data frames from a lidar sensor, i.e., point clouds, or for multi-sensor setups. Independent grouping and correction are performed for the different types of data points. By manually correcting only the samples required for training, a large portion of the input data can be automatically annotated as soon as the retrained neural network is available for the respective type of data point.

[0085] By leveraging the correlation between the quality of annotations and state attributes, the method according to the invention enables the targeted application of manual work to the rapid improvement of neural networks, which are then used to create automatic annotations for delivery to the customer. By processing different types of data points separately and re-annotating them, for example, only once a retrained neural network is available, computing time is used particularly effectively. Overall, larger annotation projects, such as those required for validation, are significantly accelerated.

Claims

- 32 - Claims 1. Computer-implemented method for automatically annotating sensor data, wherein the method comprises Receiving a large number of sensor data frames, Annotating the multitude of sensor data frames using at least one neural network, wherein the annotation includes assigning at least one data point to each sensor data frame and at least one state attribute to each data point, Grouping the data points based on the at least one state attribute, wherein a first group comprises data points for which the at least one state attribute lies within a defined range of values, Selecting a first sample from one or more data points from the first group and determining a quality measure for the data points in the first sample, wherein, if the computer determines that the quality measure of the first sample is below a predefined threshold, the procedure further comprises Receiving corrected annotations for the data points in the first sample, retraining the neural network based on the data points in the first sample, selecting a second sample from one or more data points of the first group that were not in the first sample, Annotating the sensor data frames of the second sample with the retrained neural network and determining a quality measure for the data points in the second sample, However, once the computer determines that the quality measure of the first or second sample is above a predefined threshold, the procedure continues. Annotating the remaining sensor data frames of the first group with the neural network, and Exporting the annotated sensor data frames of the first group. - 33 - Method according to claim 1, wherein if the computer determines that the quality measure of the second sample is below a predefined threshold, the method further comprises Receiving corrected annotations for the data points in the currently checked sample, retraining the neural network based on the data points in the currently checked sample, Selections of a further sample from one or more data points of the first group that were not part of a previous sample, Annotating the sensor data frames of the further sample with the neural network and determining a quality measure for the data points of the further sample, wherein the further steps are repeated until the computer determines that the quality measure for the frames in the further sample is above the predefined threshold or no sensor data frames with uncorrected data points remain for the sample, and wherein, once the quality measure for the sensor data frames of a sample is above the predefined threshold, the procedure further comprises Annotating the remaining sensor data frames of the first group with the neural network, and Exporting the annotated sensor data frames of the first group. A method according to any of the preceding claims, wherein at least one of the state attributes is determined by a dedicated neural network based on the sensor data frame and / or at least one of the state attributes is determined based on additional sensor data recorded simultaneously with the sensor data frame.

4. A method according to any of the preceding claims, wherein the sensor data frames are image data frames, i.e., they comprise data from an imaging sensor, wherein for image data frames the state attribute is a geographical location, a time of day, a weather condition, a visibility condition, a road type, a distance to an object and / or a traffic density, a size of a boundary box, an extent of occlusion and / or clipping, an ego vehicle speed, a camera parameter, a color range and / or a contrast measure of an area encompassed by a boundary box, a direction of travel of the ego vehicle, astronomical information such as the sun's position relative to the direction of travel of the ego vehicle, and / or wherein for image data frames the at least one data point is a position of an object, a class of an object, coordinates of a boundary box, coordinates of a line, a clipping of an object, an occlusion of an object,This includes a correlation of an object in the image data frame with an object in a preceding or subsequent image data frame and / or the activation of a light indicator, such as a turn signal or brake light.

5. Method according to any of the preceding claims, wherein the sensor data frames are audio frames, i.e., comprise data from an audio sensor, wherein for audio frames the state attribute is a geographical location, a gender and / or an age of a speaker, a room size and / or a measure of background noise and / or wherein for audio frames the at least one data point comprises a phoneme and / or one or more words of a text recognized from the audio frame.

6. A method according to any of the preceding claims, wherein the grouping of the plurality of data points comprises determining clusters in a multidimensional space, in particular using a nearest-neighbor algorithm and / or a Unsupervised learning approach and / or a machine learning classification model.

7. The method of claim 6, wherein the annotation of the sensor data frames comprises assigning at least one data point of a first type and at least one data point of a second type to the individual sensor data frames, and wherein data points of the first type are grouped based on the determination of clusters in a first multidimensional space and data points of the second type are grouped based on the determination of clusters in a second multidimensional space, wherein the multidimensional space for a data point is spanned by a number of state attributes.

8. Method according to claim 6 or 7, wherein the first group is defined on the basis of a first cluster for which the at least one state attribute lies in a first defined range of values, and a second group is defined on the basis of a second defined range of values, wherein the first range of values ​​and the second range of values ​​are disjoint for at least one state attribute and / or all state attributes that are assigned to each data point.

9. The method of claim 8, wherein an error probability is determined for each data point based on whether the data point is in the first or the second group, and wherein more samples are taken for data points in the group with the higher error probability.

10. Method according to any of the preceding claims, wherein the annotation of sensor data frames with a first type of data points is based on a first neural network, and the annotation of sensor data frames with a second type of data points is based on a second neural network, - 36 - and wherein the further procedural steps for the data points of the first type and the further procedural steps for the data points of the second type are carried out independently of each other.

11. Method according to any of the preceding claims, wherein the selection of sensor data frames for the first sample depends on the type of data points for which the quality measure is to be determined, in particular a random selection of single images for data points relating to object detection, and / or a random selection of stacks of successive frames for data points relating to object tracking.

12. Method according to one of the preceding claims, wherein the annotation of sensor data frames and the reception of sensor data frames are performed alternately or simultaneously, and wherein, when the computer determines that the quality measure for a sample is below a predefined threshold, the transmission of sensor data frames is requested in which the at least one state attribute is within the defined range of values.

13. Method according to one of the preceding claims, wherein receiving corrected annotations for a data point comprises receiving a plurality of preliminary annotations and determining a corrected annotation based on the plurality of preliminary annotations, in particular a selection based on a mean or a majority decision.

14. Non-volatile, computer-readable medium containing instructions which, when executed by a processor of a computer system, cause the computer system to execute a method according to any of the preceding claims.

15. Computer system comprising a host computer, wherein the host computer comprises a processor, a memory, a display, a - 37 - The input device comprises a non-volatile memory, wherein the non-volatile memory comprises instructions which, when executed by the processor, cause the computer system to execute a method according to any of the preceding claims.